PWSATT~2.DOC

Attachment 2: Research Designs to Conduct Three Studies for AmeriCorps NCCC Organization of the Research Design Report This report is organized into four sections. The first section describes the development and preparation of the service projects database which CNCS will use to conduct ongoing analysis on how NCCC strengthens communities, and will also serve as a data source for future studies of the impact of NCCC on member outcomes and communities served. The other three sections describe the research design for the following three future studies:

1. Leadership Development: A mixed-methods longitudinal evaluation to examine how participation in NCCC impacts members’ leadership skills and overall civic participation.

2. Member Retention: A mixed-methods evaluation to understand factors affecting retention of NCCC members at different stages and at different NCCC campuses.

3. Strengthening Communities: A mixed-methods exploratory and qualitative in-depth case study of a sample of NCCC service projects to gauge how these projects strengthen communities.

Implementing the proposed research designs positions NCCC and ORE to expand knowledge that adds to the agency’s evaluation learning agenda, showcase the unique value of NCCC, continue to promote a culture of evaluative thinking and practice, and make relevant knowledge and information accessible for internal and external stakeholders.

Each research design relies on NCCC’s service projects database that will provide information on the characteristics and outputs of all service projects since 2012. Concurrent with conducting the three studies, the evaluator will populate the NCCC service project database, currently containing project data from 2012 to spring of 2019, with quantitative and qualitative data on NCCC service projects completed between summer 2019 (approximately 400 projects) and fiscal year 2020 (approximately 900 projects). The evaluator will merge the data from the service projects database with Corps members responses in the leadership development survey to examine the relationship between service projects’ characteristics and leadership qualities and skills, and retention. The service projects database will serve as the sampling frame to select projects for the exploratory strengthening-communities study.

Leadership Development Study This section describes the impact quasi-experimental design for the leadership development study. This section includes a synopsis of the literature on youth leadership, describes the logic model for leadership development, the research questions guiding the study, participants enrollment, data collection and proposed analysis.

Literature Review JBS conducted a review of the literature on youth leadership development in national service programs. The underlying assumption of theories of youth leadership is that these qualities and skills are learnable competency. The existing theories and studies suggest that community service, in its various forms, can be an appropriate intervention for youth to develop leadership qualities and skills. These findings have important implications for structuring the NCCC program since leadership development is one of the program’s core goals. There is, however, scant research that has rigorously tested these models of youth leadership development.

JBS’s review of the literature showed that the current knowledge base around the impact of national service program participation on the four core outcomes of NCCC’s leadership development—professional skills, life skills, teamwork, and civic engagement. The body of research suggests there is a positive impact of national service experience on professional skills such as employment aptitude, employment outcomes, employability and career goals, and civic engagement.

Despite these positive findings, there are some areas of leadership development among youth volunteering that merit further research. For example, Jastrzab et al.’s (2004) findings of a significant negative impact among members’ appreciation of cultural and ethnic diversity warrants a better understanding of the reasons behind this finding and their long-term implications on leadership qualities and skills. For many other aspects of leadership development, studies have primarily focused on short-term outcomes around knowledge, skills, and attitudes. Few studies have examined the impact of service on life skills, teamwork, and professional development.

There is an extensive body of research on professional skills and civic engagement. For these two outcomes there is an extensive and well-established set of measures. The body of research has primarily focused on perception-based outcomes, such as participants’ perceptions of employability or acceptance of responsibility for employment outcomes. There is limited research on long-term impacts related to professional skill development. it is important for future research to examine how the short-term attitudes and perceptions translate into behaviors and experiences, such as alumni employment and professional success.

There is limited research on teamwork and life skills. Life skills as one of the attributes of leadership remains relatively less well-defined than areas such as professional skills and civic engagement. NCCC might consider how to develop a conceptual model and indicators of NCCC’s contribution to the development of life skills outcomes with a focus on soft skills (such as social, emotional, and cognitive skills) as well as self-efficacy.

Logic Model and Theory of Change The AmeriCorps theory of change is to contribute to the creation of empowered and prepared citizens who will be more civically engaged, will work to strengthen their communities, and will be dedicated to improving the lives of those in their community. The roots of the pathway to achieve this vision lie in the member service placement which enables members to personally experience the challenges and needs of the community and provides opportunities for them to develop and engage in solutions to address these problems.

The primary intervention for Corps members’ leadership development is NCCC’s Training and Professional Development combined with service placement over a period of 10 months (this is 12 months for FEMA Corps). The training and service placement ensure member success in the program and enhance member personal and professional development. The logic model shows the specific activities (training and participation in service projects) that embody the intervention intended to foster leadership and civic participation. The training and service placement activities contribute to the goal of creating members who are empowered and prepared citizens via four essential pathways:

· Broaden knowledge in hard skills such as project management and soft skills such as strategic thinking. This knowledge increases members’ ability to work effectively both in the workplace and in the community.

· Provide an environment through the service experiences to gain hands-on experiences to solve problems and promote the ability to plan, prepare, execute and assess an endeavor. As such, members develop a change and action orientation to the work and sense of self-efficacy.

· Develop the ability to work effectively with and within diverse groups which is essential for uniting different groups to achieve a common goal.

· Develop civic attitudes and behaviors such as volunteering and voting to help propel community change.

The logic model for the NCCC Developing Leaders evaluation states that when participants successfully complete their trainings and service projects and receive the support and mentoring from staff, team leaders and sponsors, then they can improve on four core outcomes 1) gain professional skills; 2) develop life skills; 3) able to work as a team; and 4) engage in civic life.

Research Questions The leadership development study is guided by the following research questions:

1. What is the profile of NCCC members, including their demographic characteristics, sources to learn about NCCC, and motivations for serving? How do members perceive the NCCC experience prior to the start of their service? How do Traditional Corps and FEMA Corps differ?

2. How do NCCC members differ in interpersonal skills, life skills, including attitudes and behavior in group settings prior to the start of service compare to youth of similar background who do not serve with NCCC? How do Traditional Corps and FEMA Corps differ?

3. What is the impact of NCCC participation on members’ professional skills, life skills, teamwork, and civic engagement? How do these skills change at the end of their service and one year following the end of service? How does the impact vary for Traditional Corps and FEMA Corps?

4. How do members’ service projects experiences contribute to their professional skills, life skills, teamwork, and civic engagement? How does the contribution of service experiences vary for Traditional Corps and FEMA Corps?

Research Design The study will use a matched quasi-experimental design (QED), and will enroll participants from the incoming NCCC Class 26 (Winter and Summer cycles) and NCCC Class 27 (Fall cycle). The service months for a NCCC Class is divided into three cycles spanning from October through April; the Fall cycle for a class begins in October, the Winter cycle begins in February, and the Summer cycle begins in July. The spread of service for Class 26 and Class 27 is illustrated in Figure 1. Enrollment through graduation for three cycles spans from February 2020 to July 2021.

Figure 1 NCCC Class 26 and Class 27 Months of Service

Applicants who have cleared the background check and have accepted to serve will constitute the treatment group. Applicants who have been accepted but have declined to join will constitute the comparison group. Applicants in the comparison group who have not cleared the background check will not be invited to participate in the study since failure to pass the background the check would make these individuals ineligible to become NCCC Corps members. Participants will be administered a survey at three different time points or cycles. A sample of Corps members will also participate in focus groups and semi-structured interviews during their term of service. During the term of service, prior to Corps members graduating, a second comparison group will emerge from Corps members who end their service before they graduate.

An analysis using baseline survey responses will be conducted to test baseline equivalence of the treatment and comparison groups. If upon this analysis the two groups are not equivalent, within each campus, the evaluation will statistically match participants in the treatment group to those in the comparison group to make both groups as equivalent as possible. The impact analysis will be conducted twice with the matched sample. The initial impact analysis will occur at the end of the program participation period for each of the three cohorts. This will occur 10 months after program enrollment. The second impact analysis will be conducted one year after members graduate from the program.

The QED design will provide rigorous estimates of the impact of participation in NCCC on members’ leadership qualities and skills measured by their professional skills, life skills, interpersonal skills, community and civic engagement. Using a matched quasi-experimental approach, the evaluator will estimate NCCC’s impacts by comparing the outcomes of Corps members (treatment group) to the outcomes of individuals who are observationally equivalent to program participants (comparison group).

Since participants are nested within campuses, a multilevel analysis will be conducted. The macro-level unit of analysis will be the campus, while the micro-level unit of analysis will be the individual member or non-member. Statistical matching will, to the extent possible, adjust the analysis sample to maximize the observational equivalence of the treatment and comparison groups. Implementing the QED involves:

1. Collecting surveys from treatment and comparison group participants at three time points; baseline prior to the start of service, at the end of service, and one year after Corps members complete their term of service. The measures to be collected through survey include background and demographic characteristics, prior volunteer service, and leadership qualities and skills.

2. Applying matching methods to construct equivalent comparison groups, which are comprised of individuals with the same observable characteristics as participants in the treatment group.

3. Producing rigorous estimates of program impacts through outcome comparisons between the treatment and the comparison groups.

In addition, outcome analysis to assess the relationship between characteristics of service projects and members leadership qualities and skills will be conducted with the treatment group. The outcome analysis for Corps members is a mixed-methods approach that involves:

4. Merging Corps members survey responses with administrative data from CNCS’s developing service projects database on projects the members were assigned during their term of service.

5. Triangulating Corps members survey responses with focus groups and semi-structured interviews data.

Matching methods are a reliable approach for producing rigorous impact evaluations, particularly when a random assignment design is not feasible. Matching methods provide credible impact estimates when: 1) the data include large samples of non-members; and 2) matching is performed based on rich information on member and non-member characteristics. Rich data collection on treatment and comparison group participants is necessary to ensure that observable characteristics influencing program participation can be sufficiently approximated.

Comparison group participants should ideally be as similar as possible to Corps members prior to statistical adjustments in terms of similar geographic locations and background characteristics as Corps members. To construct matched comparison group data, the study will rely on member survey and program administrative data which will provide rich information on the characteristics of all applicants who applied for membership in the NCCC program. The available sample for the comparison group will need to be large enough to locate matches for Corps members in the treatment group and will be comprised of Corps members who did not complete their term of service and applicants who met the criteria to serve and were selected but did not become members.

The evaluator may use propensity score methods. Researchers use the propensity scores in several ways to reduce confounding factors, including matching on the propensity score, stratification on the propensity score, inverse probability of treatment weighting (IPTW) using the propensity score, and covariate adjustment using the propensity score (Austin, 2016; Stuart, 2010). For example, IPTW or full matching could be used if the decision based on the available is to estimate the average treatment effect (ATE). Alternatively, given the possibility that there might not be as many comparison participants as treatment participants, appropriate matching choices might be subclassification and weighting by the odds to estimate the average treatment on treated (ATT). Once matching is achieved, it is necessary to test if treatment and comparison individuals share similar characteristics. These tests involve comparisons of variable means and standard deviations between the treatment and the comparison group. If treatment-comparison differences in characteristics are detected, the evaluator will modify the model specification used to match (e.g., include polynomials to capture nonlinearities or multiplicative terms to capture variable interactions, changing the method of matching) to eliminate such differences and ensure that a successful matching is achieved. This is a particularly necessary step for the internal validity of the impact analysis.

The qualitative research component for the leadership study will use focus groups and semi-structured in-depth interviews to further explore members’ leadership qualities and skills. The evaluator will meet with a diverse group of NCCC stakeholders to record their views and experience on members’ leadership qualities and skills as a result of service with NCCC. The information captured will be used to complement the quantitative survey results. The discussion questions will allow study participants to launch into in-depth discussions of their own conceptualizations of the topics for the leadership development study.

The evaluators will utilize the Office of Management and Budget (OMB) approved instrument for each target group at the scheduled time period for the data collection. The moderator guides for each group will encompass specific questions to address the research questions for the leadership study and conclude with a discussion of recommendations and suggestions to improve the experience NCCC stakeholders have had with the NCCC program.

Participants In the most recent application cycles for Class 24, NCCC received 9,207 applications for both Traditional and FEMA Corps. Of those, 3,645 received an acceptance letter. Of those accepted 1,788 confirmed and 1,857 declined, withdrew or were subsequently rejected. Preliminary applications data for Class 25 suggest that enrollment rate might be declining. Based on this most recent available data on applications received and acceptance rate, it is estimated that the incoming members from the Winter and Summer cycles from Class 26 and the Fall cycle from Class 27 will consist of approximately 1,600 applicants who accept to serve. As previously stated, the comparison participants will be drawn from accepted applicants who declined to serve. Based on previous years’ enrollment data, typically half of the accepted applicants will decline to serve. However, it is anticipated that the comparison group will be more difficult to reach and enroll in the study. For this reason, the evaluator will over recruit comparison participants. The comparison group participants will be recruited from the applicants list from Class 25 (Fall, Winter, Summer), Class 26 (Fall, Winter and Summer), and two cycles from Class 27 (Fall and Winter). Across these eight cycles during the enrollment period, it is expected that the total number of accepted applicants who declined to serve will be approximately 3,800. Among the comparison group the acceptance rate to participate is expected to be 30 percent, yielding an enrollment of 1,140 comparison participants at baseline. The projected number of participants is shown in Table 1. Once enrolled in the study, the evaluator will follow an effective strategy to minimize respondent attrition and achieve an 80 percent rate at each follow-up survey.

Table 1 Projected Number of Participants at Each Data Collection Point, Treatment and Comparison Groups

Note: Comparison group will also include accepted applicants from Class 25.

Recruitment for baseline survey administration. CNCS will provide the evaluator with two lists. One list consisting of incoming members for each of three cycles with members’ name, NCCC ID number, email address, phone number, campus where member will serve, whether member will serve in Traditional or FEMA Corps. The second list will include the list of accepted applicants who declined to serve; applicants who did not clear background check will not be included in the list. The second will also include applicants name, email address, phone number, whether member was accepted for the Traditional or FEMA Corps program. Surveys will be available online and by phone, including online or paper-and-pencil self-administered surveys and telephone interview. Mode effects will be analyzed, to ensure that differences in modality do not impact study results. The evaluator will first contact participants on both lists via email to invite them to participate in the study. The email message will include a summary of the study, assure participants that their participation is voluntary and confidential. The email will include a link to the online baseline survey. For the baseline survey administration, the evaluator will send three reminder emails to all participants (members and comparison groups) for online surveys and call up to three times to administer the survey by phone. The evaluator will provide NCCC a list of incoming members who did not complete the baseline prior to arriving on campus. Those members will be asked to complete the survey prior to the start of training.

Follow-up survey administration. The evaluator will send an email to each member and comparison participants who completed the baseline survey. The evaluator will send three reminder emails to all participants (members and comparison groups) for online surveys and call up to three times to administer the survey by phone.

Recruitment for focus groups and interviews. The evaluator will select participants for the focus groups and interviews from the leadership baseline survey respondents using a random sample stratified on gender, education, race/ethnicity, or prior volunteer experience. These factors would provide insights on how these respondent characteristics contribute to member retention, leadership, and insights about their service projects and contribution in the communities.

The NCCC staff and Sponsors/FEMA POCs focus group participants will be those that volunteer to take part in the study. The evaluator will work with campus POCs to acquire a list of 10 to 12 Sponsors/FEMA POCs that have volunteered to participate in the study, in which the evaluator will confirm with 6 to 8 participants to attend or participate in the discussion groups.

Sample Retention It is necessary for a successful evaluation study to avoid loss of the sample due to missing data and attrition. It is expected that the members will be much easier to locate and survey because most will still be serving with NCCC at the time of first follow-up survey, and for those who complete their term of service they will have some degree of connection with NCCC during the following years when the second follow-up is administered. The most focused attention for retention efforts will be necessary for members who leave the program prior to completing their term of service, and comparison participants. The evaluator will monitor differential rates of attrition between members and comparison participants for Traditional and FEMA Corps programs.

The specific strategies that will be used to maximize enrollment rate at baseline and to maximize the response rate at each of the follow-up survey:

1. Over recruitment of comparison participants;

2. Targeted communications via the survey introduction, cover letter, telephone script, and email message;

3. Using multiple reminders and multiple modes to contact and follow-up with participants; and

4. Incentive offers.

Over recruitment of comparison participants: Over recruitment of comparison group members will be used to ensure enrollment is met and to maximize the likelihood of a good match with members in the treatment group.

Targeted Communication: The assessment introduction, cover letters, and telephone scripts discussed as described in the OMB supporting statement are designed to encourage participation and reduce nonresponse.

Using Multiple Reminders, Contacts and Communication Modes: The evaluator will use multiple communication approaches including the following approaches.

· The evaluator will obtain information on key friends and family members who participants provide as a good way to reach them at the baseline survey, and maintain regular contact with participants during non-survey periods (e.g., every other month) through calls, texts, and/or postcards.

· A multimodal survey will be used to improve the response rate. The survey will be distributed to respondents via email, and/or telephone.

· Participants will be sent up to three reminder emails (for online surveys) or called up to three times (for phone surveys) at each time point.

· All participants will be informed of the significance of the survey to encourage their participation. In addition, processes to increase the efficiency of the survey (e.g., ensuring that the survey is as short as possible through pilot testing, using branching to ensure that respondents read and respond to only questions that apply to them) and the assurance of confidentiality will encourage survey completion.

Incentive Offers: The use of incentives has a positive impact on increasing the response rate with no adverse effects on reliability. Participation in the study is voluntary, however respondents will likely perceive a time cost and burden associated with their participation. The use of incentives to increase the response rate is particularly important when collecting data to assess impact of NCCC from comparison group participants who decided not to serve with NCCC or who terminated their service early.

Table 2 Proposed Incentives for Target Participants

Corps members will not receive any monetary incentive during their term of service. Once members complete their service or leave the program early, they will be more difficult to reach and will become disinterested in the study. As such, the opportunity cost for the time spent completing the survey increases. After the end of members term of service, CNCS will offer alumni a modest incentive of a $20 gift card to complete the second follow-up (final) survey. CNCS will offer a $20 gift card to members who leave early and do not complete their service to complete the first and second follow-up surveys. CNCS will offer a $20 gift card to comparison group participants for each survey completed (a total of three surveys). Respondents will be asked to provide an address where the gift card can be sent upon completion of the survey. CNCS will provide in-kind NCCC-branded promotional items to beneficiaries who participate in focus groups or interviews in the community strengthening case studies.

Mixed Methods Data Collection Prior to the start of service for each cycle, before Corps members arrive at their assigned NCCC campus, the evaluator will send an individualized link to each incoming Corps member (treatment) to complete the online baseline survey. At the same time incoming members receive their invitation, the evaluator will send an individualized link to each individual in the comparison group inviting them to participate in the study and complete the baseline survey.

The leadership-development study will use a longitudinal, mixed-methods data collection including a survey administered at three time points.

Survey administration timeline Figure 2 shows the timeline for recruiting participants into the study. The leadership development survey is included in the appendix and will be administered at:

· Baseline (first survey): prior to training before members report to their campus. Recruitment of members and comparison group will occur at least two months prior to members reporting for their service. As shows in Figure 2, incoming members for the fall cycle which begins service in October, the recruitment period is August through September; for the Winter cycle which begins service in February, the recruitment period is December through January; and for the Summer cycle which begins service in July, the recruitment period is May through June.

· Follow-up (second survey): At the end of service; the sample will consist of members who graduated, those who terminated their service early, the comparison group drawn from the list of accepted applicants who decline to serve with NCCC. The second survey will be administered over a period of two months near the end of the term of service to members and comparison group respondents who completed the baseline survey.

· Follow-up (third/final survey): Approximately one year following the end of the term of service. All participants who completed the second survey.

Figure 2 Leadership Survey Administration Timeline, Members and Comparison Group

Assuming enrollment into the study begins with Class 26 (Winter cycle), the data collection period for the baseline and first follow-up surveys will span 22 months. The third survey, intended to be administered one year following the end of service, will span 11 months. The baseline survey will be administered to Corps members prior to training, and applicants who are accepted but will not join (e.g., declined acceptance); applicants who are accepted but did not show up for training will be in the comparison group sample. The evaluator will reach out to accepted applicants who have successfully completed the background check. Outreach to comparison group participants will occur at the same time as incoming Corps members.

This is primarily an online survey with the possibility of paper and phone survey with participants who do not complete the online survey. The goal is to enroll all Corps members invited to participate into the study.

The first follow-up survey will be administered to all respondents who completed the baseline survey, including those who leave the program early. First follow-up is expected to occur at the end of the program which is 10 months for Traditional NCCC and 12 months for FEMA Corps. Comparison group participants will complete the first follow-up at 10 months and 12 months (depending on which NCCC Corps members they are a matched for). The second follow-up survey will be administered to all respondents who completed the first follow-up survey, including those who leave the program early.

Measures The initial baseline survey will include questions on demographic characteristics, motivation to serve, and baseline outcome characteristics which will be used to form the matched samples and will be included as controls in the impact analysis regression models. The background and demographic variables to be collected include age, gender, living arrangement, parental education and employment, prior volunteer experience, education, prior employment history and area of residence. These measures were designed to encompass the various factors that may determine both the likelihood that an applicant may be selected for participation in NCCC, as well as an individual’s education level and employment status. The study’s outcomes will be measured at each time point. Outcomes will be assessed in professional skills, life skills, team behavior, community and civic engagement. Questions will assess changes in desired outcomes in each of these areas.

Survey responses will also be merged with CNCS’s developing service projects database maintained by NCCC on the characteristics of each members’ service deployment during their term of service. The data on service projects will be important to gain exploratory insight into the heterogeneity of members’ experiences, and the association that the nature of participation activities shares with leadership qualities and skills measured in the survey. Characteristics of projects might include length of project, location of project (ZIP Code), community need addressed, project goals, outputs and outcomes.

Focus groups and semi-structured interviews The evaluator will collect qualitative data through focus groups and semi-structured interviews from active Corps members, NCCC staff, and NCCC Sponsors/FEMA POCs at each of the four campuses with all three enrollment cycles - Winter Class 26, Summer Class 26, and Fall Class 27. The focus groups and interviews will occur near the end of members’ service year before they leave campus – after members’ last round of service and before member graduation or Life After AmeriCorps (LAA) day. The evaluator will schedule the qualitative data collection around the same time as the first follow-up leadership survey to keep data collection continuity and members engagement with the study. The qualitative data collection timeframe is illustrated in Figure 3. For the Winter Class 26, the qualitative data collection period is September to January of 2020; for the Summer Class 26, the qualitative data collection period is February to June 2021; and Fall Class 27 the qualitative data collection period is May to September 2121.

Figure 3 Leadership and Retention Studies Qualitative Data Collection Period

The focus groups and interviews will allow members to retrospectively assess their NCCC experiences and their development. The evaluator will conduct on-site focus groups or semi-structured interviews with active members (Corps members and team leaders), NCCC staff, sponsors, and community members/stakeholders (see Table 3).

Table 3 Data Collection by Target Group

End of Service Year

The proposed stakeholder groups will provide a comprehensive view of member leadership development as they would be able to share their perspectives and experiences with NCCC and members, and thus could directly speak about the training their organization provide their members to grow leadership skills and succeed in their service projects.

Active NCCC Corps Members: Participants for the member focus groups will include Traditional NCCC and FEMA Corps members. The focus group will discuss members perceptions on how NCCC, sponsor organizations and FEMA program contribute to their skills development, and what can these programs do to better support their development. Member discussions will also provide valuable insight on the factors that motivate and retain members to remain in the NCCC program.

The evaluator will conduct the focus groups at campuses soon after completing their last round of service which will allow members to reflect on their accomplishments and growth stemming from their entire term of service not simply the completed projects following a round.

NCCC Team Leaders: Team leader participants will be from the same Class as the active Corps members. The focus group will take a closer look at how NCCC supports team leaders with their skill developments. The focus group will also explore how team leaders perceive their role to support members’ leadership skills and maximize member retention.

NCCC Staff - Unit Leaders and Assistant Program Department Staff: Participants in the staff focus groups will include Unit Leaders who have oversight of members, team leaders, and staff that are responsible for member programming at their campus. This group will provide perspectives on member training that supports leadership skills and how the training maps to the NCCC competencies. This group will provide unique perspective on how NCCC’s programmatic structure supports member leadership qualities and skills, support members in strengthening communities through their service projects, and retaining members through service completion.

Sponsors/FEMA Points of Contact (POCs): Sponsors/FEMA POCs are the sponsoring organizations and the supervisors that oversee Traditional and FEMA teams. This group will provide insights on the training and resources members received to support their leadership qualities and skills, and how the service experience members have with their organization will inspire members to remain in service, and support members in strengthening and impacting the communities they serve.

Each focus group will consist of four to five participants. The smaller groups will allow participants to explore group dynamics (such as consenting or dissenting views) and give the moderator opportunities to probe with follow-up questions. Tables 4 and 5 list the suggested number of focus groups and interviews that would yield adequate information to compare groups within and across campuses, while being cost-effective.

NCCC Corps Members: two in-person, on-campus focus groups with members. If campus has FEMA projects, conduct one focus group with FEMA members and one with Traditional members. Follow-up telephone interviews will be conducted with up to 10 members after 1 or 2 months of completing the program.

NCCC Team Leaders: two in-person, on-campus focus groups with members. If campus has FEMA projects, conduct one focus group with FEMA members and one with Traditional members. Additionally, 10 follow-up telephone interviews will be conducted with Team Leaders after 1 or 2 months of completing the program.

NCCC Staff: one in-person, on-campus focus group with Unit Leaders and Programming Department staff.

Sponsors/FEMA POCs: four in-person interviews, web-based phone interviews, or telephone interviews with Traditional NCCC Site Supervisors and FEMA POCs if applicable.

Table 4 Focus Groups or Interviews by Target Group, Campuses with only Traditional NCCC

Traditional NCCC

*5 participants per Member and Team Leader focus groups; 4 participants in NCCC Staff focus group

Table 5 Focus Groups or Interviews by Target Group, Campuses with Traditional NCCC and FEMA Corps

Sponsors/FEMA POCs

The on-campus focus group discussions will be 90 minutes; and interviews (in-person, web-based phone calls, or telephone calls) will be 30 minutes. The suggested time allows the evaluator to engage and discuss the questions with the participants, as well as explore themes that emerged during the discussions or interviews.

Analysis Descriptive Analyses Prior to conducting any impact analyses, the evaluator will develop descriptive analyses for treatment and matched comparison group participants based on data from the baseline survey. These analyses will provide an overview of the baseline background and demographic characteristics of treatment and matched comparison group participants, and outcome characteristics prior to the start of service.

The results will provide an overall characterization of NCCC members and their matched comparison cases. These analyses will also provide evidence on whether the matching approach was effective in identifying comparison cases that are observationally equivalent to the treatment cases. To show that matching was effective, the study will use comparisons of variable means and standard deviations between the treatment and the comparison group.

Additionally, the evaluator will produce descriptive statistics of treatment and matched comparison group outcomes at the end of the program participation period (first follow-up) and one year after the end of program participation period (second follow-up). Descriptive analyses of these measures will enable the evaluator to observe patterns in leadership qualities and skills, as measured by professional skills, life skills, team behavior, community and civic engagement.

Impact Analyses To estimate the impact of NCCC on members’ leadership qualities and skills, the evaluator will examine outcome differences between the treatment group and the comparison group. To estimate the impact with increased statistical efficiency, the evaluator will use multivariate multilevel regression models using the post-match sample, controlling for baseline characteristics. The dependent variable in this model will be each participant’s leadership qualities and skills measured by professional skills, life skills, team behavior, community and civic engagement. The parameter constituting the ‘impact estimate’ in this model will be the regression-adjusted treatment effect of NCCC participation on each outcome of interest.

It is anticipated that the outcome measures will likely not be normally distributed across participants. There are several regression models that can be explored such as Poisson, negative binomial or logistic, and adjustments can be made to the model estimation procedures such that the model makes fewer assumptions of outcome distribution normality.

Thematic and Content Analysis The quantitative analysis will be triangulated with the focus groups and semi-structured interview data. One of the major strengths of qualitative research is that it provides an excellent way to explore the range of understandings and themes by including both those that are common and those that are rare. Understanding the range of responses is particularly important for future NCCC research. The evaluator will use a thematic and content analysis that focuses on the research questions.

Concepts, categories, and assumptions will be continuously compared and refined as new data, findings and themes emerge.[footnoteRef:2] The analyzing of multiple data sources to answer whether and how members develop leadership qualities and skills will enable the evaluator to triangulate the data across instruments within stakeholder groups and compare and contrast across stakeholder groups and campuses. [2: Delbert C. Miller, Neil J. Salkind SAGE Publications, Jan 16, 2002]

The evaluator will analyze the focus groups and semi-structured interviews data collected from members, NCCC staff, and project sponsors and FEMA POCs to determine what factors contributed to leadership development in members while serving with NCCC. The narrative responses gathered from members will be used to complement the outcomes from the quantitative leadership surveys. This approach offers more context to better interpret the quantitative findings and allows for statistical findings to be considered in understanding the nuances of the qualitative findings. Moreover, it provides members and other study participants a voice and grounds study findings in participants’ experiences.

Addressing non-response At baseline the evaluator will enroll members and comparison group into the study. CNCS expects that all incoming members will participate in the study, but that 30 percent of the comparison group will accept the invitation to participate in the study. The evaluator will use available application data to determine whether the comparison group who accept to participate differ significantly from the those who decline to participate on all applicants. It is not anticipated that those who participate will differ in any significant way from those who choose not to participate. The lower acceptance rate to enroll will likely be the result that the comparison group is disinterested in participating in a study about the impact of NCCC given that they decided, for whatever reason, not to serve with that program. The evaluator will apply matching methods to establish baseline equivalence between treatment and comparison group respondents which should minimize any perceived bias in the differential enrollment rate between the two groups.

Non-response at the follow-up surveys Even with the most aggressive and comprehensive retention efforts, there is a possibility of attrition in that some respondents that participated in the baseline survey may not be found for the follow-up (i.e., the respondent moved abroad) or the respondent, even if found, may decline to participate in the follow-up. All completion rates at follow-up are conditioned on the respondent having been a baseline respondent, since follow-up data collection will be attempted only for those who responded at baseline. If the completion rate is less than 80 percent at either the first follow-up or second follow-up survey, we will conduct a non-response bias analysis to determine whether the follow-up respondents differ significantly from the baseline treatment and comparison group members that enrolled in the study, and whether attrition differs for the two groups.

If necessary, we will calculate a non-response weight adjustment. The evaluator will compute survey weights that account for differential non-response at each round of data collection. Follow-up survey weights will account for both the propensity to participate in the baseline survey and the follow-up. The follow-up survey weights will be used in analyzing the combined baseline and follow-up data. Analyses using the weights will be representative of the original study population. The evaluator will assess whether it is necessary to make weight adjustments for additional missing data arising from the matching process.

Since almost all questions on the survey are going to be required of respondents, item nonresponse will likely be very minimal, and will result primarily from early termination of the survey. These will be classified as incomplete, but included, cases and non-sufficient responses. We will assess whether both partial survey completion and answers to survey items differ significantly by survey modality (online or phone survey). The evaluator will use multivariate logistic regression for the analysis of response completion by mode and logistic, multinomial, and Ordinary Least Squares (OLS) regressions for the analysis of item response. The analysis used will be determined by the nature (i.e., binomial, categorical with more than two groups, continuous) of the survey item in question. The models will be multivariate, controlling for individual demographic characteristics. Estimates of how item response changes based on response mode can also be calculated, substituting different response conditions (e.g., online versus other formats) for ‘response’ and ‘nonresponse’ categories.

Missing Data For phone interviews or online surveys, respondents may decline to answer some questions and some data will be missing in the final analysis dataset. If any item response rate is less than 70%, the evaluator will conduct an item nonresponse analysis to determine if the data are missing at random at the item level.

Cases in the matched dataset with missing data will be included in the final analysis using Full Information Maximum Likelihood (FIML), or, if it becomes necessary due to sample size and outcome distribution, Bayesian estimation. This technique is preferable to listwise deletion, which has strong potential for producing biased parameter estimates. Multiple Imputation, which is another alternative to FIML, can be cumbersome to employ when data are continually being collected and datasets are being managed and adjusted. Cases with missing values on the outcomes will be included in the analysis.

Member Retention Study This section describes the impact quasi-experimental design for the member retention study. This section includes a synopsis of the literature on retention of youth in national service programs, the research questions guiding the study, participants enrollment, data collection and proposed analysis.

Literature Review JBS conducted a review of the literature on the factors that contribute to young adult retention in national service. The literature review identified motivations of young adults to work in service programs, highlighting both altruistic and instrumental motivations in the decision to volunteer and the retention of members. There is substantial evidence to indicate that members are driven to volunteer out of a desire to acquire career-related benefits, experience social benefits, and have meaningful and substantive experiences. Motivations may be either altruistic and value-based or instrumental but should be assessed during screening to ensure that the service experience can be properly calibrated to member expectations and be able to address members’ needs and desires, which will contribute to retention.

There are varied factors that contribute to retention and could be difficult to identify. Some of these factors include proper recruitment, selection, training, effective volunteer management, strong social networks, and the members’ satisfaction with tasks and roles. Although effectively navigating these factors may lead to increase member retention, it is important to note there is still the risk of members experiencing burnout at some point during their service. Burnout may play a significant role in attrition and identifying symptoms in NCCC members may be a useful way to assess early warning signs of dropout. In doing so, NCCC could take steps to address the causes of members burnout and potentially reduce attrition.

The literature review documented best practices, such as: screening volunteers to assess interests and risks; defining roles and expectations; providing effective training and orientation, along with ongoing training opportunities; creating a welcoming environment for volunteers; enhancing the meaningfulness of volunteer work; involving volunteers in decision-making; and providing volunteers with feedback and recognition. Though some best practices in volunteer management were identified in the literature, there is still relatively little knowledge available about the specific practices that contribute to volunteer retention, particularly with a focus on the retention of NCCC members.

Research Questions The retention study is guided by the following research questions:

1. What is the profile of NCCC members who remain in service, including their demographic characteristics, sources to learn about NCCC, and motivations for serving compared to members who leave? How do these members perceive the NCCC experience prior to the start of their service compared to members who leave?