Draft IDIQ PWS.docx

DRAFT – Indefinite Delivery-Indefinite Quantity Performance Work Statement

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation

DRAFT

Indefinite Delivery-Indefinite Quantity (IDIQ) Performance Work Statement (PWS)

EVALUATING CAREER PATHWAYS PROGRAMS AND STRATEGIES

I. INTRODUCTION

The U.S. Department of Education’s (ED) Institute of Education Sciences (IES) requires one or more contractors to assist IES in conducting evaluations of career pathways programs and strategies. These programs and strategies are designed to improve the career outcomes of youth and adults—including those with disabilities—and can include career and technical education (CTE), adult education (AE), and other educational programs with career exploration, career development, workforce development and/or related activities and supports. These evaluations may use new data collection and/or existing data sources to: (1) describe the implementation and costs of federally-funded career pathways programs and strategies; (2) analyze trends in participation and outcomes in federally-funded career pathways programs and strategies; (3) describe the characteristics and activities of schools, providers, districts, postsecondary institutions, and states that are implementing career pathways programs and strategies; (4) examine the effectiveness and cost effectiveness of policy-relevant career pathways programs and strategies through systematic evidence reviews, and demonstration or low-cost opportunistic impact evaluations; and (5) identify and describe the core components of policy-relevant career pathways programs and strategies, including the conditions necessary to achieve impact for key “high-need” youth and adult populations. The program office responsible for the evaluations will be the National Center for Education Evaluation and Regional Assistance within IES, working in collaboration with other ED offices, such as the Office of Career, Technical, and Adult Education (OCTAE) and with other centers within IES.

A. Legislative Authority

Section 114(d)(2) of Public Law 115-224 (Strengthening Career and Technical Education for the 21st Century Act of 2018, also referenced here as Perkins V[footnoteRef:2]) requires a series of independently-conducted evaluation and research initiatives to assess CTE programs under the law. Perkins V also requires the Department to submit evaluation reports to Congress. [2: https://www.govinfo.gov/content/pkg/COMPS-3096/pdf/COMPS-3096.pdf ]

Section 242(b)(4) of Public Law 113-128 (Title II of the Workforce Innovation and Opportunity of 2014, also referenced here as WIOA[footnoteRef:3]) mandates independently-conducted evaluation and research initiatives to assess adult education programs under the law. WIOA requires the Department to conduct a national evaluation at least once every four years. [3: https://www.govinfo.gov/link/plaw/113/public/128 ]

Section on Rehabilitation Services of Public Law 117-328 (the Consolidated Appropriations Act of 2023[footnoteRef:4]) authorizes the Department to allot certain funds to states for innovative activities aimed at increasing competitive integrated employment for youth and other individuals with disabilities, with the program known as the Disability Innovation Fund (DIF). The Act also authorizes evaluations of DIF grantee activities and the DIF program, as well as studies of outcomes for students receiving services and supports under both the Individual with Disabilities Education Act and Titles, I, V, and VI of the Rehabilitation Act. This authority is expected to continue in subsequent appropriations acts or be codified in reauthorization of the Rehabilitation Act or Title III of the Workforce Innovation and Opportunity Act (WIOA), where funds are provided to states to support vocational rehabilitation services for individuals with disabilities. [4: https://www.congress.gov/117/plaws/publ328/PLAW-117publ328.pdf ]

The Education Sciences Reform Act of 2002[footnoteRef:5], Public Law 107–279, established IES as the unit within ED to evaluate federal education programs and practices. [5: https://www.govinfo.gov/content/pkg/COMPS-747/pdf/COMPS-747.pdf ]

B. Background

A critical workforce challenge in the U.S. is the skills gap, particularly among jobs that require either a high school diploma, postsecondary certificate, or associate’s degree.[footnoteRef:6] Jobs requiring these “middle skills” outnumber the adults in the workforce who possess them, and this gap presents a barrier to economic mobility for many Americans. There are 30 million jobs in the United States that do not require a bachelor’s degree that pay median earnings of $55,000 or more.[footnoteRef:7] Additionally, over half of all adults in the labor force reported that it is essential to obtain training and develop new skills throughout their career to keep up with the changes in the workplace.[footnoteRef:8] [6: Burrowes, J., Young, A., Restuccia, D., Fuller, J., and Raman, M. Bridge the Gap: Rebuilding America’s Middle Skills, 2014. Retrieved from : https://www.hbs.edu/competitiveness/Documents/bridge-the-gap.pdf] [7: Georgetown University Center on Education and the Workforce, Good Jobs That Pay without a BA: A State-by-State Analysis, 2017. Retrieved from: https://goodjobsdata.org/wp-content/uploads/Good-Jobs-States.pdf] [8: Pettey, Richard. It’s Time to Rethink Career and Technical Education. Homeroom: The Official Blog of the U.S. Department of Education, December 19, 2018. Retrieved from: https://blog.ed.gov/2018/12/time-rethink-career-technical-education/]

To prepare for the future, youth and adults can take various career pathways through educational and workforce development systems. These career pathways provide an important avenue for youth and adults, including those with disabilities, to gain marketable skills through secondary, postsecondary, and/or adult education and through industry-recognized credentialing programs. Policymakers have long emphasized the importance of these pathways to an educated and skilled workforce as essential to economic mobility and American competitiveness.

As such, key federal education laws pertaining to career pathways, most recently reauthorized through Perkins V and Title II of WIOA, aim to improve the academic and technical skills of youth and adults to help prepare them for careers in high-skill, high-wage, or in-demand occupations and for lifelong learning.

CTE represents an important career pathway strategy and plays a vital role in the education-to-workforce pipeline in the United States. Annually, more than 8 million secondary students and over 3 million postsecondary students participate in CTE programs nationwide, with over $1.4 billion in federal funds invested in CTE in fiscal year 2023. Through Perkins V, educators are expected to help secondary and postsecondary students develop a set of academic, technical, and employability skills for entry into high-skill, high-wage, or in-demand careers.

Similarly, Title II of WIOA authorizes programming to support workforce development, but for adults who lack basic literacy or language skills and/or who lack a high school diploma. Annually, approximately one million learners participate in federally-funded adult education programs nationwide, with over $700 million in federal funding invested in fiscal year 2023. Funding is intended to support programs and services designed to help adults obtain the education credentials, knowledge, and skills necessary for employment and economic self-sufficiency.

Collectively, these federal programs are aimed at improving career outcomes for America’s youth and adults. National evaluations of the policies and strategies they promote will provide much-needed information regarding how they are implemented, who they serve, and the extent to which they may impact education and workforce outcomes.

II. OBJECTIVES

This contract will be an Indefinite Delivery-Indefinite Quantity (IDIQ) vehicle. The IDIQ structure is intended to help IES design and conduct evaluations that take advantage of new initiatives in the field, emerging research being conducted by others, and the need to report to Congress on a regular basis. By strategically awarding task orders to one or more contractors with appropriate content area, data collection and methodological expertise, IES expects to reduce the time and costs involved in starting up separate studies and to better link across them. Specifically, IES seeks the following:

· Access to one or more contractors that each possesses wide-ranging knowledge of career pathways policies (federal and local), programs, and strategies intended to improve the credential attainment and workforce outcomes of youth and adults. To be able to respond to emerging evaluation needs quickly, the IDIQ must have experts already familiar with these likely policies, programs, and approaches.

· The ability to tap into broad expertise in evaluation methods, including cutting-edge statistical approaches, and to conduct evaluation research aligned to IES’ Standards for Excellence in Education Research (SEER). Tasks under this IDIQ might require complex sampling for nationally-representative surveys, rigorous analytic strategies for random assignment or regression discontinuity impact evaluations, predictive analytics, processing of natural language and text data, and other innovative methods for collecting, processing, and analyzing policy data and measuring intervention effects. Tasks under this IDIQ will also require attending to the most important dimensions of high-quality education research, such as measuring the implementation fidelity, context, and/or resource costs of an intervention, or focusing an impact study on a limited number of meaningful outcomes. A contractor or team with this full set of skills is important because IES may want to “nest” or interconnect studies to increase their relevance – for example, evaluating the effectiveness of a particular strategy in the context of how commonly that strategy is used across the country, based on survey data collected from respondents or data extracted from policy documents.

· Access to a contractor or team with the capacity to both design and conduct multiple large-scale evaluations that use differing methods at the same time, and to start up evaluation activities within a month of task order awards. This capacity is important because successfully interconnecting studies may, for example, require a large implementation study and randomized controlled trial to be conducted simultaneously or in rapid succession.

· Access to a contractor or team with skills in communicating to a wide range of audiences through a variety of different evaluation products. This includes not only IES’ new report format (see, for example, https://ies.ed.gov/ncee/pubs/2023003/ and https://ies.ed.gov/ncee/pubs/2023001/, an impact and an implementation report, respectively), but also shorter data-oriented Snapshots (for example, https://ies.ed.gov/ncee/pubs/2021007/), “how-to” guides where evaluations show strategies are effective (for example, https://ies.ed.gov/ncee/pubs/2022001/index.asp), and other documents that extend the return-on-investment in the studies.

The IDIQ shall have an ordering period of 60 months. All task orders shall begin within the 60-month ordering period, but may have a duration that extends for up to 60 months beyond the end of the ordering period. Task orders may be issued as Firm Fixed Price, Time and Materials, Cost Reimbursement, or Cost Plus Fixed Fee contracts. Each task order will include one or more tasks related to designing and implementing an evaluation study. The scope of each task order will depend on IES’ needs, including which activities are appropriate for each desired study or stage of the study. For example, some task orders may include only design work, while other task orders may include external data collection activities.

III. SCOPE OF WORK

The scope of work performed under this contract shall be defined by the following tasks and/or subtasks. Each task order specified by IES shall include one or more of the tasks or subtasks, combined in various ways, to accomplish needed evaluation studies.

Task 1—Meetings

The contractor shall undertake activities to assist IES in managing the project successfully and fostering communication between IES and contractor staff.

· Initial meeting to discuss details regarding the tasks, scheduling activities, and other issues related to the IDIQ or a particular task order. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to them.

· Additional meetings to discuss the progress of contract activities, including all current task orders, and any issues or anticipated problems and proposed solutions to those problems.

· Study briefing to solicit feedback from stakeholder groups on study plans or to summarize interim or final results. Stakeholder groups may comprise, for example, ED program staff or grantees.

Task 2—Technical Working Group (TWG)

The contractor shall undertake activities to gather advice and feedback from a TWG on an evaluation of career pathways policies, programs, or strategies.

The role of the TWG will be to advise the contractor and IES on an individual task order or collection of task orders that address a similar topic. The TWG advises by providing feedback and suggestions to the contractor and IES, typically as part of meetings where the contractor presents evaluation plans or results and seeks TWG input in real time, or by providing written feedback after a review of materials sent by the contractor. Each task order awarded under this IDIQ will specify whether a new TWG will be established, for example, if new types of expertise are needed, or a previously established TWG can be used to provide ongoing advice on the new work.

· Establish the TWG by proposing to the COR a set of primary and alternate members for each TWG panel. Each panel shall be comprised of eight to ten individuals that, together, have expertise as specified in a task order. The contractor shall evaluate the potential for conflicts of interest by assessing whether any individual considered for inclusion on the TWG has a bias or financial interest in the outcome of the study or study tasks. These conflicts of interest can be perceived or actual, and are described in a form that IES provides to the contractor. The contractor shall have TWG members complete conflict of interest forms before appointment to the TWG as consultants.

· Convene TWG Meeting. The first meeting of a TWG typically involves discussion of issues related to the design of the study or study tasks. Subsequent TWG meetings may cover different topics, such as methodological and policy reactions to preliminary study findings, as specified in a task order. Meetings may include all TWG members or may involve a subset of TWG members to discuss a narrow issue related to a task order that requires more immediate and targeted input or is an opportunity to update the TWG on new activities.

Task 3— Feasibility and Design

The contractor shall undertake activities to assist IES in determining the key parameters of an evaluation of career pathways policies, programs, and strategies.

· Generate and propose study design options. As specified in a particular task order, the contractor, or the contractor in consultation with the COR, shall develop and propose ideas and design options for evaluation studies that are organized around research questions important to policymakers. Proposed studies may, for example, describe the implementation of policy-relevant career pathway programs; or test the relative effectiveness of two or more such programs; or examine the impact of a program or a strategy on an outcome with practical significance to policymakers. Evaluation studies should reflect and test hypotheses that address whether key federal policy objectives are being realized.

To generate study ideas and design options, the contractor may gather information, as appropriate, such as through interviews with federal, state, and local educators and administrators, with federal grantees, and with program and strategy providers. The purpose of the interviews may be to, for example, obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites. This activity may also generate ideas for new data collection(s) requisite for shedding light on important issues in the area of career pathways. To propose a study, the contractor shall use NCEE’s standard report planning template to indicate the topical focus, the overarching research question to be addressed in the report, and why it might be important to policymakers seeking to improve career pathways or career outcomes for youth and adults.

· Feasibility assessment for effectiveness or outcomes studies. As specified in a particular task order, the contractor in consultation with the COR, shall identify one or more policy-relevant programs and strategies, and assess the feasibility of examining the strategies through a random assignment, regression discontinuity design, other quasi-experimental, or descriptive outcomes evaluation. A program and strategy may be a particular service or approach to career pathways that is allowable under and, preferably, newly emphasized in, federal policy and have some suggestive evidence of effectiveness.

To conduct the feasibility assessment, the contractor shall gather information, as appropriate, such as through interviews with federal, state, and local educators and administrators, with federal grantees, and with program and strategy providers. The purpose of the interviews may be to, for example, obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites. This type of exploration may inform, for example, the feasibility of obtaining sufficient data to conduct a regression discontinuity design or the likely validity of such a design. Following collection and analysis of this information, the contractor shall develop options regarding the study design methods, outcomes, sample size (through statistical power calculations), and data collection approach.

· Identify and define the treatment for effectiveness or outcomes studies. As specified in a particular task order, the contractor in consultation with the COR shall identify and define one or more policy-relevant career pathway programs or strategies for possible inclusion in an impact or outcome study. A program or strategy may be a particular service or approach that is allowable under and, preferably, newly emphasized in, federal policy and have some suggestive evidence of effectiveness.

To identify and define the treatment for an effectiveness or outcome study, the contractor shall gather information, as appropriate. For example, the contractor may conduct interviews with federal, state, and local educators and administrators, with federal grantees, with program and strategy providers, and relevant content experts. The purpose of the interviews, for example, may be to obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites. The contractor may also conduct a broad or targeted review of relevant literature to collect information on the evidence of effective or promise of various programs and strategies. Information may be gathered and analyzed, as appropriate, from existing repositories.

This type of exploration may inform, for example, the selection and definition of one or more programs and strategies for inclusion in an effectiveness or outcome study. Following collection and analysis of this information, the contractor shall develop a description of relevant programs and strategies, their policy relevance, intended implementation, training requirements, and evidence of effectiveness or promise.

· Survey sample design. As needed, the contractor shall assist ED in determining the design options for one or more nationally-representative surveys. In consultation with the COR, the contractor shall identify feasible and policy relevant stratification approaches and the number of local schools, districts, postsecondary institutions, providers; teachers, faculty; employers; states; or other type of respondent needed to fulfill the objective of the survey, as specified by the COR.

· Study data collection and research plan. The contractor shall develop a plan for addressing the research questions for one or more studies, as specified by the COR through a task order. A plan shall first describe the information topics, domains, and constructs that are necessary to achieve the objectives of the study and, at a high level, how that information might be obtained and analyzed. Such a plan must be written in a way that it could be included in an Office of Management and Budget (OMB) Forms Clearance package and, in the case of effectiveness studies, a pre-registered research plan. A plan shall also then provide an outline for the intended report(s), which clearly specify how each section of the report will be structured to ensure that each research question is answered in turn. The contractor shall use this outline to create a detailed data collection instrument crosswalk, modeled after NCEE’s standard report planning template, which lays out what specific data are needed to support each part of the outlined report.

· Data collection instruments. The contractor shall develop study intake forms, survey questionnaires, data extraction forms or instructions, outreach or recruitment letters, or other instruments, as specified by the COR, to be used in conducting a study under one or more task orders in this contract. These instruments shall be developed in accordance with the data collection plan, and specifically the report plan (crosswalk) developed in the previous bullet. The instruments shall minimize burden on respondents while ensuring that key data items are collected.

· Select intervention provider(s). For effectiveness studies of interventions, programs, or strategies, one or more external vendors may be needed to provide support for implementation or technical assistance to study sites. In these cases, the contractor, in consultation with the COR and as specified in a task order, shall establish an expert panel to advise on selection of providers, announce the provider opportunity widely including a request for proposals (RFP), collect the submitted proposals, and support the expert panel in reviewing and recommending providers for the specified role in an evaluation study.

Task 4—Systematic Evidence Review

The contractor shall undertake activities to assist IES in reviewing the available evidence on key career pathways policies, programs, and strategies.

The contractor shall conduct an evidence review following the procedures and standards used by the WWC, including the use of WWC-certified study reviewers. IES will validate certification by cross-checking proposed study reviewers with WWC’s database of certified reviewers. The work shall include determining policy-relevant categories of career pathway strategies to examine, developing a review protocol, searching the literature to identify relevant studies, screening in eligible studies, coding eligible studies, assessing the methodological quality of each eligible study, analyzing the results of eligible studies that meet WWC standards or meet standards with reservations, and synthesizing the results across studies.

The evidence report that is the culmination of this task shall address the question of what is known about the effectiveness of one or more programs and strategies – either in one topic area or many, as specified by the COR – in improving key outcomes such as educational attainment, occupational skills, or employment and earnings, among other key outcomes. The report shall include a description of the methodology of the review and the state of the literature on career pathway strategies (number of studies identified in the scan for particular outcome measures, the number that meet WWC evidence standards). Most importantly, the report shall summarize the evidence from the studies meeting WWC standards and meeting WWC standards with reservations and identify gaps in the literature.

· Create an evidence review plan. Prior to conducting the review, the contractor shall submit a proposed plan for conducting the evidence review. The evidence review plan shall include parameters for the systematic evidence review, such as databases and other sources that the contractor will consult, search terms the contractor will use, and any limitations that will be imposed (e.g., in terms of publication years). The evidence review plan shall also include the contractor’s plans for systematically compiling the evidence base in an evidence report (e.g., report outline, proposed tables shells, intervention summary sheets, appendices).

· Conduct the review. Using the final evidence review plan, the contractor shall conduct the review and submit a draft evidence review report. The report length will be specified in the particular task order, but is expected to be no more than 15 pages, with information on the technical aspects of the review and full documentation of the interventions, data, analytic methods, and results of studies included in appendices. The contractor shall also include a one-page summary of the evidence report that contains enough information to form the basis for a press release.

The evidence review report will undergo review from IES’ Standards and Review Office (SRO). The contractor shall be responsive to any SRO comments and make revisions to the report until it clears SRO. The contractor shall provide a web-ready (i.e. 508-compliant), final report to the COR after clearing SRO.

Task 5 – Data Collection Approvals and Documentation

The contractor shall undertake activities to obtain relevant approvals prior to data collection.

· Prepare OMB forms clearance packages, document Institutional Review Board approval, and pre-register research plan. The contractor shall prepare a forms-clearance package in order to receive regular approval from OMB for data collection. The contractor shall ensure that the clearance package follows the required OMB format, justifying the necessity for collecting data and comprehensively responding to each required item in the instructions, including: (1) the objectives of the study or collection of studies, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, (7) limitations of the study(ies) and their potential use for policy decisions, and (8) table shells indicating how data will be tabulated and analyzed.

All forms clearance packages shall be brief, clear, and well written. Each package shall include the forms and instructions that will be used to obtain data or recruit respondents. The COR will submit the final package to begin the OMB approval process. During the initial 60-day public comment period, the contractor shall pretest the data collection forms so that burden information can be updated in the final package. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process.

The contractor shall obtain approval from its Institutional Review Board (IRB) for the evaluations conducted under any task order. The effort to obtain initial approval and to update approval as new task orders are issued should be conducted in conjunction with OMB clearance package efforts.

The contractor shall pre-register the research plan for any effectiveness evaluations conducted under a task order, once the recruited sample sizes are known. The registry shall be chosen by the contractor with approval from the COR. An example of an acceptable registry is the Registry of Efficacy and Effectiveness Studies (REES).

· Prepare OMB forms clearance packages for an expedited review. The contractor shall prepare a forms-clearance package in order to receive expedited approval from OMB under certain conditions when IES plans to collect preliminary or exploratory information to aid in study design. The contractor shall ensure that the clearance package follows the required OMB format for expedited reviews, justifying the necessity for collecting data and comprehensively and concisely responding to each required item in the instructions, including: (1) the objectives of the study or collection of studies, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, and (7) limitations of the study(ies) and their potential use for policy decisions.

The package shall be developed to obtain OMB approval using IES’ Generic (expedited) Clearance vehicle, which requires less documentation and results in approval within roughly 15 days in comparison to a typical information collection request submitted through the regular OMB clearance process. All forms clearance packages shall be brief, clear, and well written. Each package shall include the forms and instructions that will be used to obtain data for ED internal decision-making. The COR will submit the final package to begin the OMB approval process.

The contractor shall obtain approval from its Institutional Review Board (IRB) for the evaluations conducted under any task order. The effort to obtain initial approval and to update approval as new task orders are issued should be conducted in conjunction with OMB clearance package efforts.

Task 6 – Outreach and Site Recruitment

The contractor shall develop and implement a strategy for identifying and recruiting schools, districts, providers, grantees, or other type of participant to meet the sample size requirements specified in a task order.

As part of the recruitment process, the contractor shall prepare notification letters and information packets for the prospective study sites. The notification packet shall include a non-technical brochure describing the study that is suitable for distribution to a broad audience of policy makers and educators. The contractor shall include in the letters and information packets general information on the study as well as specific information on the importance of the study, its purposes, products, timelines, and the expected benefits to be derived from participation in the study. Recruitment may occur through mailings, telephone calls, in-person visits, or discussion with potential sites at key conferences or meetings, as specified in the task order.

The contractor shall submit an outreach and/or recruitment plan, draft letters, and all other notification materials to the COR for review. Once approved by the COR, the contractor shall send the notification materials to potential sites and execute the recruitment plan. The contractor shall consult with the COR to finalize the list of sites for the study, and subsequently submit documentation of memoranda of understanding for those sites to the COR.

As part of the recruitment process, the contractor shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants.

Task 7— Intake and Random Assignment for Effectiveness Studies

The contractor may need to set up a process for enrolling participants into a study (“intake,” including obtaining consent) and for random assignment in a given task order. Intake may occur at one or more specific points in time – e.g., at the beginning of each semester or of a school year -- or on a rolling basis over an extended period. Random assignment may be of students, teachers/classroom, schools, or other units depending on the evaluation design and will be specified in the task order. Random assignment may be conducted by the contractor or in the field (at schools or other sites) to accommodate rolling intake. The contractor shall develop protocols and procedures to conduct intake and random assignment to be minimally disruptive to participants and program staff who may be involved in intake.

Task 8—Implement and Monitor the Programs and Strategies in Effectiveness Studies

The contractor shall undertake activities for conducting studies of effectiveness. These activities might be carried out under one or more task orders.

· Training and implementation support. For task orders that involve conducting an effectiveness study where the contractor or an external vendor provides technical assistance to grantees or training/implementation support to study sites, the contractor shall ensure that these supports are provided with fidelity, according to the study plans. These supports may include the provision of in-person or virtual training during the summer or throughout the implementing school year, the provision of materials (such as electronic equipment, curriculum, textbooks, technological hardware or software, etc.).

· Monitoring implementation. The contractor shall monitor implementation of the intervention. The contractor shall ensure there is an appropriate firewall, such that there is a clear separation between those who are providing and supporting the implementation and those who are monitoring the implementation.

Task 9—Data Collection

IES expects that studies conducted under these task orders will rely on a variety of data sources. These may include surveys, interviews, or focus groups of agency staff at the state or local level, parent, student, teacher, or school surveys, federal records, state records, district records, school and provider records, or records from other administrative systems, including ones that might be maintained by other agencies. Other data sources could include those collected through vendor systems, classroom observations, and site visits. For example, the U.S. Department of Health and Human Services has the National Directory of New Hires (NDNH), which is a useful source of data on employment and earnings. In addition, some task orders may call for analyzing data collected and maintained by IES’ National Center for Education Statistics (NCES). The specific data to be collected or extracted shall be specified by task order.

· Administer state education agency surveys. The contractor shall administer the relevant survey to the administrators in each state education agency whose work most closely aligns with the study topic. Either the ED program office or the contractor may determine which role(s) at the state are appropriate to sample and compile lists of these individuals. The contractor shall follow up with the universe of state administrators through mail, email, and telephone calls in order to achieve a response rate of 100 percent.

· Administer nationally-representative surveys of local educational agencies, providers, or grantees. Following outreach under task 6, as approved by the COR, the contractor shall administer the relevant survey to a sample of school district and/or provider respondents in charge of overseeing local implementation of career pathways programs and strategies. Sampled populations may include any eligible entity awarded funds under federal grant programs such as Perkins V and Title II of WIOA. The contractor shall follow up with sampled respondents through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

· Administer youth or adult surveys. The contractor shall administer approved notification and consent, as specified in the task order, and the relevant survey to youth or adults in the most efficient manner possible. Respondents may be, for example, those participating in an implementation or impact study of career pathways programs or strategies. Respondents may also comprise a nationally representative sample of the target population for these programs and strategies. The contractor shall follow up with respondents through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

· Administer school or provider surveys. The contractor shall administer the relevant survey to schools or providers in the most efficient manner possible. School or provider respondents may be, for example, those participating in an implementation or impact study of career pathways programs or strategies. Respondents may also comprise a nationally representative sample of the target population for these programs and strategies. The contractor shall follow up through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

· Administer teacher or faculty surveys. The contractor shall administer the relevant survey to teachers or faculty in the most efficient manner possible. Teacher or faculty respondents may be, for example, those participating in an implementation or impact study of career pathways programs and strategies. The contractor shall follow up through mail, email, and telephone calls to achieve a response rate of 85 percent.

· Administer parent surveys. The contractor shall administer approved notification and consent, as specified in the task order, and the relevant survey to parents in the most efficient manner possible. Parent respondents may be, for example, those participating in an implementation or impact study of career pathways programs and strategies . The contractor shall follow up with respondents through mail, email, and telephone calls to achieve a response rate of 85 percent.

· Conduct classroom observations. The contractor shall conduct live or video observations when necessary for evaluation studies of career pathways programs and strategies . The contractor shall coordinate scheduling and follow up to achieve a response rate of 85 percent.

· Collect student or adult learner records data. The contractor shall obtain records for students or adult learners using the most efficient process possible, when such records are necessary for evaluation studies of career pathways programs and strategies. These records may be obtained from state data systems (e.g., states’ longitudinal data systems) or from districts, schools, or providers directly, and include some combination of students’ or adult learners’ achievement scores on state or local assessments, demographic characteristics, course taking, grades, attendance, attainment, or participation and outcome data from the programs or strategies under study. The contractor shall follow up to achieve a response rate of 85 percent.

· Collect teacher or faculty records data. The contractor shall obtain records for teachers or faculty in using the most efficient process possible, when such records are necessary for evaluation studies of career pathways programs and strategies. These records may be obtained from state data systems (e.g., states’ longitudinal data systems) or from districts, schools, or providers directly, and include some combination of teachers’ or faculty’s evaluation scores, certification, experience, education, grade/subject assignments, retention, or demographic characteristics, as specified in the task order. The contractor shall follow up to achieve a response rate of 85 percent.

· Administer student or adult learner assessment. The contractor shall administer a student or adult learner assessment in schools or entities participating in an evaluation study of career pathways programs and strategies. Such an assessment may be necessary if existing state or adult educator provider assessments do not provide the appropriate sensitivity or focus for the intervention tested. These study-administered assessments could focus on outcomes of interest such as math or ELA conceptual knowledge, or technical skills proficiency. The contractor shall follow up to achieve a response rate of 85 percent.

· Administer teacher or faculty assessment. The contractor shall administer a teacher or faculty assessment in schools or entities participating in an evaluation study of career pathways programs and strategies. Such an assessment may be necessary if teacher or faculty knowledge on particular issues is a core outcome of the intervention being tested. These study-administered assessments could focus on outcomes such as subject-matter content knowledge or knowledge of other relevant content with special subgroups of students. The contractor shall follow up to achieve a response rate of 85 percent.

· Conduct interviews or focus groups with state/district/school/provider administrators, teachers/faculty, parents, or students. The contractor shall conduct interviews or focus groups with one or more respondent groups. The contractor shall strategically recruit, schedule, and follow up with respondents to achieve a response rate of 85 percent.

· Obtain state or local extant data and documentation. The contractor shall obtain existing records from states, local educational agencies, or other entities that are relevant to an evaluation study of career pathways programs and strategies. This could include existing documents related to subgrant competitions, identification of high-need schools or providers, technical assistance materials, lists of approved/recommended strategies, Memoranda of Understanding between institutions collaborating on service provision, professional development materials for teachers or faculty, or school/provider budgets. These existing documents should be collected using the most efficient process possible.

· Collect data from intervention providers. As specified in the task order, the contractor shall obtain relevant data from providers who support implementation of career pathways programs and strategies. Such data could come from meta-data in the provider’s system, other existing documents, or interviews or surveys of the vendors – for the purposes of providing information about the programs and strategies, including implementation and costs. These existing documents should be collected using the most efficient process possible.

· Access other administrative records. The contractor shall assist IES in setting up a Memo of Understanding with another agency or organization to obtain data on services or outcomes for students in an evaluation study of career pathways programs and strategies. The contractor shall specify the data to be extracted and, if the data will be transferred to the contractor, arrange for secure transfer, or use the data on site at the other agency/organization.

Task 10—Data Analysis and Reporting

The contractor shall undertake activities to analyze data and report on policy-relevant findings of an evaluation of career pathways policies, programs, and strategies.

· Analysis plan. The contractor shall update the study data collection and research plan (report plan/crosswalk) developed in Task 3, as specified in the task order, which laid out what specific data are needed to support each part of the planned report(s). The contractor shall then develop an analysis plan that further details how each of the elements in the report plan/crosswalk will be carried out, including describing the study design, and identifying the data sources, key measures, and intended data analyses that will be included in the evaluation’s report products, as specified in the task order. The plan shall include table shells and, if relevant, the statistical power for measuring impacts on the planned outcomes with the achieved sample size. The plan shall specify any sensitivity tests the contractor plans to conduct.

· Evaluation report. The contractor shall conduct data analyses and prepare a report describing what is being studied and the results of the analysis. The contractor shall borrow the introduction, study design description, and data sources from the final analysis plan, rather than developing this from scratch, to the extent that it meets the COR’s approval.

The contractor shall make sure that the report is written and organized so that it is accessible to policy makers and research-savvy practitioners. The report text length is expected to be no more than 15 pages, with information on the technical aspects of the study included in appendices. To the extent feasible, the report shall include visually intuitive figures and graphics rather than tables. The contractor shall also include a one-page, stand-alone summary of the report that contains enough information to form the basis for a press release.

Both the draft report and one-page summary will be submitted by the COR for approval through IES’ Standards and Review Office (SRO). The SRO process includes review by two external researchers with expertise in the relevant topic. The contractor shall be responsive to SRO’s comments and make revisions to the report until it clears SRO. After the report and 1-pager are cleared by SRO, the contractor shall provide web-ready (i.e. 508-compliant) versions of the report and one-pager.

· Internal study brief. The contractor shall conduct data analyses as outlined in an approved plan. The contractor shall prepare a brief that includes mostly exhibits (figures, graphs, tables) and bullet points of key findings. The text of the brief shall be no more than 15 pages long but the set of exhibits the text draws upon may be significantly longer.

· Study snapshot. The contractor shall conduct data analyses as outlined in an approved analysis plan. The contractor shall prepare a snapshot that includes 3-5 main headlines with a set of supporting bullets. Each headline should be accompanied by one or two exhibits (figures, graphs, tables). The snapshot is meant to provide a quick first look at data, with minimal text and details, as compared to an evaluation report. The main body of the snapshots are typically less than 15 pages.

The snapshot will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the snapshot until it clears SRO. After the snapshot is cleared, the contractor shall provide web-ready (i.e. 508-compliant) versions of the final snapshot.

· Compendium of data tables. The contractor shall develop a comprehensive set of data tables based on a given data collection to supplement the analyses and findings featured in an evaluation report. The contractor shall present the tables in an organized manner and display them using a reader-friendly format.

The compendium will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the compendium until it clears SRO. After the compendium is cleared, the contractor shall provide web-ready (i.e. 508-compliant) versions of the final compendium

· How-to guide. The contractor shall prepare a How-To guide that is geared toward practitioners. The purpose is to provide guidance on how best to implement a particular career pathways program or strategy whose effectiveness was studied and found to be promising. The guide may include, as specified by the COR, considerations of vendor selection, capacity requirements to implement, flexibilities for implementation, costs, potential pitfalls and ways to overcome them, as well as other necessary technical details. There is not a page limit for the guide, but the contractor shall balance the need for sufficient detail with the need to make the guide approachable and accessible to a practitioner audience.

The guide will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the guide until it clears SRO. After the guide is cleared, the contractor shall provide web-ready (i.e. 508-compliant) versions of the final guide.

· Data file. The contractor shall prepare data files that contain data collected for an evaluation report, with all direct personal identifiers removed. The contractor shall provide documentation for the files including variable labels, category labels, the unit of observation, weights, and how missing data were addressed.

The contractor shall also provide IES with a risk disclosure analysis plan for submission to NCES’ Disclosure Review Board (DRB). After DRB approval of the codebooks and risk disclosure analysis plan, the contractor shall submit to the COR the data files consistent with NCES standards. ASCII or SAS file formats are acceptable.

· Data management plan. For instances in which study data cannot be released in public or restricted use data files, the contractor shall develop a data management plan.

Task 11—Logistical Support for Departmental Activities Related to Synthesizing Findings

IES may wish to convene meetings or develop reports and syntheses of work for Congress or ED that relate to the topics under this IDIQ, in particular as they describe or evaluate specific grant programs…