Postsecondary IDIQ 2025 PWS draft for RFI.docx

U.S. Department of Education Performance Work Statement Institute of Education Sciences National Center for Education Evaluation

Performance Work Statement

STUDYING THE IMPLEMENTATION AND EFFECTIVENESS OF

POSTSECONDARY EDUCATION ACCESS AND SUCCESS STRATEGIES AND PROGRAMS

The U.S. Department of Education’s (ED) Institute of Education Sciences (IES) requires one or more contractors to assist IES in designing and conducting evaluations of ED’s postsecondary education access and success programs and strategies that may improve outcomes for participants in these programs.

A. Legislative Authority

Title IV Part A of the Higher Education Opportunity Act of 2008 authorizes the Secretary to conduct evaluations of ED’s postsecondary access and success programs. Section 402 authorizes the Secretary to make grants or enter into contracts to rigorously evaluate the effectiveness of strategies and practices supported under the TRIO programs. Section 404 authorizes the Secretary to make grants or enter into contracts to evaluate the effectiveness of the Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) program.

The Consolidated Appropriations Act of 2022 and subsequent annual appropriations acts authorize the Secretary to reserve up to 5% of fiscal year appropriations across all programs authorized under the Higher Education Opportunity Act, with the exception of the Pell grant program and Student Aid Administration, for the evaluation of any program authorized under the Higher Education Opportunity Act.

Title I Part D Section 173 of the Education Sciences Reform Act of 2002 authorizes IES to conduct evaluations of federal education programs in order to determine the impact of such programs.

B. Background

Making postsecondary education affordable and closing equity gaps in enrollment and completion rates remain key federal policy goals. Recent legislation, regulations, and directives designed to simplify access to and experiences with the federal financial aid system may be reshaping who participates and the outcomes they experience. Overall college matriculation rates declined during the years of the COVID-19 pandemic but recently increased, due largely to expansions in community college programs and dual enrollment of high school students in college courses.[footnoteRef:2] Still, the turnaround in enrollment and the interaction with federal financial aid has been uneven, with certain kinds of students and schools continuing to face challenges. [2: https://nces.ed.gov/programs/coe/indicator/cpb/college-enrollment-rate; https://nscresearchcenter.org/current-term-enrollment-estimates/]

ED has a large and varied portfolio of programs aimed at promoting postsecondary access and success, each with evaluation needs. These programs span multiple offices—including the Office of Postsecondary Education, the Office of Career Technical and Adult Education, and the Office of Federal Student Aid—serve different populations and have different emphases. IES seeks a pool of study teams with the capacity and expertise to design and conduct multiple complex evaluations of these programs simultaneously, and to help IES link across the studies in order to maximize learning.

IES has multiple ongoing partnerships to conduct evaluations of ED’s postsecondary-related programs to explore ways to improve college preparation and success, particularly for those from disadvantaged backgrounds. For instance, IES conducts evaluations to:

· Provide information that could be used to improve ED’s TRIO and GEAR UP programs. The TRIO programs are a set of eight outreach and student services programs that, together, provide assistance to students at risk from as early as middle school through the academic pipeline into post-baccalaureate programs. The GEAR UP program supports states and local partnerships of school districts and higher education institutions seeking to improve high school graduation rates and the college preparation of middle and high school students from low incomes. The TRIO and GEAR UP programs employ a wide variety of strategies including information supports, mentoring, tutoring, academic and career counseling, and trips to college campuses.

· Identify successes and challenges for participants in ED’s federal student aid programs. Each year, the Office of Federal Student Aid (FSA) provides approximately $112 billion in grant, work-study, and loan funds to help over 10 million students pay for postsecondary education.[footnoteRef:3] Exploring participation rates and outcomes of the Pell Grant program and various loan programs and policies can provide relevant information for FSA and Congress. [3: https://studentaid.gov/about/

In addition to these ongoing partnerships, other Higher Education Opportunity Act programs raise important research questions regarding how best to support postsecondary students and the institutions that serve them. This includes, for instance, programs that support minority-serving institutions, programs that promote international education, and programs that support special populations such as migrant college students and students with child care needs.

IES may design and conduct evaluations of any of these programs through task order awards under this Indefinite-Delivery Indefinite-Quantity (IDIQ) contract.

C. Objectives

This requirement will be a multiple award IDIQ vehicle. The IDIQ structure is intended to help IES design and conduct evaluations of postsecondary programs and strategies rigorously, quickly, and efficiently in order to make information available and more actionable for policymakers and practitioners. By strategically awarding task orders to one or more contractors with data collection and methodological expertise, as well as relevant content expertise, IES expects to reduce the time and costs involved in starting up separate studies and to better link across them. Specifically, IES seeks the following:

· Access to one or more contractors that each possesses wide-ranging knowledge of college access and success related policies (federal and local), programs, and strategies. To be able to respond to emerging evaluation needs quickly, the IDIQ must have experts already familiar with the kinds of interventions and settings that may be the focus of task orders issued under this IDIQ.

· The ability to tap into broad expertise in evaluation methods, including cutting-edge statistical approaches, and to conduct evaluation research aligned to IES’ Standards for Excellence in Education Research (SEER). Tasks under this IDIQ might require complex sampling for nationally representative surveys, rigorous analytic strategies for random assignment or regression discontinuity impact evaluations, or predictive analytics, processing of natural language and text data, and other innovative methods for collecting, processing, and analyzing policy data and measuring intervention effects. Tasks under this IDIQ contract will also require attending to the most important dimensions of high-quality education research, such as measuring the implementation fidelity, context, and/or resource costs of an intervention, or focusing an impact study on a limited number of meaningful outcomes. A contractor or team with this full set of skills is important because IES may want to “nest” or interconnect studies to increase their relevance – for example, evaluating the effectiveness of a particular strategy in the context of how commonly that strategy is used across the country, based on survey data collected from respondents or data extracted from policy documents.

· Access to a contractor or team with the capacity to both design and conduct multiple large-scale evaluations that use differing methods at the same time, and to start up evaluation activities within a month of task order awards. This capacity is important because successfully interconnecting studies may, for example, require a large implementation study and randomized controlled trial to be conducted simultaneously or in rapid succession.

· Access to a contractor or team with skills in communicating to a wide range of audiences through a variety of different evaluation products. This includes not only IES’ new report format (see, for example, https://ies.ed.gov/ncee/pubs/2023003 and https://ies.ed.gov/ncee/pubs/2023001/ an impact and an implementation report, respectively), but shorter data-oriented Snapshots (for example, https://ies.ed.gov/ncee/pubs/2021007/), “how-to” guides that provide implementation advice on strategies research has shown to be effective (for example, see https://ies.ed.gov/ncee/pubs/2022001/index.asp), and other documents that extend the return-on-investment in the studies.

The IDIQ shall have an ordering period of 60 months. All task orders shall begin within the 60-month ordering period but may have a duration that extends for up to 60 months beyond the end of the ordering period. Task orders may be issued as Firm Fixed Price, Time and Materials, Cost Reimbursement, or Cost Plus Fixed Fee contracts.

Each task order will include one or more tasks related to designing and/or conducting an evaluation study. The scope of each task order will depend on IES’ needs, including which activities are appropriate for each desired study or stage of the study. For example, some task orders may include only design work, while other task orders may include external data collection activities.

D. Scope of Work

The scope of work performed under this contract shall be defined by the following tasks. Each task order specified by IES shall include one or more of these tasks, combined in various ways, to accomplish needed evaluation studies.

Task 1—Meetings

The contractor shall undertake activities to assist IES in managing the project successfully and fostering communication between IES and contractor staff. These activities may take place in-person at ED offices in Washington DC or virtually, depending on the needs of the individual task orders.

· Initial meeting to discuss details regarding the tasks, scheduling activities, and other issues related to the IDIQ or a particular task order. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to them.

· Additional meetings to discuss the progress of contract activities, including all current task orders, and any issues or anticipated problems and proposed solutions to those problems.

· Study briefing to solicit feedback from stakeholder groups on study plans or to summarize interim or final results. Stakeholder groups may comprise, for example, ED program staff or grantees.

Task 2—Technical Working Group (TWG)

The contractor shall undertake activities to gather advice and feedback from TWGs convened for evaluations of postsecondary access and success policies, programs, or strategies.

The role of the TWG will be to advise the contractor and IES on an individual task order or collection of task orders that address a similar topic. The TWG advises by providing feedback and suggestions to the contractor, typically as part of meetings where the contractor presents evaluation plans or results and seeks TWG input in real time, or by providing written feedback after a review of materials sent by the contractor. Each task order awarded under this IDIQ contract will specify whether a new TWG will be established, for example, if new types of expertise are needed, or a previously established TWG can be used to provide ongoing advice on the new work.

· Establish the TWG by proposing to the COR a set of primary and alternate members for each TWG panel. Each panel shall be comprised of approximately eight to ten individuals that, together, have expertise as specified in a task order. The contractor shall evaluate the potential for conflicts of interest by assessing whether any individual considered for inclusion on the TWG has a bias or financial interest in the outcome of the study or study tasks. These conflicts of interest can be perceived or actual and are described in a form that IES provides to the contractor. The contractor shall have TWG members complete conflict of interest forms before appointment to the TWG as consultants.

· Convene TWG. The first meeting of a TWG typically involves discussion of issues related to the design of the study or study tasks. Subsequent TWP meetings may cover different topics, such as methodological and policy reactions to preliminary study findings, as specified in a task order. Meetings may include all TWG members or may involve a subset of TWG members to discuss a narrow issue related to a task order that requires more immediate and targeted input or is an opportunity to update the TWG on new activities.

Task 3— Feasibility and Design

The contractor shall undertake activities to assist IES in determining the key parameters of an evaluation of policies, programs, and/or strategies to increase postsecondary access and credential attainment.

· Generate and propose study design options. As specified in a particular task order, the contractor, or the contractor in consultation with the COR, may develop and propose ideas and design options for evaluation studies that are organized around research questions important to policymakers. Proposed studies may, for example, describe the implementation of policy-relevant college access strategies; or test the relative effectiveness of two or more such strategies; or examine the impact of a program or a strategy on an outcome with practical significance to policymakers. Evaluation studies should reflect and test hypotheses that address whether key federal policy objectives are being realized.

To generate study ideas and design options, the contractor may gather information, as appropriate, such as through interviews with federal, state, and local educators and administrators, with federal grantees, and with program and strategy providers. The purpose of the interviews may be to, for example, obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites. This activity may also generate ideas for new data collection(s) requisite for shedding light on important issues in the area of postsecondary access and success. To propose a study, the contractor shall use the National Center for Education Evaluation’s (NCEE’s) standard report planning template (Appendix A) to indicate the topical focus, the overarching research question to be addressed in the report, and why it might be important to policymakers seeking to improve postsecondary access and credential attainment for youth and adults.

· Conduct a feasibility assessment for effectiveness or outcomes studies. As specified in a particular task order, the COR, the contractor, or the contractor in consultation with the COR, may identify one or more policy-relevant program or strategies to study, and assess the feasibility of examining the strategies through a random assignment, regression discontinuity design, other quasi-experimental, or descriptive outcomes evaluation. A program or strategy may be a particular service or approach to postsecondary access and credential attainment that is allowable under and, preferably, newly emphasized in, federal policy and have some suggestive evidence of effectiveness.

To conduct the feasibility assessment, the contractor shall gather information, as appropriate, such as through interviews with federal, state, and local educators and administrators, with federal grantees, and with program and strategy providers. The purpose of the interviews may be to, for example, obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites, such as the National Student Clearinghouse. This type of exploration may inform, for example, the feasibility of obtaining sufficient data to conduct a regression discontinuity design or the likely validity of such a design. Following collection and analysis of this information, the contractor shall develop options regarding the study design methods, outcomes, sample size (through statistical power calculations), and data collection approach.

· Identify and define the treatment for effectiveness or outcomes studies. As specified in a particular task order, the contractor in consultation with the COR may identify and define one or more policy-relevant postsecondary access and success programs or strategies for possible inclusion in an impact or outcome study. A program or strategy may be a particular service or approach that is allowable under and, preferably, newly emphasized in, federal policy and have some suggestive evidence of effectiveness.

To identify and define the treatment for an effectiveness or outcome study, the contractor shall gather information, as appropriate. For example, the contractor may conduct interviews with federal, state, and local educators and administrators, with federal grantees, with program and strategy providers, and relevant content experts. The purpose of the interviews, for example, may be to obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites. The contractor may also conduct a broad or targeted review of relevant literature to collect information on the evidence of effective or promise of various programs and strategies. Information may be gathered and analyzed, as appropriate, from existing repositories.

This type of exploration may inform, for example, the selection and definition of one or more programs and strategies for inclusion in an effectiveness or outcome study. Following collection and analysis of this information, the contractor shall develop a description of relevant programs and strategies, their policy relevance, intended implementation, training requirements, and evidence of effectiveness or promise.

· Survey sample design. The contractor may assist ED in determining the design options for one or more nationally representative surveys. In consultation with the COR, the contractor may identify feasible and policy relevant stratification approaches and the number of identify feasible and policy relevant stratification approaches and the number of local schools, districts, postsecondary institutions, providers; teachers, faculty; states; or other type of respondent needed to fulfill the objective of the survey.

· Study data collection and research plan. The contractor may develop a plan for addressing the research questions for one or more studies, including presenting design options. A plan shall describe the information topics, domains, and constructs that are necessary to achieve the objectives of the study and, at a high level, how that information might be obtained and analyzed. Such a plan must be written in a way that it could be included in an Office of Management and Budget (OMB) Forms Clearance package and, in the case of effectiveness studies, a pre-registered research plan. A plan shall also provide an outline for the intended report(s), which clearly specify how each section of the report will be structured to ensure that each research question is answered in turn. The contractor shall use this outline to create a detailed data collection instrument crosswalk, which lays out what specific data are needed to support each part of the outlined report.

· Data collection instruments. The contractor may develop study intake forms, survey questionnaires, data extraction forms or instructions, outreach or recruitment letters, or other instruments to be used in conducting a study under one or more task orders in this contract. These instruments shall be developed in accordance with the data collection plan, and specifically the report plan (crosswalk) developed in the previous bullet. The instruments shall minimize burden on respondents while ensuring that key data items are collected.

· Select intervention provider(s). For effectiveness studies of interventions, programs, or strategies, one or more external vendors may be needed to provide support for implementation or technical assistance to study sites. In these cases, the contractor, in consultation with the COR and as specified in a task order, shall establish an expert panel to advice on selection of providers, announce the provider opportunity widely including a request for proposals (RFP), collect the submitted proposals, and support the expert panel in reviewing and recommending providers for the specified role in an evaluation study.

In some cases, off-the-shelf interventions suitable for study may not exist and adaptation and refinement of existing interventions may be needed prior to study. In these cases, working with external vendors or its own implementation team, the contractor shall conduct work to modify existing strategies to fit study needs. This work may require, for example, gathering information and feedback from grantees, schools, students and/or individuals with expertise closely related to the strategy and iteratively adapting a strategy such that it is promising and feasible for study. In conducting this work IES expects that the contractor will maintain a firewall between those responsible for adaptation and implementation and those responsible for collecting and analyzing study data.

· Pilot intervention. For interventions that have been adapted or that have not been implemented with the study’s intended population and/or setting, the contractors shall pilot implementation of the selected intervention(s). Piloting shall be conducted in the most efficient means possible. For instance, it should be limited to the number of participants and the scope of data collection required to achieve the objectives of the pilot test.

Task 4—Systematic Evidence Review

The contractor shall undertake activities to assist IES in reviewing the available evidence on key postsecondary access and success policies, programs, and strategies.

The contractor shall conduct an evidence review following the procedures and standards used by the WWC, including the use of WWC-certified study reviewers. The work shall include determining policy-relevant categories of postsecondary access and success strategies to examine, developing an evidence review plan, searching the literature to identify relevant studies, screening in eligible studies, coding eligible studies using the WWC’s review of individual studies protocol, assessing the methodological quality of each eligible study, analyzing the results of eligible studies that meet WWC standards or meet standards with reservations, and synthesizing the results across studies.

The evidence report that is the culmination of this task shall address the question of what is known about the effectiveness of one or more programs and strategies – either in one topic area or many, as specified by the COR – in improving key outcomes such as postsecondary enrollment, FAFSA submission or acceptance, credit accrual, and degree attainment. The report shall include a description of the methodology of the review and the state of the literature on the strategy or strategies examined (number of studies identified in the scan for particular outcome measures, the number that meet WWC evidence standards). Most importantly, the report shall summarize the evidence from the studies meeting WWC standards and meeting WWC standards with reservations and identify gaps in the literature.

· Create an evidence review plan. Prior to conducting the review, the contractor shall submit a proposed plan for conducting the evidence review. The evidence review plan shall include parameters for the systematic evidence review, such as databases and other sources that the contractor will consult, search terms the contractor will use, and any limitations that will be imposed (e.g., in terms of publication years). The evidence review plan shall also include the contractor’s plans for systematically compiling the evidence base in an evidence report (e.g., report outline, proposed tables shells, intervention summary sheets, appendices).

· Conduct the review. Using the final evidence review plan, the contractor shall conduct the review and submit a draft evidence review report. The report length will be specified in the particular task order but is expected to be no more than 15 pages, with information on the technical aspects of the review and full documentation of the interventions, data, analytic methods, and results of studies included in appendices. The contractor shall also include a one-page summary of the evidence report that contains enough information to form the basis for a press release.

The evidence review report will undergo review from IES’ Standards and Review Office (SRO). The contractor shall be responsive to any SRO comments and make revisions to the report until it clears SRO. The contractor shall provide a web-ready (i.e., 508-compliant), final report to the COR after clearing SRO.

Task 5 – Prepare OMB Forms Clearance Packages, Document Institutional Review Board Approval, and Pre-Register Research Plan

The contractor shall undertake activities to obtain relevant approvals prior to data collection.

· Prepare OMB forms clearance packages, document Institutional Review Board (IRB) approval[footnoteRef:4], and pre-register research plan. The contractor shall prepare a forms-clearance package in order to receive approval from OMB for data collection. The contractor shall ensure that the clearance package follows the required OMB format, justifying the necessity for collecting data and comprehensively responding to each required item in the instructions, including: (1) the objectives of the study or collection of studies, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, (7) limitations of the study and its potential use for policy decisions, and (8) table shells indicating how data will be tabulated and analyzed. [4: https://www.ecfr.gov/current/title-34/subtitle-A/part-97/subpart-A/section-97.102]

All forms clearance packages shall be brief, clear, and well written. Each package shall include the forms and instructions that will be used to obtain data or recruit respondents. The COR will submit the final package to begin the OMB approval process. During the initial 60-day public comment period, the contractor shall pretest the data collection forms so that burden information can be updated in the final package. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process.

The contractor shall obtain approval from its IRB for the evaluations conducted under any task order. The effort to obtain initial approval and to update approval as new task orders are issued should be conducted in conjunction with OMB clearance package efforts.

The contractor shall pre-register the research plan for any effectiveness evaluations conducted under a task order, once the recruited sample sizes are known. The registry shall be chosen by the contractor with approval from the COR. An example of an acceptable registry is the Registry of Efficacy and Effectiveness Studies (REES).

· Prepare OMB forms clearance packages for an expedited review. The contractor shall prepare a forms-clearance package in order to receive expedited approval from OMB under certain conditions when IES plans to collect preliminary or exploratory information to aid in study design. The contractor shall ensure that the clearance package follows the required OMB format for expedited reviews, justifying the necessity for collecting data and comprehensively and concisely responding to each required item in the instructions, including: (1) the objectives of the study or collection of studies, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, and (7) limitations of the study(ies) and their potential use for policy decisions.

The package shall be developed to obtain OMB approval using IES’ Generic (expedited) Clearance vehicle, which requires less documentation and results in approval within roughly 15 days in comparison to a typical information collection request submitted through the regular OMB clearance process. All forms clearance packages shall be brief, clear, and well written. Each package shall include the forms and instructions that will be used to obtain data for ED internal decision-making. The COR will submit the final package to begin the OMB approval process.

The contractor shall obtain approval from its Institutional Review Board (IRB) for the evaluations conducted under any task order. The effort to obtain initial approval and to update approval as new task orders are issued should be conducted in conjunction with OMB clearance package efforts.

Task 6 – Outreach and Site Recruitment

The contractor shall develop and implement a strategy for identifying and recruiting districts, schools, grantees, or other type of participants to meet the sample size requirements specified in a task order.

As part of the recruitment process, the contractor shall prepare notification letters and information packets for the prospective study sites. The notification packet shall include a non-technical brochure describing the study that is suitable for distribution to a broad audience of policymakers and educators. The contractor shall include in the letters and information packets general information on the study as well as specific information on the importance of the study, its purposes, products, timelines, and the expected benefits to be derived from participation in the study. Recruitment may occur through mailings, telephone calls, in-person visits, or discussion with potential sites at key conferences or meetings, as specified in the task order.

The contractor shall submit an outreach and/or recruitment plan, draft letters, and all other notification materials to the COR for review. Once approved by the COR, the contractor shall send the notification materials to potential sites and execute the recruitment plan. The contractor shall consult with the COR to finalize the list of sites for the study, and subsequently submit documentation of memoranda of understanding for those sites to the COR.

As part of the recruitment process, the contractor shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants.

Task 7— Intake and Random Assignment for Effectiveness Studies

The contractor may need to set up a process for enrolling participants into a study (“intake,” including obtaining consent) and for random assignment. Intake may occur at one or more specific points in time – e.g., at the beginning of each semester or of a school year – or on a rolling basis over an extended period. Random assignment may be of students, teachers/classroom, schools, grantees, or other units depending on the evaluation design and will be specified in the task order. Random assignment may be conducted by the contractor or in the field (at schools or other sites) to accommodate rolling intake. The contractor shall develop protocols and procedures to conduct intake and random assignment to be minimally disruptive to participants and program staff who may be involved in intake.

Task 8—Implement and Monitor the Programs and Strategies in Effectiveness Studies

The contractor shall undertake activities for conducting studies of effectiveness. These activities might be carried out under one or more task orders.

· Training and implementation support. For task orders that involve conducting an effectiveness study where the contractor or an external vendor provides technical assistance to grantees or training/implementation support to study sites, the contractor shall ensure that these supports are provided with fidelity, according to the study plans. These supports may include the provision of in-person or virtual training, the provision of materials (such as electronic equipment, curriculum, textbooks, technological hardware or software, etc.).

· Monitoring implementation. The contractor shall monitor implementation of the intervention. The contractor shall ensure there is an appropriate firewall, such that there is a clear separation between those who are providing and supporting the implementation and those who are monitoring the implementation.

Task 9—Data Collection

IES expects that studies conducted under these task orders will rely on a variety of data sources. These may include surveys, interviews, or focus groups of agency staff at the state or local level, parent, student, teacher, or school/college surveys, federal records, state records, district records, school and provider records, or records from other administrative systems, including ones that might be maintained by other agencies such as the U.S. Department of Health and Human Services’ National Directory of New Hires. Other data sources could include those collected through vendor systems, classroom observations, and site visits. The specific data to be collected, extracted, and/or coded shall be specified by task order.

Sample data collections include:

· Administer state education agency surveys. The contractor shall administer the relevant survey to the administrators in each state education agency whose work most closely aligns with the study topic. Either the ED program office or the contractor may determine which role(s) at the state are appropriate to sample and compile lists of these individuals. The contractor shall follow up with the universe of state administrators through mail, email, and telephone calls in order to achieve a response rate of 100 percent.

· Administer nationally representative surveys of local educational agencies, postsecondary institutions, providers, or grantees. Following outreach under Task 6, as approved by the COR, the contractor shall administer the relevant survey to a sample of school district, postsecondary institution, and/or provider respondents in charge of overseeing local implementation of postsecondary access and success programs and strategies. Sampled populations may include any eligible entity awarded funds under federal grant programs such as TRIO and GEAR UP. The contractor shall follow up with sampled respondents through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

· Administer youth or adult surveys. The contractor shall administer approved notification and consent, as specified in the task order, and the relevant survey to youth or adults in the most efficient manner possible. Respondents may be, for example, those participating in an implementation or impact study of postsecondary access and success programs or strategies. Respondents may also comprise a nationally representative sample of the target population for these programs and strategies. The contractor shall follow up with respondents through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

· Administer school, postsecondary institution, or provider surveys. The contractor shall administer the relevant survey to schools, postsecondary institutions, or providers in the most efficient manner possible. School, postsecondary institution, or provider respondents may be, for example, those participating in an implementation or impact study of postsecondary access and success programs or strategies. Respondents may also comprise a nationally representative sample of the target population for these programs and strategies. The contractor shall follow up through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

· Administer teacher or faculty surveys. The contractor shall administer the relevant survey to teachers or faculty in the most efficient manner possible. Teacher or faculty respondents may be, for example, those participating in an implementation or impact study of postsecondary access and success programs and strategies. The contractor shall follow up through mail, email, and telephone calls to achieve a response rate of 85 percent.

· Administer parent surveys. The contractor shall administer approved notification and consent, as specified in the task order, and the relevant survey to parents in the most efficient manner possible. Parent respondents may be, for example, those participating in an implementation or impact study of postsecondary access and success programs and strategies. The contractor shall follow up with respondents through mail, email, and telephone calls to achieve a response rate of 85 percent.

· Conduct classroom observations. The contractor shall conduct live or video observations when necessary for evaluation studies of postsecondary access and success programs and strategies. The contractor shall coordinate scheduling and follow up to achieve a response rate of 85 percent.

· Collect student records data. The contractor shall obtain records for using the most efficient process possible, when such records are necessary for evaluation studies of postsecondary access and success programs and strategies. These records may be obtained from state data systems (e.g., states’ longitudinal data systems) or from districts, schools, postsecondary institutions, or providers directly, and include some combination of students’ achievement scores on state or local assessments, demographic characteristics, course taking, grades, attendance, attainment, persistence, graduation, or participation and outcome data from the programs or strategies under study. The contractor shall follow up to achieve a response rate of 85 percent.

· Collect teacher or faculty records data. The contractor shall obtain records for teachers or faculty in using the most efficient process possible when such records are necessary for evaluation studies of postsecondary access and success programs and strategies. These records may be obtained from state data systems (e.g., states’ longitudinal data systems) or from districts, schools, postsecondary institutions, or providers directly, and include some combination of teachers’ or faculty’s evaluation scores, certification, experience, education, grade/subject assignments, retention, or demographic characteristics, as specified in the task order. The contractor shall follow up to achieve a response rate of 85 percent.

· Conduct interviews or focus groups with state/district/school/postsecondary institution provider administrators, teachers/faculty, parents, or students. The contractor shall conduct interviews or focus groups with one or more respondent groups. The contractor shall strategically recruit, schedule, and follow up with respondents to achieve a response rate of 85 percent.

· Obtain state or local extant data and documentation. The contractor shall obtain existing records from states, local educational agencies, postsecondary institutions, or other entities that are relevant to an evaluation study of postsecondary access and success programs and strategies. This could include existing documents related to identification of high-need schools or providers, lists of approved/recommended strategies, or Memoranda of Understanding between institutions collaborating on service provision. These existing documents should be collected using the most efficient process possible.

· Collect data from intervention providers. As specified in the task order, the contractor shall obtain relevant data from providers who support implementation of postsecondary access and success programs and strategies. Such data could come from meta-data in the provider’s system, other existing documents, or interviews or surveys of the vendors – for the purposes of providing information about the programs and strategies, including implementation and costs. These existing documents should be collected using the most efficient process possible.

· Access other administrative records. The contractor shall assist IES in setting up a Memorandum of Understanding with another agency or organization to obtain data on services or outcomes for students in an evaluation study of postsecondary access and success pathways programs and strategies. The contractor shall specify the data to be extracted and, if the data will be transferred to the contractor, arrange for secure transfer, or use the data on site at the other agency/organization.

· Extract and code records obtained from extant sources. As specified in the task order, the contractor shall extract and code relevant data to create analytic variables. This could include extracting and coding student transcripts or student records, the Department’s Federal Student Aid data, other federal agency data, other survey data, or matching with National Student Clearinghouse data.

To comply with federal data security requirements, IES expects the selected contractor for applicable task orders to collect, maintain, and analyze data through the certified and secure IES Data Center (IESDC). As applicable, the task order contractor will be provided multiple licenses to an industry-standard survey system widely used across the federal government to design and administer web-based data collections. Limited technical assistance on the use of the new survey system will be provided by IES or its IT contractors. The contractor will analyze all data collected by the survey system within IESDC. Currently, users of IESDC have access to Stata MP-16, SPSS 26, MPlus 8.4, SAS 9.4, HLM 6.08, and R with R Studio. Contractors preferring software other than those listed above must indicate those preferences in their technical proposal for the given task order. IES cannot guarantee access to software packages other than those listed above. Access to the IESDC Application Server is provided via a web-based remote desktop connection.

Task 10—Data Analysis and Reporting

The contractor shall undertake activities to analyze data and report on policy-relevant findings of an evaluation of postsecondary access and success policies, programs, and strategies.

· Analysis plan. The contractor shall update the study data collection and research plan (report plan/crosswalk) developed in Task 3, which laid out what specific data are needed to support each part of the planned report(s). The contractor shall then develop an analysis plan that further details how each of the elements in the report plan/crosswalk will be carried out, including describing the study design, and identifying the data sources, key measures, and intended data analyses that will be included in the evaluation’s report products. The plan shall include table shells and, if relevant, the statistical power for measuring impacts on the planned outcomes with the achieved sample size. The plan shall specify any sensitivity tests the contractor plans to conduct.

· Evaluation report. The contractor shall conduct data analyses and prepare a report describing what is being studied and the results of the analysis. The contractor shall borrow the introduction, study design description, and data sources from the final analysis plan, rather than developing this from scratch, to the extent that it meets the COR’s approval.

The contractor shall make sure that each publication is written and organized so that it is accessible to policy makers and research-savvy practitioners. The report text length is expected to be no more than 15 pages, with information on the technical aspects of the study included in the appendices. To the extent feasible, publications shall include visually intuitive figures and graphics rather than tables. The contractor shall also include a one-page, stand-alone summary of the report that contains enough information to form the basis for a press release.

Both the draft report and one-page summary will be submitted by the COR for approval through IES’ Standards and Review Office (SRO). The SRO process includes review by two external researchers with expertise in the relevant topic. The contractor shall be responsive to SRO’s comments and make revisions to the report until it clears SRO. After the report and 1-pager are cleared by SRO, the contractor shall provide web-ready (i.e. 508-compliant) versions of the report and one-pager.

· Internal study brief. The contractor shall conduct data analyses as outline in an approved plan. The contractor shall prepare a brief that includes mostly exhibits (figure, graphs, tables) and bullet points of key findings. The text of the brief shall be no more than 15 pages long but the set of exhibits the text draws upon may be significantly longer.

· Study snapshot. The contractor shall conduct data analyses as outlined in an approved analysis plan. The contractor shall prepare a snapshot that includes 3-5 main headlines with a set of supporting bullets. Each headline should be accompanied by one or two exhibits (figures, graphs, tables). The snapshot is meant to provide a quick first look at data, with minimal text and details, as compared to an evaluation report. The main body of the snapshots are typically less than 15 pages.

The snapshot will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the snapshot until it clears SRO. After the snapshot is cleared, the contractor shall provide web-ready (i.e., 508 compliant) versions of the final snapshot.

· Compendium of data tables. The contractor shall develop a comprehensive set of data tables based on a given data collection to supplement the analyses and findings featured in an evaluation report. The contractor shall present the tables in an organized manner and display them using a reader-friendly format.

The compendium will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the compendium until it clears SRO. After the compendium is cleared, the contractor shall provide web-ready (i.e. 508-compliant) versions of the final compendium.

· How-to guide. The contractor shall prepare a How-To guide that is geared toward practitioners. The purpose is to provide guidance on how best to implement a particular postsecondary access and success program or strategy whose effectiveness was studied and found to be promising. The guide may include, as specified by the COR, considerations of vendor selection, capacity requirements to implement, flexibilities for implementation, costs, potential pitfalls and ways to overcome them, as well as other necessary technical details. There is not a page limit for these products, but the contractor shall balance the need for sufficient detail with the need to make them approachable and accessible to a practitioner audience.

The guide will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the guide until it clears SRO. After the guide is cleared, the contractor shall provide web-ready (i.e., 508-compliant) versions of the final guide.

Task 11—Data Files

The contractor shall undertake activities to prepare data files containing data collected for evaluation reports and/or study snapshots.

· Restricted use data files. The contractor shall prepare a set of data files that contain all data collected for an evaluation report and/or study snapshot, as specified by the COR, with all direct personal identifiers removed. The contractor shall provide documentation for the files including variable labels, category labels, the unit of observation, weights, and how missing data were addressed.

The contractor shall also provide IES with a risk disclosure analysis for submission to NCES’ Disclosure Review Board (DRB). The contractor shall provide the data codebooks. After DRB approval of the codebooks and risk disclosure analysis, the contractor shall submit to the COR the data files consistent with NCES standards4. ASCII or SAS file formats are acceptable.

· Files for long-term followup. IES may want to collect outcome data on study participants at a time point that is beyond the end of the contract. To allow for this, the contractor shall prepare a data file containing all data collected for a study, including direct personal identifiers for study participants needed for matching to administrative databases such as those of the National Student Clearinghouse and the National Directory of New Hires. IES will store this file and the appropriate documentation on its secure data center and may share it under a later contract for the purpose of conducting long-term followup.

Task 12—Logistical Support for Departmental Activities Related to Synthesizing Findings

IES may wish to convene meetings or develop reports and syntheses of work for Congress or ED that relate to the topics under this IDIQ.