Attachments_A-D.pdf

Attachment A- Performance Work Statement

U.S. Department of Education

Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

COMPREHENSIVE LITERACY PROGRAM EVALUATION

Performance Work Statement

I. INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (NCEE) of the U.S. Department of Education’s (the Department) Institute of Education Sciences (IES) requires a contract to examine the implementation of the Striving Readers Comprehensive Literacy (SRCL) Program and to assess the feasibility of an impact evaluation of the Comprehensive Literacy State Development (CLD) Program

(with an option to conduct the impact evaluation, in Option Period I, if deemed feasible).

SRCL is a discretionary federal grant program authorized as part of Title III of Division H of the

Consolidated Appropriations Act of 2016 (P.L. 114-113) under the Title I demonstration authority (Part

E, Section 1502 of the Elementary and Secondary Education Act (ESEA). SRCL competitively awards grants to state educational agencies (SEAs) that then subgrant at least 95 percent of the award to local educational agencies (LEAs) and early learning providers and also provide leadership in the form of a comprehensive literacy plan and technical assistance.

The CLD program is authorized by Title I, Part B, Subpart 2, Section 2222 of the Elementary and

Secondary Education Act as amended by the Every Student Succeeds Act (ESSA). The CLD Program is the successor program to SRCL and, pending the availability of funds, will competitively award grants to

SEAs that will then provide subgrants to LEAs and early learning providers. Both the SRCL and CLD programs intend subgrantees to implement evidence-based comprehensive literacy instruction. The ultimate goal of both programs is to advance literacy skills for students from birth through grade 12, with an emphasis on disadvantaged students.

There are three major components of this requirement: an implementation evaluation of the SRCL program, an impact design/feasibility study for the CLD program, and an optional impact evaluation of the CLD program.

PERIOD OF PERFORMANCE

This contract will include a 48-month Base Period and a 52-month Option Period, which will overlap with the Base Period by 40 months. The total contract period of performance, if the option period is exercised is 60 months. The base period includes the implementation evaluation of the SRCL program and the impact design/feasibility study for the CLD impact evaluation. The option period would include activities to complete the optional CLD impact evaluation.

The periods of performance are as follows:

BASE OPTION PERIOD I

February 23, 2018 – February 22, 2022 October 1, 2018 – February 22, 2023

48 Months 52 Months

Months 1-48 Months 8-60

A. Legislative Authority

Section 1502(b) requires the Department to evaluate demonstration projects supported under Section

1502, such as SRCL. An evaluation of the CLD program is mandated by Section 2225 of ESSA, which states that “From funds reserved under section 2222(b)(1), the Director of the Institute of Education

Sciences shall conduct a national evaluation of the grant and subgrant programs assisted under this subpart. Such evaluation shall include high-quality research that applies rigorous and systematic procedures to obtain valid knowledge relevant to the implementation and effect of the programs and shall directly coordinate with individual State evaluations of the programs’ implementation and impact.”

B. Background

In 2011, the Department awarded SRCL discretionary grants to six selected SEAs (Texas, Pennsylvania, Georgia, Nevada, Louisiana, and Montana) and invested a total of $778 million in these grants between

2011 and 2015.

The Department is investing an additional $380 million in Fiscal Year (FY) 2017, which will support an estimated 5 SEAs over a three-year grant period.

The purpose of the SRCL program is to provide support for the implementation of high-quality comprehensive literacy instruction programs to advance literacy skills for all children from birth through grade 12 with a special emphasis on disadvantaged children. The SRCL Program requires SEA grantees to:

Subgrant no less than 95 percent of grant funds to eligible subgrantees and ensure that of the subgranted funds: 15 percent serves children from birth through age five, 40 percent serves students in kindergarten through grade five, and 40 percent serves students in middle and high school;

Develop or revise a State comprehensive literacy plan that is informed by a comprehensive needs assessment conducted within the past five years and review and update the plan annually;

Ensure they only fund subgrantees with a local literacy plan that: is informed by a comprehensive needs assessment, is aligned with the State comprehensive literacy plan, provides for professional development, includes interventions and practices supported by moderate evidence or strong evidence (where applicable and available), and includes a plan to track children’s outcomes consistent with applicable privacy requirements;

Prioritize eligible subgrantees serving greater numbers or percentages of disadvantaged children;

This section includes a brief summary of the SRCL Program. For detailed information, see:

https://www2.ed.gov/programs/strivingreaders-literacy/index.html.

The Department also awarded formula grants of about $150,000 each in FY 2010 to 46 states, the District of

Columbia, and Puerto Rico. The main purpose of these grants was to develop or enhance a state comprehensive literacy plan.

https://www2.ed.gov/programs/strivingreaders-literacy/index.html

Use data to inform continuous improvement and decision-making, to improve children’s outcomes, and to ensure disadvantaged children are served;

Report on the performance measures set by the Department.

SEA grantees must also use grant funds to supplement, and not supplant any non-Federal funds that would be used to reach the same objectives of this program. Grantees must also fund only subgrantees that provide a written assurance to cooperate with a national evaluation of SRCL.

Eligible subgrantees for the SRCL program include one or more LEAs or, in the case of early literacy, one or more LEAs or nonprofit providers of early childhood education, with a demonstrated record of effectiveness in improving language and early literacy development of children from birth through age five and in providing professional development in language and early literacy development.

The Department has established the following performance measures for the SRCL program:

The percentage of participating four-year-old children who achieve significant gains in oral language skills.

The percentage of participating fifth-grade students who meet or exceed proficiency on State reading/language arts assessments under section 1111(b)(2)(B)(v)(I) of the ESEA, as amended by the ESSA.

The percentage of participating eighth-grade students who meet or exceed proficiency on State reading/language arts assessments under section 1111(b)(2)(B)(v)(I) of the ESEA, as amended by the ESSA.

The percentage of participating high school students who meet or exceed proficiency on State reading/language arts assessments under section 1111(b)(2)(B)(v)(I) of the ESEA, as amended by the ESSA.

C. Key Evaluation Questions, Activities, and Design Issues

The purpose of this contract is to examine the implementation of the SRCL program, to generate design options that can provide credible and useful evidence on the impact of the CLD program on student outcomes, and to conduct an optional impact evaluation of the CLD program, in Option Period I, if feasible. In performing the tasks described in this Performance Work Statement (PWS), the contractor shall address the following questions.

Implementation Evaluation of the SRCL Program

The purpose of the implementation evaluation of the SRCL program is to examine and document the implementation of the SRCL grants awarded in 2017. The key questions to be addressed are:

1. How do SEAs make subgrant award decisions?

How do SEAs determine the extent to which proposed subgrant interventions and practices were aligned with local needs and the State’s comprehensive literacy plan?

How do SEAs determine the extent to which proposed subgrant interventions and practices were supported by relevant moderate or strong evidence?

How do SEAs target awards to eligible subgrantees with the greatest numbers or percentages of disadvantaged children?

2. What technical assistance do the SEAs provide to their subgrantees on identifying, selecting, and implementing interventions and practices that are aligned with local needs and supported by moderate or strong evidence?

3. How do the SEAs inform continuous improvement and evaluate the effectiveness of subgrantees projects? What information do the SEAs collect?

4. How do subgrantees target SRCL awards to eligible schools and early learning programs? What were the characteristics of the funded subgrantees and teachers and students in funded subgrantees?

5. What literacy interventions and practices are used by SRCL schools and early learning programs?

To what extent are SRCL interventions and practices aligned with local needs and the

SEA’s State comprehensive literacy plan?

To what extent are SRCL interventions and practices supported by relevant moderate or strong evidence?

To what extent are SRCL interventions and practices differentiated and appropriate for children from birth through age five and children in kindergarten through grade 5?

To what extent do SRCL subgrantees use assessments to identify student needs, inform instruction, and monitor progress?

To what extent do SRCL subgrantees provide age-appropriate, explicit and systematic instruction and frequent practice in reading and writing, including phonological awareness, phonic decoding, vocabulary, language structure, fluency, and comprehension?

6. What are the literacy outcomes for students in SRCL schools and early learning programs, as measured by existing State and/or local assessment data? Do outcomes improve over time? Are outcomes better in SRCL-funded schools than non-funded schools?

The implementation evaluation will focus on the SRCL grants awarded by the Department in 2017. The contractor’s data collection plan shall describe the proposed respondents, target sample sizes, methods, content, schedule and frequency. The Department assumes the following proposed activities:

Grant application review. Reviews of all funded SEA applications in spring 2018;

Grantee interviews. Interviews with all funded SEAs in spring 2018, spring 2019, and spring

2020 to address questions about state grant activities;

Subgrantee surveys. Survey of all funded LEAs (estimated to be about 150) in fall 2018 to address questions of school-level targeting and local literacy plans;

SRCL school surveys. Surveys of principals and a sample of teachers in a representative sample of 500 funded schools in spring 2019 and spring 2020 to examine the implementation of comprehensive literacy instruction;

Student assessment records. Collection of state and local extant reading/language arts assessment data for a pre-SRCL baseline period and all three years of the SRCL grants (2017-

2018, 2018-2019, and 2019-2020) to examine student outcomes across sites and over time, if possible;

Summary of existing evidence reviews. Summarize existing What Works Clearinghouse

(WWC) intervention reports, single study reviews, and quick reviews as well as studies

Available on the WWC website at http://ies.ed.gov/ncee/wwc/ http://ies.ed.gov/ncee/wwc/ conducted by IES grantees of relevant education research programs to provide independent verification of the extent to which interventions and practices used by SRCL subgrantees (as identified by surveys) are supported by moderate or strong evidence;

Optional systematic evidence reviews. Systematic review of the strength of evidence supporting key SRCL-funded interventions without existing evidence reviews to provide independent verification of whether these interventions are supported by moderate or strong evidence;

Optional fidelity site visits. Site visits to a small number of schools to provide independent information on implementation fidelity for key SRCL-funded interventions of interest

It is beyond the scope of this contract to administer a common direct assessment to students in SRCL-funded schools. The contractor for this evaluation must discuss plans for obtaining the assessment data necessary for addressing the key evaluation questions from grantees and subgrantees. The evaluation cannot rely on the information grantees provide to the Department in their performance reporting. The contractor who compiled the latest SRCL national performance report noted concerns about data inaccuracies, changes in state assessments, and the limitations of analysis because the data are submitted only at the state program level. The contractor must discuss how their data collection plan will overcome challenges such as differences in state assessments across grantees, changes in assessments during the grant period, and grantee and subgrantee concerns about providing disaggregated student records data.

The key products of this component of the evaluation will be:

Implementation evaluation brief based on first full year of implementation (2018-2019)

Final implementation evaluation report based on two full years of implementation (through 2019-

2020)

The following table shows the proposed data collection activities and timeline.

Data

Collection

Activity

Respondents Fall

Spring

Fall

Spring

Fall

Spring

Grant application review

All funded SEAs

(est. 5)

X

Grantee interviews

All funded SEAs

(est. 5)

X X X

Subgrantee surveys

All funded LEAs

(est. 150)

X

SRCL school surveys

Representative survey of principals and teachers

X X

Student assessment records

All funded SEAs

(est. 5);

All funded LEAs

(est. 150)

X X X

Summary of existing evidence reviews

Up to 30 most widely-used interventions or practices

X

IES-funded research grants are listed for each program area at http://ies.ed.gov/funding/grantsearch/index.asp

Optional systematic evidence reviews

Up to 10 most widely-used interventions or practices

X

Optional fidelity site visits

Up to 5 SRCL-funded interventions; up to 100 classrooms

X

Impact Design/Feasibility Study for CLD Impact Evaluation

The purpose of the impact design/feasibility study is to use information from the FY 2017 cohort of

SRCL grantees to assess the feasibility of and generate design options for a rigorous impact evaluation of the CLD program. The final proposed study design options must use the most credible scientific methods feasible, and have the potential to provide information about the CLD program that is useful to policymakers and practitioners. The key questions to be addressed are:

1. Does the SRCL subgrant award process from SEAs to LEAs and schools provide any opportunities for a random assignment impact evaluation design?

How do grantee SEAs identify eligible LEAs and prioritize them for subgrant awards?

Are there more eligible LEAs than SEA grantees can fund?

How do subgrantee LEAs identify eligible schools and prioritize them for subgrant awards? Are there more eligible schools than subgrantee LEAs can fund?

2. What are the most policy-relevant impact questions that can be credibly answered about the CLD program? For example, is it the impact of the federal grant/subgrant money? Is it the impact of implementing an average program funded by CLD grants? Is it the impact of a particular CLD-funded intervention or practice?

3. What are the most credible and feasible study designs for each of the relevant impact questions?

4. What is the most credible and feasible counterfactual for each of the relevant impact questions?

How will an appropriate counterfactual be identified and recruited?

5. What are the most appropriate and relevant outcomes to collect for each of the relevant impact questions? How will these outcomes be measured, and how often will they be collected?

6. What coordination with the CLD program grantees and subgrantees would be helpful (or necessary) for carrying out each of the proposed study design options, and what are the key required timelines to ensure that such coordination is feasible?

The Department assumes the following proposed activities will inform the study design options:

interviews with up to 9 SEA grantees and up to 9 LEA subgrantees spring 2018;

SRCL Implementation Evaluation LEA surveys fall 2018

Optional Impact Evaluation of the CLD Program

The key questions for this component of the evaluation are:

1. What is the impact of the CLD program or CLD-funded interventions/practices on classroom reading instruction?

2. What is the impact of the CLD program or CLD-funded interventions/practices on student reading outcomes?

II. SCOPE OF WORK

This contract will be cost plus fixed fee (CPFF) and as mentioned earlier, will include a 48-month base period and a 52-month option period. The total contract period of performance, if the option period is exercised, is 60 months. The base period includes the implementation evaluation of the SRCL program and the impact design/feasibility study for the CLD impact evaluation. The option period would include activities to complete the optional CLD impact evaluation.

The specific tasks and subtasks that the contractor shall perform are described below.

BASE PERIOD

Task 1—Communicate with the Department

Subtask 1.1 Kick-Off Meeting

The contractor’s project director and up to three key project staff shall meet with the Contracting Officer

(CO), Contracting Officer’s Representative (COR) and other appropriate Department staff within 2 weeks of contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, D.C. The contractor shall come prepared to identify any areas of concern not contained in the accepted technical proposal or discussed during contract negotiations, and to suggest ways of responding to these concerns. At least two days prior to the meeting, the contractor shall submit an agenda for the meeting to the COR. Within one week following the meeting, the contractor shall submit a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the study.

Deliverables: Kick-Off meeting agenda

Kick-Off meeting summary

Subtask 1.2 Ongoing Meetings

The contractor shall participate in regular meetings with the COR (and other Department staff, as relevant and appropriate) to update the COR on the progress of contract activities and to discuss any issues or anticipated problems and proposed solutions to those problems. The Department assumes the meetings will take place via conference call or webinar every other week during the first 12 months of the base period, and monthly for the remainder of the base period (the contractor shall also be available for additional calls if requested by the COR).

The contractor shall submit a proposed agenda to the COR no later than two working days before each call. Within three working days of each call, the contractor shall submit to the COR a brief summary of the key issues and concerns discussed along with the contractor’s understanding of the COR’s technical direction and next steps.

Deliverables: Meeting agendas

Summaries of telephone calls

Subtask 1.3 Reporting Requirements

The contractor shall submit one copy of the following performance reports each month through the

Invoice Processing Platform.

Monthly Progress Report/Exception Reports: The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task or subtask regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the COR.

Monthly Manpower/Expenditure Reports: The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures; segregating project costs by individual and by task or subtask, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure reports

Task 2—Assess Feasibility of and Explore Design Options for CLD Impact Evaluation

Subtask 2.1 Develop and Refine Feasibility Discussion Protocol and Plan

The contractor shall develop and submit to the COR a draft protocol and plan to address each of the key research questions for the Impact Design/Feasibility Study for CLD Impact Evaluation (listed in Section

C) no later than March 25, 2018. In the protocol the contractor shall include a section to guide discussions with FY 2017 SRCL grantees and subgrantees. The general purpose of this protocol shall be to elicit information that will inform study design options, as well as the feasibility of recruitment, data collection, and timelines for an impact evaluation of the CLD program. The plan that accompanies the protocol shall explain the contractor’s approach to determining how many and which grantees, subgrantees, and any other entities to include in feasibility discussions. The plan shall also explain how each of the proposed items in the protocol will inform the key research questions related to feasibility and design options.

Based on COR feedback, the contractor shall refine and submit a final version of the feasibility plan and protocol no later than April 22, 2018.

The contractor shall include the final feasibility plan and protocol (along with the draft SRCL implementation evaluation design and data collection instruments in Task 4) in the preparation of presentation materials for the first technical working group meeting (Task 3).

Subtask 2.2 Implement Feasibility Plan and Protocol

Following COR approval, the contractor shall implement the final feasibility plan and protocol. As part of implementation, the contractor shall conduct interviews using this protocol with up to 9 grantees and up to 9 subgrantees. The contractor shall complete the interviews no later than June 24, 2018 to ensure sufficient time to prepare the draft study design options report (see Task 5). The contractor shall provide to the COR weekly status updates on interview progress.

Deliverables: Draft Feasibility Plan and Protocol

Final Feasibility Plan and Protocol

Weekly interview status updates

Task 3—Technical Working Group (TWG)

Subtask 3.1 Establish Technical Working Group

The role of the TWG will be to advise the contractor and the Department on the conduct of the implementation evaluation of SRCL and the impact design/feasibility study for CLD, including but not limited to: study design, instrumentation, data collection, and analysis. The TWG shall be comprised of up to 12 individuals, and must include national experts in: evidence-based practices and interventions in literacy instruction across the birth through grade 12 span of the programs, measurement, and evaluation methodology (both experimental and quasi-experimental, as well as conducting high-quality implementation and impact studies in the context of large federal grant programs). The contractor shall submit a list of proposed TWG members no later than March 11, 2018. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. The contractor shall also clearly explain the background and expertise of each proposed TWG member, and how it is specifically relevant to the needs of the study. After the CO and COR approve the proposed TWG members, the contractor shall contact each approved member to secure interest and intention to serve on the TWG.

Deliverable: List of proposed TWG members

Subtask 3.2 Convene Technical Working Group—Meeting 1

The contractor shall convene a meeting of the TWG no later than May 20, 2018 to update the TWG on and obtain input on the conduct of the CLD impact design/feasibility study and to obtain input on the

SRCL implementation evaluation design and data collection instruments. The contractor shall submit a written summary of the meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The contractor shall not move forward with proposed actions unless approved by the COR. While the Department prefers an in-person meeting, this meeting can also be held as a conference call or webinar given the short amount of time between expected contract award and the meeting date. The contractor’s project director and up to three additional key staff shall attend the meeting in person at the Department’s offices in Washington, D.C.. If the TWG members attend the meeting in person, the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Deliverable: Summary of TWG meeting 1

Subtask 3.3 Convene Technical Working Group—Meeting 2

The contractor shall convene a meeting of the TWG no later than October 21, 2018 to discuss the CLD impact evaluation design options. This TWG meeting shall focus on the proposed key research questions, the study design options, identification and recruitment feasibility of a relevant counterfactual comparison group, data collection, and analysis issues. The contractor shall submit a written summary of the meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The contractor shall not move forward with proposed actions unless approved by the COR. This meeting shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Deliverable: Summary of TWG meeting 2

Subtask 3.4 Convene Technical Working Group—Meeting 3

The contractor shall convene a meeting of the TWG no later than October 20, 2019 obtain input on the preliminary analyses for the SRCL Implementation Evaluation Brief and on potential outcomes analyses for the SRCL for the Implementation Evaluation Final Report. The contractor shall submit a written summary of the meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The contractor shall not move forward with proposed actions unless approved by the COR. This meeting shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Deliverable: Summary of TWG meeting 3

Subtask 3.5 Convene Technical Working Group—Meeting 4

The contractor shall convene a meeting of the TWG no later than January 31, 2021 to obtain input on the preliminary analyses for the SRCL Implementation Evaluation Final Report. The contractor shall submit a written summary of the meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The contractor shall not move forward with proposed actions unless approved by the COR. This meeting shall be held at the

Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Deliverable: Summary of TWG meeting 4

Task 4—Finalize SRCL Implementation Evaluation Design

Subtask 4.1 Refine SRCL Implementation Evaluation Design

The contractor shall refine the study design submitted in their proposal and use this design, once approved by the Department, to guide the evaluation. The contractor shall include in the evaluation design all elements outlined below. The contractor shall submit the draft design to the COR no later than March 25, 2018. Based on COR feedback, the contractor shall refine and submit a revised version of the SRCL implementation evaluation design no later than April 22, 2018. This deliverable shall be a topic for discussion at the first TWG meeting (see subtask 3.2). The contractor shall submit a final evaluation design to the COR, incorporating comments from the Department and comments received from the TWG no later than June 17, 2018. The contractor shall not make any significant changes in the design based solely upon the recommendation of technical working group members without the written approval of the

COR.

The elements that the contractor shall include in the evaluation design include, but are not limited to the following:

Logic model: a graphical and/or text representation of the key components and implementation supports of the SRCL program, the expected mediating and longer-term outcomes, and the specific mechanisms of change that describe the theoretical (and any empirical) basis for how the key components and implementation supports impact mediators and child outcomes and the timeframe for achieving these impacts. The contractor shall align the evaluation design to this logic model.

Sampling Plan: While a nationally representative sample is not required in a strict statistical sense, the

Department assumes inclusion of all of the states receiving SRCL grants in 2017. The contractor shall provide evidence that the proposed sampling plan will yield a sample with acceptable external validity. Note that while the Department expects SRCL grantees and subgrantees to be cooperative, per Department of Education regulation requiring grantee cooperation with evaluations conducted by the Secretary (34 C.F.R. section 76.591), obtaining child-level school district records data might require more effort on the part of the contractor to secure cooperation. The contractor shall discuss proposed methods for securing the participation and cooperation of the sampled respondents, including any special permissions or clearances required.

Data collection plan: a description and rationale for the domains to be measured; the methods and activities to be used for measurement of each domain; proposed respondents, sample, and grades

(between preschool and grade 12) for each measure; timeline, frequency, and duration for each measure; properties of proposed existing measures; how new measures will be developed; how data collectors will be trained.

Data analysis and reporting plan: a description of how the contractor proposes to analyze the data to address the key questions and an outline and schedule for the implementation brief and final report

(see task 10). For all data sources, the contractor shall identify the specific types of data analyses that will be done, including the unit of analysis, analyses of power or precision, possible aggregations, method of display, and specific evaluation questions that will be addressed by each set of analyses.

Deliverable: Draft design

Revised design

Final design

Subtask 4.2 Prepare Data Collection Forms and Office of Management and Budget (OMB)Clearance

Package

The contractor shall design data collection instruments to address the key questions. The contractor shall make use of or refine existing instruments to the extent possible and applicable. For instruments where validity or reliability are not yet well established, the contractor shall pilot test each of the instruments with no more than nine respondents per instrument to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures for administering the instruments are effective. Piloting is to be conducted prior to the final 60 days in the

OMB submission process. The contractor shall ensure that instruments are revised as needed based on pilot test results.

The contractor shall identify all data collection instruments proposed for use in the evaluation and submit the draft data collection instruments to the COR no later than March 25, 2018. The contractor shall include in this deliverable a justification for each proposed instrument, including information on the relevant construct the instrument attempts to measure, other studies that have used the instrument or component of the instrument, and the reliability and validity of the instrument. Based on COR feedback, the contractor shall refine and submit a revised version of the data collection instruments no later than

April 8, 2018. This deliverable shall be a topic for discussion at the first TWG meeting (see subtask 3.2).

The contractor shall submit final data collection instruments to the COR, incorporating comments from the Department and comments received from the TWG no later than June 17, 2018. The contractor shall not make any significant changes in the design based solely upon the recommendation of technical working group members without the written approval of the COR.

The contractor shall prepare all relevant forms and develop a clearance package for the Department to submit to OMB under procedures of the Paperwork Reduction Act and 5 CFR 1320. In the package, the contractor shall include a description of the study design and data collection instruments, as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions. The contractor shall submit the draft forms clearance package to the COR no later than

March 25, 2018. The contractor shall revise the package to reflect COR comments and submit a revised package to the COR no later than April 8, 2018.

After submission of the revised package on April 8, 2018, the OMB clearance process will take a minimum of 5 months. The contractor shall be prepared to make revisions to the OMB forms clearance package, to address questions from OMB, and to submit to the COR any necessary modification to the forms and the OMB clearance package within 1 week of receiving comments. Allowing 5 months, OMB clearance is expected to be received in early September 2018.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB. The contractor shall submit documentation of IRB approval to the COR by September 30, 2018.

Deliverable: Draft data collection instruments and justification

Revised data collection instruments and justification

Final data collection instruments

Draft OMB clearance package

Revised OMB clearance package

Final OMB clearance package

Documentation of IRB approvals

Subtask 4.3 Prepare materials for First Technical Working Group Meeting

The contractor shall use the feasibility plan and protocol developed under subtask 2.1, the SRCL implementation evaluation design developed under subtask 4.1, and the data collection instruments developed under subtask 4.2 as the basis for materials for informing the TWG of the design prior to the first technical working group meeting as well as for the preparation of presentation materials for the first technical working group meeting (subtask 3.2).

The contractor shall submit to the COR a draft of these presentation materials and an agenda for the first technical working group meeting no later than four weeks prior to the first TWG meeting. The contractor shall revise the materials for the first TWG meeting in response to any COR comments and submit revised materials to the COR no later than one week prior to the first TWG meeting.

Deliverables: Draft TWG materials and agenda

Revised TWG materials and agenda

Task 5—CLD Impact Evaluation Design Options Report

The contractor shall prepare an impact evaluation design options report for the CLD program that addresses the issues raised in Section C of the PWS, drawing in part on the results of implementing the feasibility plan and protocol (Subtask 2.2) and also in part on the results of LEA data collection in Fall

2018 for the implementation evaluation of SRCL (Task 7).

In the design options report, the contractor shall discuss the pros and cons of each study design option, both in terms of its methodological strength and its practical ease to implement in the field. The report shall also discuss instrumentation, data collection, and analysis issues, particularly as they relate to implementing each of the proposed study design options. The contractor shall provide a draft of the design options report to the COR no later than August 19, 2018. The contractor shall use the design options report as the basis for materials for informing the TWG of the design options prior to the second

TWG meeting as well as for the preparation of presentation materials for the second TWG meeting

(subtask 3.3). The contractor shall submit to the COR a draft of these presentation materials and an agenda for the second TWG meeting no later than four weeks prior to the second TWG meeting. The contractor shall revise the materials for the second TWG meeting in response to any COR comments and submit revised materials to the COR no later than two weeks prior to the second TWG meeting.

The contractor shall revise the design options report as appropriate subsequent to the discussion with the

TWG and in consultation with the COR. The contractor shall submit the final design options report to the

COR no later than November 18, 2018. If TWG members recommend any significant deviations from the study design, data collection, or data analysis options indicated in the study design options report, the contractor shall submit the recommendation to the COR for approval prior to inclusion in the final study design report. The contractor shall not make any significant changes in the study design options report based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review products, as appropriate, when group meetings are not required.

Deliverables: Draft CLD Impact Evaluation Design Options Report

Draft TWG Materials Based on Design Options Report

Revised TWG Materials Based on Design Options Report

Final CLD Impact Evaluation Design Options Report

Task 6– Select, Notify, and Recruit Sample

The contractor shall prepare notification materials (at least a letter and short evaluation summary) for the states, districts, and schools selected for the sample. The contractor shall submit the draft notification materials to the COR no later than March 25, 2018. The contractor shall incorporate comments and submit to the COR the final notification materials no later than June 17, 2018.

Using the Department-approved sampling plan, the contractor shall select the sample of states, districts, and schools to be included in the study and send the approved notification materials to the sampled states, districts, and schools and follow-up to secure their cooperation. The contractor shall submit biweekly updates on the status of the recruitment to the COR during the recruitment period. The contractor shall secure the sampled districts’ participation in the study no later than August 19, 2018. The contractor shall secure the sampled schools’ participation in the study no later than December 30, 2018.

Deliverables: Draft notification materials

Final notification materials

Recruitment updates

Task 7—SRCL Implementation Evaluation Data Collection

Following the Department-approved data collection plan and using instruments approved by OMB, the contractor shall collect data from the sample selected and recruited for the evaluation. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National

Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf). The contractor shall complete the collection of primary data for the 2018-2019 school year (and any pre-SRCL baseline data) no later than the end of June 2019 and for the 2019-2020 school year no later than the end of June 2020.

The contractor shall complete the collection of extant data for the 2018-2019 school year no later than the end of October 2019 and for the 2019-2020 school year no later than the end of October 2020. The contractor shall submit biweekly updates on the status of the data collection to the COR during each period of data collection.

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual http://nces.ed.gov/pubs92/92021.pdf distributions of scores), such that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program with training manuals for data collection and data coding. The manuals shall detail the topics covered in the training, the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. The contractor shall submit the training manuals to the COR within 4 weeks of the start of data collection. The contractor shall submit data collection updates to the COR every two weeks during the data collection.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request. When possible, the contractor shall ensure that individuals collecting and/or coding data do not have knowledge of the study participant’s treatment condition. The contractor, and all subcontractors, shall comply with: The Privacy

Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education

Sciences Reform Act of 2002 (ESRA) (20 U.S.C. 9573).

Deliverables: Data Collection Training Manuals

Data collection updates

Task 8—Evidence Reviews (OPTIONAL TASK)

To build on the summary of existing evidence reviews conducted as part of Task 7 and to provide independent verification of the extent to which SRCL practices are supported by moderate or strong evidence, the contractor shall conduct a systematic review of the strength of evidence supporting the effectiveness of up to ten key SRCL-funded interventions or practices.

Prior to conducting the reviews, the contractor shall prepare a proposed plan for conducting the evidence reviews. In the plan, the contractor shall discuss how interventions will be identified and prioritized for inclusion (providing a prioritized list of proposed interventions), parameters for literature review for each intervention, the WWC topic area protocol that will be used, challenges and how they will be addressed, and how results will be summarized. The contractor shall submit the draft evidence review plan to the

COR no later than May 12, 2019. The COR will review the draft evidence review plan within two weeks.

The contractor shall make revisions to reflect COR comments and submit a final evidence review plan to the COR no later than June 9, 2019.

Using the Department-approved evidence review plan, the contractor shall conduct the review using

WWC-certified reviewers to determine whether existing studies meet WWC evidence standards and an effectiveness rating for each included intervention. The contractor shall obtain and use existing WWC reviews, where available. The contractor shall submit a draft memo summarizing the results of the evidence review to the COR no later than July 21, 2019. Based on COR feedback, the contractor shall refine and submit a revised version of the evidence review memo to the COR no later than September 15, 2019. This deliverable shall be a topic for discussion at the third TWG meeting (see subtask 3.4). The contractor shall submit a final evidence review memo to the COR, incorporating comments from the

Department and comments received from the TWG no later than November 17, 2019.

Deliverables: Draft evidence review plan

Final evidence review plan

Draft evidence review memo

Revised evidence review memo

Final evidence review memo

Task 9—Site Visits (OPTIONAL TASK)

To build on the data collected through self-report in Task 7 and to provide independent information on implementation fidelity for key SRCL-funded interventions of interest, the contractor shall conduct site visits to complete classroom implementation fidelity checklists for up to five key SRCL-funded interventions or practices.

Prior to conducting the site visits, the contractor shall prepare a proposed plan for conducting the site visits. In the plan the contractor shall discuss how interventions will be identified and prioritized for inclusion (providing a prioritized list of proposed interventions), a sampling plan, how site visitors will be trained, challenges and how they will be addressed, and the implementation fidelity checklist or protocol to be used for each intervention. The contractor shall submit the draft site visit plan to the COR no later than November 17, 2019. The COR will review the plan within three weeks. The contractor shall make revisions to reflect COR comments and submit a final site visit plan to the COR no later than January 5, 2020.

Using the Department-approved site visit plan, the contractor shall train site visitors and conduct the site visits between January and April 2020. The contractor shall provide biweekly updates on site visit progress to the COR during data collection.

Deliverables: Draft site visit plan

Final site visit plan

Site visit progress updates

Task 10—Data Analysis and Reports

Subtask 10.1 Refine Analysis Plan

The contractor shall prepare a detailed data analysis plan that documents the intended analyses to be conducted using the data collected in tasks 7 through 9 (tasks 7 and 8 shall inform the implementation brief and tasks 7 and 9 shall inform the final report) to address the evaluation questions. The final plan shall be used to inform a full summary of analyses on the data (to be discussed at the briefings in subtask

10.2 and 10.3), in preparation for the third and fourth TWG meetings (subtasks 3.4 and 3.5) and the implementation brief (subtask 10.2) and final report (subtask 10.3).

The contractor shall submit the draft data analysis plan that covers the entire evaluation but focuses specifically on informing the implementation brief (subtask 10.2) to the COR no later than February 24, 2019. The COR will review the draft analysis plan within two weeks. The contractor shall make revisions to reflect COR comments and submit a revised analysis plan to the COR no later than April 7, 2019.

Prior to drafting the final report (subtask 10.3), the contractor shall update the analysis plan to reflect the full set of analyses proposed to inform the final report. The contractor shall submit an updated analysis plan for the final report to the COR no later than February 23, 2020. The COR will review the updated analysis plan within two weeks. The contractor shall make revisions to reflect COR comments and submit the final analysis plan to the COR no later than April 5, 2020.

Deliverables: Draft implementation brief analysis plan

Revised implementation brief analysis plan

Updated analysis plan for final report

Final analysis plan for final report

Subtask 10.2 Prepare Implementation Evaluation Brief

Using the Department-approved analysis plan, the contractor shall conduct analysis of data collected in task 7 and optional…