Title_I_Pilots_updated_DRAFT_PWS_2-20-19.pdf

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

DRAFT Performance Work Statement Version: February 20, 2019

EVALUATION OF TITLE I PILOTS THAT PROVIDE FLEXIBILITY UNDER THE

EVERY STUDENT SUCCEEDS ACT

I. INTRODUCTION

The National Center for Education Evaluation (NCEE) of the U.S. Department of Education’s (Department) Institute of Education Sciences (IES) intends to conduct an evaluation of up to two pilot programs authorized by the Every Student Succeeds Act: (1) Innovative Assessment and Accountability Demonstration Authority; (2) Flexibility for Equitable Per-Pupil Funding. The evaluation will focus on describing implementation in initial pilot sites, challenges faced, and approaches to mitigating challenges.

The evaluation may also examine how key educational inputs, such as instructional practices and school-level spending, as well as student outcomes evolve during the pilot.

This contract includes a 42-month base period and four options (to be exercised at the Department’s discretion) that could result in an up to 4.5-year (54-month) contract period of performance.

The base of the contract will be to conduct the evaluation of the Innovative Assessment and Accountability Demonstration Authority. It includes refinement of the study design, two Technical Work Group (TWG) meetings, development of data collection forms, two rounds of data collection, data analysis, and preparation of two reports.

The first option will be to add district, school, and teacher surveys to the evaluation of the Innovative Assessment and Accountability Demonstration Authority in the base contract. Tasks include refinement of the sampling plan, development of survey instruments, the Office of Management and Budget (OMB) forms clearance package, two rounds of survey administration, analysis of survey data, and preparation of results in one report. The period of performances for the first option and base fully overlap, and all analogous tasks will be conducted in full coordination to maximize efficiency.

The second option will be to add up to two pilot states to the evaluation of the Innovative Assessment and Accountability Demonstration Authority. Tasks under this option include a TWG meeting, development of data collection forms, three rounds of data collection, data analysis, and preparation of one report. The second option period overlaps with the base and first option periods, and extends it by 12 months. All analogous tasks will be conducted in coordination to maximize efficiency.

The third option, analogous to the first option, will be to add district, school, and teacher surveys to the second option. Tasks under this option include refinement of the sampling plan, development of survey instruments, the Office of Management and Budget (OMB) forms clearance package, three rounds of survey administration, analysis of survey data, and preparation of results in one report. The second and third option periods fully overlap, and all analogous tasks will be conducted in coordination to maximize efficiency.

The fourth option will be to conduct the evaluation of the Flexibility for Equitable Per-Pupil Funding. It includes refinement of the study design, two TWG meetings, development of data collection forms and the OMB forms clearance package, three rounds of data collection, data analysis, and preparation of two reports, one of which will synthesize the findings and broader lessons learned from the two pilot programs. The fourth option period overlaps with the base period by 39 months, and if this option is exercised, many of the tasks that are analogous with the base and first three option periods will be conducted in coordination to maximize efficiency.

Contract Periods of Performance

Period Timeframe Duration Contract Month

Base July 2019 – December 2022 42 months Months 1 – 42

Option I July 2019 – December 2022 42 months Months 1 – 42

Option II July 2019 – December 2023 54 months Months 1 – 54

Option III July 2019 – December 2023 54 months Months 1 – 54

Option IV October 2019 – December 2023 51 months Months 4 – 54

A. Legislative Authority

An evaluation of the Innovative Assessment and Accountability Demonstration Authority is mandated by Title I, Section 1204 of the Elementary and Secondary Education Act (or ESEA, as amended by the Every Student Succeeds Act, P.L. 114-95). An evaluation of the Flexibility for Equitable Per-Pupil Funding is mandated by Title I, Section 1501 of ESEA.

B. Background

ESEA is the signature federal law for K-12 education, and the Title I program is by far the largest in ESSA, with about $16 billion in annual funding. The goal of Title I is to provide all children with significant opportunities to receive a fair, equitable, and high-quality education, and to close educational achievement gaps. It primarily does so by requiring states to hold districts and schools accountable for their students’ achievement (for example through annual testing), and by distributing formula grants to help ensure that high-poverty schools are adequately resourced. ESEA was first enacted in 1965 and has been reauthorized a number of times since then, most recently in December 2015 as the Every Student Succeeds Act, or ESSA.

While ESSA retains a number of core requirements that were introduced in the prior reauthorization of ESEA (No Child Left Behind Act of 2001), it also seeks to give states and districts more flexibility overall. The idea is that this approach will allow states and districts to better tailor education policies to their unique needs and to promote diverse experimentation around the country. Two of ESSA’s new Title I pilot programs highlight this approach. The two programs aim to provide states and districts with temporary relief from some of ESSA’s requirements in the hopes of spurring innovation in two of Title I’s key areas: accountability assessments and school funding.

The first program, the Innovative Assessment and Accountability Demonstration Authority (IADA), allows up to seven states to pilot an innovative assessment system for three years in some districts, with the goal of refining and ultimately scaling up the system to the entire state for accountability purposes.1 By participating, states can avoid having to double-test students in pilot districts, although states still have to demonstrate that the innovative assessment is valid, reliable, and comparable to the existing state assessment. Two states and Puerto Rico applied in the first round, and the two states—Louisiana and New Hampshire—were approved to begin the initial three-year demonstration period in the 2018-19 school year. The Department subsequently held a second round, and two states—Georgia and North Carolina— submitted applications in response. These states’ applications were under review as of January 2019.

Subject to continued interest from states, future application rounds are possible since the seven-state maximum has not yet been reached.

The second program, Flexibility for Equitable Per-Pupil Funding (Student-Centered Funding or SCF), allows up to 50 districts to pilot a weighted-per-pupil funding system for three years.2 By participating, districts are granted the flexibility to pool funds from a number of ESEA programs (as well as state and local sources) and to pick the types of students and specific weights that will determine how the pooled funds are allocated to schools. The pooled funds do not need to be tracked by each contributing program, but the district must show that the funds are being used in a manner that is consistent with the overall purpose of the contributing programs. The district must also show that certain minimum standards for funding are met for particular groups such as low-income students and English-learners. Five districts and Puerto Rico applied over two rounds in 2018, but only Puerto Rico’s application was approved as of December 2018. Subject to continued interest from districts, future application rounds are expected since the 50-district maximum has not yet been reached.

Neither of these pilot programs comes with additional federal funding. States must have already done much of the assessment development and be ready to start piloting, although they can use existing funds from sources such as Title I, Part B (state assessment grants) to support the pilot. Similarly, districts must allocate existing funds under their student-centered funding system – they will not receive more funds to distribute under the pilot. Thus, both pilots are primarily about providing flexibility by relaxing some of the federal requirements that might otherwise impede these entities from trying out alternative approaches to testing students and funding schools.

Such an approach is consistent with the critiques of the No Child Left Behind Act being too much of a “one-size-fits-all” model and with the corresponding movement back towards a more “local control” model, which existed in the 1990s under an earlier version of ESEA (Improving America’s Schools Act).

At the same time, the pilots impose a number of minimum requirements to mitigate concerns that increased flexibility might inadvertently exacerbate inequities. This evaluation will make valuable contributions to the field if, in addition to promoting a better understanding of the development and implementation of assessment and school-funding systems, it provides broader lessons learned about the federal approach to ensuring a high-quality and equal education for all.

C. Key Research Questions

Consistent with the programs’ legislative mandates, this evaluation will focus primarily on charting initial pilot sites’ progress and identifying the lessons learned for developing and implementing alternative assessments and funding systems. To the extent feasible, the evaluation will also provide descriptive information on how educational inputs and outcomes evolve during the pilot.

1 https://www2.ed.gov/admins/lead/account/iada/index.html. Consortia of states are also allowed, but the total number of states involved in the pilot program cannot exceed seven. The initial three-year demonstration period may be extended by two years.

2 https://www2.ed.gov/policy/elsec/leg/essa/scfp/studentcentered.html. The flexibility agreement may be renewed for additional three-year periods.

https://www2.ed.gov/admins/lead/account/iada/index.html https://www2.ed.gov/policy/elsec/leg/essa/scfp/studentcentered.html

The IADA component of the evaluation will address the following questions:

• Why are states developing an innovative assessment system, and how are they expecting the system to accomplish their goals? How does the system compare to their existing assessments?

• Have states’ innovative assessment systems been implemented with fidelity and met the core requirements of the federal pilot program after three years? To what extent have teachers, principals, and other school leaders demonstrated a commitment and capacity to implement the innovative assessment system?

• What were the key challenges to developing an innovative assessment system, piloting it, and scaling it up? How were the challenges addressed, and how did participation in the federal pilot program facilitate or impede the process?

If the first option is exercised, the following research questions will be added to the IADA component of the evaluation:

• How have districts, schools, and teachers perceived and adapted to the innovative assessment system? How have practices related to instruction, professional development, and burden under the innovative assessment system compared to practices under the states’ existing assessments?

If the second and/or third options are exercised, the analogous questions from above will be addressed for the additional pilot states. In addition, the evaluation will explore if and how the experiences of the second round of pilot states compare to the first round of pilot states, given that the second round of pilot states will have started a year later. Any practical lessons learned from this comparison will be documented in the accompanying report.

If the fourth option to conduct the SCF component of the evaluation is exercised, the following research questions will be addressed:

• Why are districts developing a student-centered funding system, and how are they expecting the system to accomplish their goals? How does the system compare to their prior approach to school funding?

• Have districts’ student-centered funding systems been implemented with fidelity?

• What were the key challenges to developing and implementing a student-centered funding system? How were the challenges addressed, and how did participation in the federal pilot program facilitate or impede the process?

• How have schools and teachers perceived and adapted to the student-centered funding system?

How do instructional practices under the system compare to under the districts’ prior system?

• What is the distribution of school-level spending (overall and disaggregated by federal, state, and local sources, as well as personnel and non-personnel resources) and student achievement (overall and disaggregated by relevant subgroups)? How do these distributions compare before and after the start of the pilot?

D. Evaluation Approach

Discussed below are a number of factors that will affect the evaluation design or successful execution of the study. Other design issues may arise as the evaluation is carried out that will need to be resolved by the study team.

Burden and Duplication. Both programs require participants to produce substantial information on the progress of the pilots and to show evidence of compliance with federal requirements. A number of pilot sites also expect to conduct their own evaluation activities. The contractor shall have a study approach that maximizes the utility of existing information from pilot sites and minimizes the need for new data collections that may be duplicative and burdensome to respondents.

Sampling. The primary unit of analysis is the state for the IADA component of the evaluation and the district for the SCF component. A deeper understanding of implementation and outcomes requires further examining how the pilot plays out on the ground – with districts, schools, and teachers for IADA and with schools and teachers for SCF. While only a few IADA states and SCF districts are expected, a wide range of districts, schools, and teachers are contained within these pilot sites – and may change over time as the pilot scales up in each site. The contractor shall have a sensible sampling approach that will provide credible answers to the research questions while minimizing study costs and respondent burden. Any assumptions about policy-relevant minimum detectable differences shall be clearly described and justified.

Measurement. An important objective of the evaluation is to better understand if and how key behaviors at the ground level may be changing under the pilot programs. For example, if a new assessment system is to ultimately improve student learning, it should lead to behavioral changes in teachers, including how they provide instruction. The contractor shall have an approach that identifies the appropriate aspects of behavior to measure and measures them reliably and efficiently using a survey.

Data Access. Particularly under the SCF component of the evaluation, availability and access to detailed spending data over a number of years will be important. The contractor shall have an approach to ensure that complete and comparable data are available and accessible for the purposes of the evaluation.

Data Triangulation and Synthesis. The evaluation data are expected to come from diverse sources, including documents, interviews, surveys, and administrative data. The contractor shall have an approach to triangulating and synthesizing these data so that credible, concise, and policy-relevant results (on both the individual programs and on federal flexibility pilots more generally) can be presented in the evaluation’s reports.

II. SCOPE OF WORK

This section specifies the tasks that the contractor shall perform. The Contracting Officer’s Representative (COR) reserves the right to inspect the contractor’s work while in progress. The contractor shall submit to the COR any progress report or other written documentation of work to date that the COR requests to facilitate that inspection, in accordance with FAR 52.246-5 (Inspection of Services- Cost- Reimbursement). All tasks are expected to be cost-reimbursement.

A. Base Contract

Task 1—Kick-off Meeting

The contractor’s project director and up to three key project staff shall meet with the Contracting Officer’s Representative (COR) and other appropriate Department staff within 2 weeks after the effective date of the contract to discuss details regarding the tasks outlined in the proposal, study design, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, D.C. The contractor shall provide an agenda for the meeting to the COR within 1 week after the effective date of the contract. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within 1 week after the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the study.

Deliverables: Kick-off meeting agenda Kick-off meeting summary

Task 2—Refine Study Design and Develop Data Collection Instruments

The contractor shall prepare a study design report, which revises and augments, as appropriate, the plans submitted in the contractor’s proposal. The report shall include detailed plans for the study design, data collection, and data analyses. The draft study design report shall be submitted at least 4 weeks before the first TWG meeting (see Task 3). Based on the first TWG meeting and the COR’s technical direction, the contractor shall make revisions and submit the final study design report no later than 9 months after the effective date of the contract.

The contractor shall also draft and revise all relevant data collection instruments for the two IADA pilot sites (New Hampshire and Louisiana). The instruments shall address the key research questions outlined in the previous section. Data collection shall consist of:

Extant Documents - These forms will be used to request and extract relevant information from documents that the pilot sites are already producing (for example, as part of the required annual progress reporting to the Department). At a minimum, relevant documents from each of the first three years of the pilot shall be collected and analyzed (2018-19 school year to 2020-21 school year).

State Interviews - These forms will be used to complete follow-up interviews with state officials in the pilot sites to clarify and supplement, as needed, the information extracted from extant documents.

These interviews will be conducted twice, once for the 2019-20 school year and once for the 2020-21 school year. The first set of annual progress reports, covering the 2018-19 school year, are expected at the end of August 2019. Analogous submissions in subsequent school years are expected. As part of the instrument development process, the contractor may consult with state officials in New Hampshire and Louisiana once they submit the first annual performance report. In addition, the contractor shall prepare to have brief interviews with up to five states that are not participating in the pilot to learn about possible factors that were barriers to participation. These interviews will be conducted once in the 2019-20 school year.

Deliverables: Draft data collection instruments Revised data collection instruments

Final data collection instruments Draft study design report

Final study design report

Task 3—Technical Working Group

Establish Technical Working Group (TWG). The role of the TWG will be to advise the contractor and the Department on the conduct of the evaluation including, but not limited to issues of: study design, instrumentation, data collection and analysis, and reporting. The TWG shall be comprised of up to 5 individuals, and shall include content experts in the types of assessment systems implemented by the pilot sites, as well as experts in technical methods relevant to this evaluation design. The contractor shall submit a list of proposed TWG members no later than 2 weeks after the effective date of the contract.

When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the evaluation.

After the COR approves the proposed TWG members, the contractor shall contact each proposed member and, within 4 weeks of COR approval, submit letters of commitment as well as documentation of consulting rates for the proposed panelists to the COR and CS for approval by the CO.

Convene Technical Working Group. The contractor shall convene 2 meetings of the TWG during the base contract. The first meeting shall be held no later than 6 months after the effective date of the contract and shall focus on the study design, as well as plans for data collection and analyses. The second meeting shall be held no later than 35 months after the effective date of the contract and shall focus on the evaluation analyses, findings, and reporting. No later than 3 weeks before each meeting, the contractor shall submit to the COR draft materials for the meeting, including an agenda, presentation materials, and other materials that will be shared with the TWG. Final materials shall be fully responsive to COR comments on the draft materials. After COR approval, and no later than 1 week before the meeting, the contractor shall send applicable approved materials to the TWG.

After each meeting, the contractor shall submit a written summary of the meeting to the COR within 1 week that details suggestions and recommendations along with the contractor’s proposed actions.

Proposed actions shall not go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in Department facilities in Washington, D.C. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

The contractor shall provide copies of all reviews of materials by the TWG to the COR within 1 week of receipt by the contractor. If TWG members recommend any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes to the study based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review evaluation materials, as appropriate, when group meetings are not required.

Deliverables: List of proposed TWG members Letters of commitment and consulting rates

Draft materials for meeting 1

Final materials for meeting 1 Summary of TWG meeting 1

Draft materials for meeting 2 Final materials for meeting 2

Summary of TWG meeting 2

Task 4—Data Collection and Analysis

Using instruments approved by the COR and in accordance with the COR-approved study design under Task 2, the contractor shall collect data to address the evaluation’s key research questions. This includes collecting pilot states’ relevant extant documents at a minimum for school years 2018-19, 2019-20, and 2020-21; analyzing the documents; following up with interviews of relevant staff in each pilot state for the 2019-20 and 2020-21 school years to clarify and supplement the extant documents; and analyzing the interview data. It also includes brief interviews with up to five non-pilot states during the 2019-20 school year to learn about possible factors that were barriers to participation.

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2002/stdtoc.asp).

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated. As part of the monthly reporting requirement (Task 6), the contractor shall provide ongoing updates about the progress of data collection activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program.

The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide information on these procedures to the Department upon request.

The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; the E-Government Act of 2002, 44 U.S.C. §208; the Federal Acquisition Regulation privacy clauses 52.224-1 and 52.224-2 requiring compliance with the Privacy Act; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall be responsible for safeguarding any personally-identifiable information (PII) that is collected for the purposes of this contract. The contractor must also comply with any relevant requirements of the Privacy Act, 5 USC 552a; Federal Information Security Modernization Act of 2014, 44 USC §208; as well as relevant OMB directives, such as OMB M-06-16 and OMB M-17-

12. The contractor shall abide by and follow all Departmental privacy policies, procedures, processes, and standards. All electronically-stored sensitive data shall be encrypted at rest and in motion. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act of 2002 (ESRA) (20 U.S.C. 9573).

Task 5—Report Preparation

The contractor shall make sure that evaluation reports are well-written and organized, so that they are accessible to policy makers and research-savvy practitioners. The contractor shall ensure that reports are consistent with IES peer review standards. For each report, the following documents shall be produced unless otherwise specified:

• A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

• A 15-page report. Full technical aspects of the report shall be included in appendices.

In preparation for each report, the contractor shall prepare a detailed report outline that includes key findings and tables. The contractor shall submit a draft report outline at least 7 weeks prior to the draft report due date. The contractor shall submit a final report outline at least 4 weeks prior to the draft report due date.

The contractor shall prepare two reports that clearly address the evaluation’s key research questions:

(1) Progress Report: In accordance with the requirements of ESSA (Section 1204c), this report shall principally use pilot sites’ extant documents to summarize their progress in implementing IADA and the extent to which they are meeting federal requirements for the program.

(2) Dissemination of Best Practices: In accordance with the requirements of ESSA (Section 1204m), this report shall focus on actionable, policy-relevant lessons learned for the development and implementation of IADA.

The contractor shall submit a draft of the first report no later than 31 months after contract award and a draft of the second report no later than 37 months after contract award. The COR will review the draft and provide comments within 2 weeks. The contractor shall submit a revised draft report ready for submission to the Standard’s and Review Office (SRO) within 4 weeks of receiving COR comments. The COR will submit the draft report to SRO. Within approximately 2 months, the COR will provide comments to the contractor from SRO. The contractor shall make final revisions based on SRO comments and submit a revised final report within 1 month. The contractor shall revise the report until all peer review comments are addressed and the report clears the SRO review (current experience is an average of 3 rounds of successively less extensive comments). The contractor shall provide a web-ready and a print ready version of the report to the COR within 2 weeks after the report is cleared by SRO.

During the course of the study and at its completion, the contractor shall be prepared to present up to four briefings of results for program staff and/or policy makers if asked to do so.

Deliverables: Draft outline for report 1 Final outline for report 1 Draft report 1

Revised draft report 1, SRO-ready Final report 1 Print and web-ready report 1

Draft outline for report 2 Final outline for report 2 Draft report 2

Revised draft report 2, SRO-ready Final report 2 Print and web-ready report 2

Task 6—Project Management Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the CO and one copy to the COR, simultaneously with the monthly invoice. The base contract includes performance reports for months 1 to 42.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, staff use, and changes in personnel and their security designation. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

B. Option I

Task 7—Develop Survey Instruments and Prepare OMB Package

The contractor shall fully integrate district, school, and teacher surveys for the two IADA pilot sites (New Hampshire and Louisiana) into the activities of Task 2. This shall include refining the study design to incorporate the surveys (for example, a sampling strategy) and developing the survey instruments. Data collection shall consist of:

District survey – This survey will be administered twice, once in spring 2020 and once in spring 2021.

School survey – This survey will be administered twice, once in spring 2020 and once in spring 2021.

Teacher survey – This survey will be administered twice, once in spring 2020 and once in spring

2021.

The due dates for the survey instruments and design report shall be the same as in Task 2. In addition, the contractor shall prepare an OMB forms clearance package. The package shall include a description of the study design, sample, and data collection instruments, as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions. The contractor shall ensure that the forms clearance package includes: (1) the evaluation mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling procedures, and

(5) limitations of the study and its potential use for policy decisions. The contractor shall submit a draft forms OMB clearance package to the COR no later than 5 months after award. The COR will submit the accepted forms clearance package for Departmental review and posting for public comment. Based on comments from internal and external stakeholders (after the first 60-day clearance process), the contractor shall make all needed revisions and submit a revised forms OMB clearance package. The COR will then submit the accepted revised clearance package to OMB for the final 60 days.

The contractor shall design data collection instruments to address the key research questions. The contractor shall make use of or refine existing instruments to the extent possible and applicable. For instruments where validity or reliability are not yet well established, the contractor shall pilot test each of the instruments with no more than nine respondents per instrument to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective. Piloting is to be conducted prior to the final 60 days in the OMB submission process. The contractor shall ensure that instruments are revised as needed based on pilot test results.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full-scale data collection with the IRB approval obtained by districts from their own institutions. Specifically, the contractor is responsible for ensuring that all aspects of the data collection have been approved by the appropriate IRBs, including its own IRB.

Final data collection instruments Draft study design report

Final study design report Draft forms OMB clearance package

Revised forms OMB clearance package Final forms OMB clearance package Documentation of IRB approvals

Task 8—Technical Working Group

The contractor shall fully integrate relevant TWG activities under this option with Task 3. This shall include proposing one additional TWG member to ensure that the TWG has relevant expertise on the collection of survey data related to IADA at the district, school, and teacher levels. This shall also include ensuring that the TWG meetings incorporate design, analysis, and reporting issues relevant to the survey data to be collected under this option.

Draft materials for meeting 1

Final materials for meeting 1 Summary of TWG meeting 1

Draft materials for meeting 2 Final materials for meeting 2

Summary of TWG meeting 2

Task 9—Data Collection and Analysis

The contractor shall fully integrate relevant data collection and analysis activities under this option with Task 4. Using instruments approved by OMB under Task 7, the contractor shall collect data to address the evaluation’s key research questions. This includes collecting district, school, and teacher survey data from pilot states in spring 2020 and spring 2021; and analyzing the survey data in accordance with the study design in Task 7.

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2002/stdtoc.asp).

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated. As part of the monthly reporting requirement (Task 6), the contractor shall provide ongoing updates about the progress of data collection activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program.

The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide information on these procedures to the Department upon request.

The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; the E-Government Act of 2002, 44 U.S.C. §208; the Federal Acquisition Regulation privacy clauses 52.224-1 and 52.224-2 requiring compliance with the Privacy Act; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall be responsible for safeguarding any personally-identifiable information (PII) that is collected for the purposes of this contract. The contractor must also comply with any relevant requirements of the Privacy Act, 5 USC 552a; Federal Information Security Modernization Act of 2014, 44 USC §208; as well as relevant OMB directives, such as OMB M-06-16 and OMB M-17-

12. The contractor shall abide by and follow all Departmental privacy policies, procedures, processes, and standards. All electronically-stored sensitive data shall be encrypted at rest and in motion. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act of 2002 (ESRA) (20 U.S.C. 9573).

Task 10—Report Preparation

The contractor shall integrate relevant report preparation activities under this option with Task 5. The district, school, and teacher survey data shall be used for the second report only (Dissemination of Best Practices).

Deliverables: Draft outline for report 2 Final outline for report 2 Draft report 2

Revised draft report 2, SRO-ready Final report 2 Print and web-ready report 2

C. Option II

Task 11—Refine Study Design and Develop Data Collection Instruments

The contractor shall fully integrate the items in this task with Task 2 in the base period. The contractor shall draft all relevant data collection instruments for up to two additional IADA pilot sites (Georgia and North Carolina). The instruments shall address the key research questions. Data collection for these two additional sites shall consist of:

Extant Documents - These forms will be used to request and extract relevant information from documents that the pilot sites are already producing (for example, as part of the required reporting to the Department). At a minimum, relevant documents from each of the first three years of the pilot shall be collected and analyzed (2019-20 school year to 2021-22 school year).

State Interviews - These forms will be used to complete follow-up interviews with state officials in the pilot sites to clarify and supplement, as needed, the information extracted from extant documents.

These interviews will be conducted three times, once for the 2019-20 school year, once for the 2020- 21 school year, and once for the 2021-22 school year. As in Task 2, the contractor shall assume that annual progress reports for these additional pilot sites will similarly be due in August and cover the prior school year.

Final data collection instruments Draft study design report

Final study design report

Task 12—Technical Working Group

The contractor shall convene a third meeting of the TWG, which shall focus on the evaluation analyses, findings, and reporting for all of the data collected on up to two additional pilot sites. The TWG meeting shall also focus on comparing the findings from the initial two pilot sites and the subsequent pilot sites (which began a year later), for the purposes of deriving practical lessons for the field. This third TWG meeting shall be held no later than 44 months after the effective date of the contract. No later than 3 weeks before each meeting, the contractor shall submit to the COR draft materials for the meeting, including an agenda, presentation materials, and other materials that will be shared with the TWG. Final materials shall be fully responsive to COR comments on the draft materials. After COR approval, and no later than 1 week before the meeting, the contractor shall send applicable approved materials to the TWG.

After each meeting, the contractor shall submit a written summary of the meeting to the COR within 1 week that details suggestions and recommendations along with the contractor’s proposed actions.

Proposed actions shall not go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in Department facilities in Washington, D.C. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

The contractor shall provide copies of all reviews of materials by the TWG to the COR within 1 week of receipt by the contractor. If TWG members recommend any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes to the study based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review evaluation materials, as appropriate, when group meetings are not required.

The contractor shall also ensure that the design materials developed under Task 11 for these two additional pilot sites are fully integrated into the first TWG meeting planned under Task 3.

Deliverables: Draft materials for meeting 1

Summary of TWG meeting 1 Draft materials for meeting 3 Final materials for meeting 3

Summary of TWG meeting 3

Task 13—Data Collection and Analysis

The contractor shall integrate the data collection for the additional pilot sites (per the plan refined in Task

11) with the data collection for the initial pilot sites under Task 4.In addition, the contractor shall collect a third round of data for the additional pilot sites that covers their third year of implementation (2021-22 school year).

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2002/stdtoc.asp).

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated. As part of the monthly reporting requirement (Tasks 6 and 15), the contractor shall provide ongoing updates about the progress of data collection activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program.

The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide information on these procedures to the Department upon request.

The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; the E-Government Act of 2002, 44 U.S.C. §208; the Federal Acquisition Regulation privacy clauses 52.224-1 and 52.224-2 requiring compliance with the Privacy Act; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall be responsible for safeguarding any personally-identifiable information (PII) that is collected for the purposes of this contract. The contractor must also comply with any relevant requirements of the Privacy Act, 5 USC 552a; Federal Information Security Modernization Act of 2014, 44 USC §208; as well as relevant OMB directives, such as OMB M-06-16 and OMB M-17-

12. The contractor shall abide by and follow all Departmental privacy policies, procedures, processes, and standards. All electronically-stored sensitive data shall be encrypted at rest and in motion. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act of 2002 (ESRA) (20 U.S.C. 9573).

Task 14—Report Preparation

The contractor shall ensure that evaluation reports are well-written and organized, so that they are accessible to policy makers and research-savvy practitioners. The contractor shall ensure that reports are consistent with IES peer review standards. For each report, the following documents shall be produced unless otherwise specified:

• A 15-page report. Full technical aspects of the report shall be included in appendices.

In preparation for each report, the contractor shall prepare a detailed report outline that includes key findings and tables. The contractor shall submit a draft report outline at least 7 weeks prior to the draft report due date. The contractor shall submit a final report outline at least 4 weeks prior to the draft report due date.

For this task, the contractor shall prepare one report that clearly address the evaluation’s key research questions. This report shall take a similar format to the Progress Report described in Task 5 but focus on the additional pilot site(s), as opposed to the initial two pilot sites. In addition, the report shall describe practical lessons learned for the field based on comparing the experiences of the initial pilot sites and the additional pilot site(s).

The contractor shall submit a draft of the report no later than 47 months after contract award. The COR will review the draft and provide comments within 2 weeks. The contractor shall submit a revised draft report ready for submission to the Standard’s and Review Office (SRO) within 4 weeks of receiving COR comments. The COR will submit the draft report to SRO. Within approximately 2 months, the COR will provide comments to the contractor from SRO. The contractor shall make final revisions based on SRO comments and submit a revised final report within 1 month. The contractor shall revise the report until all peer review comments are addressed and the report clears the SRO review (current experience is an average of 3 rounds of successively less extensive comments). The contractor shall provide a web-ready and a print ready version of the report to the COR within 2 weeks after the report is cleared by the SRO.

The…