Draft_PWS.doc

U.S. Department of Education

Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

DRAFT Performance Work Statement

Impact Evaluation of Training in Multi-Tiered Systems of Support for Reading in Early Elementary School

I. INTRODUCTION

The National Center for Education Evaluation (NCEE) of the U.S. Department of Education’s (Department) Institute of Education Sciences (IES) intends to evaluate the impact of offering schools training in the implementation of multi-tiered systems of support for reading (MTSS-R) in early elementary school on staff practice and student outcomes. The training is expected to be intensive and focused on enabling school staff to successfully implement the key components of MTSS-R. This contract includes a 34-month base period and seven Optional Task Periods (to be exercised at the Government’s discretion) that would result in a 6-year (72-month) contract period of performance.

The base period includes refinement of the study design, selection of training provider, a Technical Work Group (TWG) meeting, development of data collection forms and the Office of Management and Budget (OMB) forms clearance package, recruitment of the study sample, and the conduct of readiness activities and summer training of one MTSS-R strategy in the spring and summer prior to the first implementation year (2020-2021). It also includes support for the provider activities associated with the readiness activities and summer training of one MTSS-R strategy. Based on the refinement of the study design and the outcome of the training provider selection, ED may decide to evaluate two MTSS-R strategies rather than one, [conduct a 3-arm study instead of a 2-arm study]. Optional Task Period I includes the recruitment necessary to increase the study sample necessary to evaluate a second MTSS-R strategy and the conduct of readiness activities and summer training of a second MTSS-R strategy in the spring and summer prior to the first implementation year (2020-2021) and support for the provider activities associated with the readiness activities and summer training of a second MTSS-R strategy. Optional Task Period I is 34 months and overlaps with the Base for all 34 months.

Optional Task Period II includes the conduct of on-going provider support to schools, support for the provider activities, and data collection for the 2020-2021 school-year associated with one MTSS-R strategy. Optional Task Period IIA includes on-going provider support to schools, support for the provider activities, and data collection for the 2020-2021 school-year associated with a second MTSS-R strategy. Optional Task II and Optional Task Period IIA are 22 months and overlap with the Base (and Optional Task Period I, if executed) by 10 months.

Optional Task Period III includes the conduct of summer training of one MTSS-R strategy and on-going provider support to schools across the second implementation year (2021-2022), support for the provider activities, and data collection associated with one MTSS-R strategy. Optional Task Period IIIA includes the conduct of summer training of a second MTSS-R strategy and on-going provider support to schools across the second implementation year (2021-2022), support for the provider activities, and data collection associated with a second MTSS-R strategy. Optional Task III and Optional Task Period IIIA are 25 months and overlap with the Optional Task Period II (and Optional Task Period IIA, if executed) by 13 months.

Optional Task Period IV includes a TWG meeting as well as data analysis, report preparation, and development of a restricted-use file that includes one MTSS-R strategy. Optional Task Period IVA includes the additional data analysis, report preparation, and restricted-use file development associated with a second MTSS-R strategy. Optional Task Period IV and Optional Task Period IVA are 24 months and overlap with the Optional Task Period III (and Optional Task Period IIIA, if executed) by 10 months.

Exhibit 1. Contract Periods of Performance

Months 1 - 34

A. Legislative Authority

This impact evaluation of training in multi-tiered systems of support for reading in early elementary school is authorized in Section 664 of the Individuals with Disabilities Education Act (IDEA, P.L. 108-446), as amended, which assigns to IES the responsibility to conduct studies and evaluations of the implementation and impact of programs supported under IDEA.

B. Background

There is great variation in young students’ attainment of foundational reading skills necessary for later literacy development as well as acquiring content knowledge in other subjects. About a third of US students fail to develop basic reading skills by 4th grade, and students with disabilities fail to develop these skills at rates much higher than their peers without disabilities (67% of students with disabilities fail to attain basic reading skills by 4th grade compared to 25% of students without disabilities) . Students who are not proficient in reading by the 3rd grade continue to perform poorly in reading in later grades and are four times less likely than proficient students to graduate from high school

Stakeholders have increasingly turned to the implementation of multi-tiered systems of support for reading (MTSS-R) with the goal of addressing these concerns. MTSS-R is a framework for providing high-quality instruction for all students, identifying students needing supplemental or more intensive supports, and providing supplemental or more intensive support for those who need it. There are several ways that the implementation of MTSS-R is consistent with the goals of the Individuals with Disabilities Education Act of 2004 (IDEA).

First, MTSS-R includes a focus on providing high-quality instruction to all students, including students with disabilities (SWDs) who are served in the general education classroom, by supporting the general education teacher as well as the collaboration between the general education teacher and special education and other staff (e.g. content specialists and aids). This is consistent with IDEA’s emphasis on providing SWDs access to the general education curriculum, in the regular classroom, to the extent possible. SWDs are most commonly identified as having a Specific Learning Disability (SLD; 39% of all students receiving special education) and students in this disability category typically experience reading challenges . Students served in the SLD category typically spend almost all of their day in the general education classroom and, in theory, would benefit from high-quality general education reading instruction.

Second, the implementation of MTSS-R is intended to promote better identification of students needing supplemental support as well as students in the category of SLD. Prior to IDEA 2004, SLD determination relied on an ‘educationally significant discrepancy’ between ability and academic achievement. Critics assert that waiting until students’ achievement fell substantially below their ability to provide intervention services essentially constitutes a ‘wait to fail’ approach that precludes students from receiving timely support. Additionally, critics argue this approach does not appropriately take into account whether a student’s low achievement is due to ineffective instruction. IDEA 2004 specifies that states cannot require districts to use the discrepancy approach and must permit districts to make this determination based on the student’s response to scientific, research-based interventions. In theory, MTSS-R would provide information on the progress of student learning while receiving a quality general education, as well as targeted supports, that could be used when determining eligibility in the SLD category as well as, potentially, more quickly identify struggling students for more intensive supports.

Finally, MTSS-R is an approach used to provide students support in order to prevent or mitigate issues that students may develop in the classroom. IDEA emphasizes that preventive supports for children and youth, supported by IDEA funds and referred to as coordinated early intervening services (CEIS), could reduce referrals for issues that could be addressed through relatively simple supports. Although CEIS is currently implemented in a small portion of districts nationally, CEIS is most common in elementary schools, in the area of literacy instruction, and often through the implementation of multi-tiered systems of support. In theory, the provision of high-quality core instruction as well as effective supplemental supports could prevent the development of more significant reading problems.

The Office of Special Education Programs at the US Department of Education (ED) has provided support for the study and implementation of MTSS-R through a number of national technical assistance centers. Almost all state education agencies encourage their districts to implement MTSS-R and districts commonly report that MTSS-R is implemented in their schools . However, a recent ED-sponsored study found that schools that report implementing MTSS-R are not fully implementing the model in ways that align with best practice . For example, only about half of the schools that reported implementing MTSS-R reported providing supplemental support for struggling readers outside the core reading block. Others replaced a portion of the core reading block with these ‘supplemental’ supports.

As there is strong support for the implementation of MTSS-R and MTSS-R is prevalent across elementary schools, it’s important to determine if the implementation of MTSS-R is effective. Moreover, because there is wide variation in how MTSS-R is being implemented, understanding the effectiveness of training in a promising MTSS-R model is necessary is critical. While studies of MTSS-R components have found promising results, rigorous evidence of the effectiveness of training in a comprehensive MTSS-R model is needed.

C. Key Research Questions

ED’s expectation is that the contractor will be responsible for ensuring that all training activities [readiness activities, trainings, and on-going support] will be offered to all schools, grades 1 and 2, assigned to treatment status starting the spring prior to the first implementation year (2020-2021) and continuing through the second implementation year (2021-2022). Data collection will include both implementation years of the study.

This study will test support to implement MTSS-R that is consistent with practices supported by research including evidence-based universal reading instruction for all students and targeted supports for struggling students as well as a system for identifying students for these supports and monitoring student progress. Should the refinement of the study design and the outcome of the training provider selection indicate that an important variation in MTSS-R strategies could inform policy and practice, ED may decide to evaluate two MTSS-R strategies rather than one (conduct a 3-arm study instead of a 2-arm study). If Optional Tasks II, III, and IV are exercised (2-arm study) the following key research questions will be addressed:

· What are the initial impacts of training in the implementation one MTSS-R strategy on school staff practice including the provision of early reading instruction and targeted supports?

· What is the impact of training in the implementation of one MTSS-R strategy on student literacy? For at-risk students?

If Optional Tasks I, IIA, IIIA, and IVA are exercised (3-arm study) the following key research questions will be addressed:

· What are the initial impacts of training in the implementation of a second MTSS-R strategy on school staff practice including the provision of early reading instruction and targeted supports?

· What is the impact of training in the implementation of a second MTSS-R strategy on early literacy? For at-risk students?

· Do impacts differ between the two MTSS-R strategies?

D. MTSS-R Training and Support Activities

The contractor shall offer readiness activities, training, and on-going support for MTSS-R implementation prior to and across two school-years: 2020-2021 and 2021-2022. ED anticipates that the contractor, through a subcontract with a MTSS-R training provider, shall train grades 1 and 2 staff in 50 elementary schools to implement one MTSS-R strategy consistent with research-based practices. If Optional Tasks I, IIA, IIIA, and IVA are exercised and a second MTSS-R strategy is evaluated (3-arm study), the contractor, through a subcontract with a MTSS-R training provider, shall train grades 1 and 2 staff in 50 additional elementary schools to implement a second MTSS-R strategy consistent with research-based practices (for a total of 100 treatment schools).

The MTSS-R strategy, supported in this study, shall include the provision of high-quality, evidence-based early reading instruction for all students (universal supports or tier I), identification of students needing supplemental supports and assignment of that support, the provision of supplemental supports for struggling students outside of the core reading block (targeted supports or tier II), and monitoring student progress. ED may decide to evaluate a second MTSS-R strategy in which one component varies from the first MTSS-R strategy. For example, the variation across MTSS-R strategies could be in MTSS-R measurement (e.g. the type or frequency of the screening / progress monitoring) or supplemental supports (e.g. a stand-alone curriculum, multiple curricula, and/or a more individualized approach).

The training may involve (1) support to school leadership and staff in order to establish the infrastructure necessary for MTSS-R implementation including establishing a school team and identifying staff for essential roles, introducing a student screening and progress monitoring system, and developing a schedule for the school day (readiness activities), (2) training for all relevant school staff to successfully implement their role(s) within the MTSS-R system including the provision of universal instruction, targeted supports, and screening and progress monitoring (training) and (3) coaching to ensure that the MTSS-R system is implemented with fidelity (on-going support).

E. Provider Selection The contractor shall be responsible for selecting the MTSS-R strategy (or strategies) to be evaluated in this study. After the contract award, the contractor shall establish a group of three experts in early literacy, measurement, and professional development (approved by the Department). In consultation with the expert panel and ED, the contractor shall develop an open solicitation and selection criteria. Then, the contractor shall release the solicitation and contact developers to ascertain their interest and ability to participate in a randomized trial of their strategy. In response to the solicitation, interested training providers will submit a description of their training program and the requirements for school staff training and implementation, along with any evidence of effectiveness. The contractor shall oversee the expert panelists’ review of proposals in order to identify up to 4 providers for inclusion in panel meetings in Washington, DC. After consideration of the COR and expert panel’s recommendations, the contractor shall select up to two strategies for inclusion in the study. The contractor shall coordinate intensive training in and support of MTSS-R implementation during the course of this contract. The contractor shall monitor the training process and facilitate, as needed, training and support between the MTSS-R training providers and the districts and schools during the evaluation.

F. Evaluation Approach This performance work statement assumes the following:

· The contractor shall, through a subcontract with a training provider, implement one MTSS-R strategy [Base, Optional Tasks II, III, and IV] across 2 years. If Optional Tasks I, IIA, IIIA, and IVA are exercised (3-arm study), the contractor shall, through a subcontract with a training provider, implement a second MTSS-R strategy (for a total of 2 MTSS-R strategies) across 2 years.

· The study shall recruit and complete data collection from a sample of 100 elementary schools and perform random assignment at the school level (all staff receiving MTSS-R training in the same school will receive the same treatment condition or will be in the control group). The contractor shall randomly assign 50 schools to MTSS-R training and 50 to a business-as-usual professional development. If Optional Tasks I, IIA, IIIA, and IVA are exercised (3-arm study), the study shall recruit an additional 50 schools (150 school total) and the contractor shall randomly assign 50 schools to one MTSS-R training, 50 schools to a second MTSS-R training, and 50 to a business-as-usual professional development. ED assumes that each elementary school will include 3 reading teachers and 60 study students at each grade level.

· The study shall be powered for an effect of no more than 0.10 on elementary school student achievement for the full sample.

· The study should include approximately 10 districts and 100 schools. If Optional Tasks I, IIA, IIIA, and IVA are exercised (3-arm study), the study should include 12-15 districts and 150 schools.

Data collection shall consist of provider logs (prior to and across the two implementation years), training content checklists completed by the contractor, teacher and reading specialist surveys in the spring of each implementation year-school, student records data, progress monitoring system data (treatment schools only; each implementation year), and student testing data in the fall of the first implementation year and the spring of each implementation year. Please see Task 6.

Specific Design Issues

Discussed below are a number of factors that will affect the evaluation design or successful execution of the study. Other design issues may arise as the evaluation is carried out that will need to be resolved by the study team.

· Random Assignment of Schools. The offeror shall address plans for carrying out random assignment by school.

· MTSS-R strategy and training. ED understands that there are various ways that MTSS-R is implemented. The offeror shall identify promising ways of implementing individual components of MTSS-R and/or approaches to MTSS-R implementation, overall, that are most likely to impact student achievement. In addition, the offeror shall identify key variations in MTSS-R implementation that are policy-relevant, potentially focused on a component(s) of MTSS-R for which there is limited research available to guide practice.

· Counterfactual. Schools (grades 1 and 2) are anticipated to be randomly assigned to be MTSS-R training and support activities or not. A key issue is the nature of support experienced by staff of the control schools, the counterfactual for those not assigned to the treatment/intervention.

· Targeted Districts/Schools. The offeror shall consider what types of districts and schools to consider for inclusion in the study and that is relevant to federal education policy. The offeror shall fully explain any assumptions made.

· Sampling of School Staff for Survey Administration. It is anticipated that the study will survey grade 1 and 2 reading teachers in the study schools, once in year 1 and once in year 2. The pros and cons of which teachers to include in each of these data collection efforts should be considered by the offeror. The offeror shall fully explain any assumptions.

· Sampling of Students for Individual Testing in Reading. It is anticipated that the study will conduct individual student testing of all first and second grade students in the fall of the first implementation year (2020-2021) and a sample of first and second grade students in the spring of the first implementation year (2020-2021) and the spring of the second implementation year (2021-2022). The offeror shall fully justify the grade-level and sample of students tested at each administration of student testing. The offeror shall fully explain any assumptions made in considering this issue. In addition, offerors are encouraged to fully discuss what can be learned from this evaluation effort given that the study incudes data collected across multiple grades and multiple years.

· Power Analyses. The study should be powered for an effect of .10 on school student achievement for the full sample. The contractor shall fully explain power calculation assumptions.

II. Scope of Work This section specifies the tasks and subtasks that the contractor shall perform. The Contracting Officer’s Representative (COR) reserves the right to inspect the contractor’s work while in progress. The contractor shall submit to the COR any progress report or other written documentation of work to date that the COR requests to facilitate that inspection, in accordance with FAR 52.246-4 (Inspection of Services- Fixed Price) and FAR 52.246-5 (Inspection of Services- Cost-Reimbursement).

A. Base Phase

Task 1—Opening Meeting with ED

The contractor’s project director and up to three project staff shall meet with the Contracting Officer’s Representative (COR) and other appropriate ED staff within 15 business days after the effective date of the contract award to discuss details regarding the tasks outlined in the proposal, study design, scheduling activities, and other issues related to the contract. This meeting shall be at ED's offices in Washington, DC. The contractor shall provide an agenda for the meeting to the COR within 10 business days after the effective date of the contract. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within 2 weeks after the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the study.

Deliverables:

Kickoff meeting agenda

Opening Meeting summary

Task 2 –Request for Proposals and Provider Selection Announcement and Request for Proposals (RFP). The contractor shall prepare a draft announcement specifying that proposals will be requested to train schools in the implementation of MTSS-R and including the expected timelines for the proposal process as well as a draft request for proposals (RFP) for training providers including the timelines for proposals, requirements for the training, and the criteria by which the proposals shall be evaluated. The RFP shall also specify the maximum amount of funding that will be available to support the training. The contractor shall submit the draft announcement and RFP to the COR for review within 6 weeks after award. The contractor shall make revisions and submit a revised announcement and RFP within 2 weeks of COR comment and send the RFP to the expert panelists for review. The contractor shall make any necessary revisions within two weeks of receiving expert panel comments. After COR approval, the contractor shall widely disseminate the announcement and, subsequently, the RFP. The RFP shall be released by 1/2/19.

Review of Proposals and Selection of Interventions. Within 3 business days of closing the request for proposals, the contractor shall send proposals to expert panel members for review. The contractor shall compile expert panel members’ reviews of proposals and recommendations of proposals for inclusion in the panel meetings in DC within one week of receiving reviews. The contractor shall arrange a meeting with expert panel members and developers of each of up to four providers recommended by the expert panel. The meeting shall be one or two days and shall include time for each provider to present their training plan and answer any questions from the panel, contractor, and/or ED. The contractor shall plan to arrange and pay for up to two staff for each provider to travel to DC, present on their training plans, and answer questions that the expert panel, the contractor, or ED may have. The contractor shall compile expert panel members’ reviews of the presentations/proposals within one week of receiving expert panel reviews. The contractor shall negotiate with and secure the full cooperation of the recommended provider(s). The contractor shall submit subcontract(s) with the selected provider by 4/16/19.

Deliverables:

Draft advance announcement and RFP Revised advance announcement and RFP

Final advance announcement and RFP

Compilation of expert panel reviews for inclusion in panels

Compilation of expert panel reviews for selection of provider

Contracts with providers

Task 3 – Expert Panel for the Selection of Training Provider

Expert Panel for the Selection of Training Provider. Based on names submitted in the proposal and in consultation with the COR, the contractor shall establish an expert panel of three persons to advise on the selection of training providers. These experts may also serve on the technical working group described in Task 5. The contractor shall submit a list of proposed expert panel members to the COR no later than 2 weeks after the Opening Meeting (Task 1). The contractor shall contact each proposed member and obtain and submit letters of commitment and documentation of consulting rates to the COR within 3 weeks of COR approval of the proposed expert panel list.

Review of the Solicitation and Proposal Review Criteria. The contractor shall request that each member of the expert panel review the contractor’s draft of the MTSS-R training provider request for proposals (RFP) and the criteria by which the proposals shall be evaluated. The contractor shall request that expert panelists provide recommended changes within 2 weeks of receiving the draft RFP with evaluation criteria, prepared under Task 2.

Selection of Training Provider(s). The contractor shall request that each member of the expert panel review the provider proposals and write a brief review of each proposal, including each provider’s strengths and weaknesses, along with recommendations of up to four training providers for inclusion in the panel meetings. The contractor shall provide the reviews and recommendations of these experts to the COR within 2 weeks of receiving the provider proposals. Expert panel members will participate in a one- or two-day meeting, organized by the contractor, with up to four providers in which each provider makes a short presentation of their training plans and responds to questions from expert panel members. Following the panel meetings, the contractor shall request that each panel member write a brief review of each proposed training plans (including each professional development program’s strengths and weaknesses) along with a recommendation of which trainings to select for inclusion in the evaluation. The contractor shall provide the reviews (and recommendation) of these experts to the COR within 2 weeks of the panel meeting(s).

List of proposed expert panel members

Letters of commitment from proposed expert panel members

Expert panel review of solicitation and criteria Expert panel review of proposals for inclusion in the panels Expert panel review of proposals post-panel and recommendation of provider(s) Task 4—Refine Study Design

The contractor shall revise the study design submitted in the proposal to include the study design, data collection, and data analyses. The draft study design report shall be submitted 4 weeks prior to the first TWG meeting. The contract shall make revisions and submit a final study design within 3 weeks after the first TWG meeting.

Draft study design report.

Final study design report.

Task 5—Technical Working Group for the Evaluation Establish Technical Working Group (TWG). The role of the TWG will be to advise the contractor and ED on the conduct of the study including, but not limited to: study design, instrumentation, data collection and analysis, and draft study reports. The TWG shall be comprised of at least 8, and up to 10 individuals, and should include experts in MTSS-R policy, training, and implementation as well as experts in evaluation methodology. The contractor shall submit a list of proposed TWG members no later than two weeks after the Opening Meeting (Task 1). When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. After the COR approves the proposed TWG members, the contractor shall contact each proposed member and, within 3 weeks of COR approval, submit letters of commitment as well as documentation of consulting rates for the proposed panelists to the COR and CS for approval by the CO.

Convene Technical Working Group. The contractor shall convene 1 meeting of the TWG in the base contract. The meeting shall be held no later than 8 months after the contract award and shall focus on remaining design issues, data collection, and analysis issues. No later than 3 weeks before the meeting, the contractor shall submit to the COR draft materials for the meeting, including an agenda, presentation materials, and other materials for the panel’s review. Final materials shall be fully responsive to COR comments on the draft materials. After COR approval, and no later than 1 week before the meeting, the contractor shall send the approved materials to the panel members.

After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S. Department of Education facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

The contractor shall provide copies of all reviews of products by the panel to the COR within five work days of receipt by the contractor. If TWG members recommend any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review products, as appropriate, when group meetings are not required.

List of proposed TWG members

Letters of commitment and consulting rates

Draft meeting materials

Final meeting materials

Summary of TWG meeting Task 6– Prepare OMB Package and Develop Data Collection Forms

The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments. The package shall include a description of the study design, site recruitment, and data collection instruments, as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions. In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the study and its potential use for policy decisions. The contractor shall submit a draft forms clearance package to the COR no later than 6 weeks after the TWG meeting in Task 5. The COR will submit the accepted forms clearance package for clearance. Based on comments from internal and external clearance (after the first 60-day clearance process) the contractor shall make all needed revisions and submit a revised OMB clearance package. The COR will then submit the accepted revised clearance package to OMB for the final 60 days. Data collection shall consist of:

· Provider logs - These forms, completed by the provider, will provide information on the provision of training and on-going support activities and shall be completed after each training activity for the duration of the MTSS-R training. The contractor shall also collect any fidelity or other assessments that the provider may conduct as part of their on-going support.

· Support, training, and coaching content checklists - These forms, to be completed by the contractor, will include data on support, training, and coaching content, and shall be collected for a sample of MTSS-R support, training, and coaching activities.

· Teacher surveys - These surveys will provide data on teacher characteristics, MTSS-R training and implementation, and school climate. These surveys shall be administered to a sample of teachers in the spring of the first year of implementation as well as the spring of the second year of implementation.

· Reading specialist surveys – These surveys of all reading specialists will provide data on reading specialist characteristics, MTSS-R activities, as well as other relevant information. These surveys shall be administered in the spring of the first and second study years. Alternate staff may be surveyed depending on staff roles in MTSS-R implementation.

· Student records data – These shall include data on student characteristics, attendance, special education eligibility, and educational placement (extent of inclusion), as well as other relevant information. These data shall be collected after each implementation year.

· Progress monitoring system data – These data will be generated from the screening and progress monitoring system used as part of MTSS-R implementation and will be available for the treatment schools only. These data shall be collected at the end of each implementation year.

· Student testing data in grades 1 and 2 - The contractor shall individually-administer an adaptive student achievement test in reading for students in the study sample in grades 1 and 2 consistent with discerning an effect size of .10. The contractor shall identify a reading test that is valid and reliable for the purposes of this evaluation. The contractor shall administer the test to all students in the fall of the first implementation year and a sample of students in the spring of each implementation year.

The contractor shall design data collection instruments to address the key study questions. The contractor shall make use of or refine existing instruments to the extent possible and applicable.

For instruments where validity or reliability are not yet well established, the contractor shall pilot test each of the instruments with no more than nine respondents per instrument in no less than two schools to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective. Piloting is to be conducted prior to the final 60 days in the OMB submission process. The contractor shall ensure that instruments are revised as needed based on pilot test results.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full-scale data collection with the IRB approval obtained by districts from their own institutions. Specifically, the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Draft forms OMB clearance package

Revised forms OMB clearance package

Final forms OMB clearance package

Documentation of IRB approvals

Task 7— Selection and Recruitment of Study Districts and Schools During the selection and recruitment process, the contractor shall frequently update the COR. The contractor shall provide monthly selection and recruitment updates in the monthly progress reports (see Task 10). The contractor shall, in consultation with the COR, develop a strategy for selecting study sites and prioritize potential sites based on criteria that the COR and contractor agree are pertinent to addressing the evaluation objectives. These shall include policy-relevant districts with capabilities sufficient to support the evaluation’s needs and a willingness to participate in the MTSS-R training and implement MTSS-R. The contractor shall reach out to the potential sites and follow-up with telephone calls, visitations, and the sharing of written materials that provide accessible details about the study, as necessary, to gauge the site’s willingness to implement MTSS-R, and to have the contractor randomly assign schools to different study conditions.

Based on the exploratory contacts with potential sites and sample selection criteria developed in consultation with the COR, the contractor shall recruit study districts and schools to participate in and cooperate with the study. The contractor shall secure final agreements to participate through signed memoranda of understanding so that a total of approximately 10 districts and 100 schools are in the study. The contractor shall use the recruited study sample to randomly assign schools to the treatment and control groups according to the study design.

The contractor shall complete all recruitment activities no later than 16 months after contract award.

As part of the recruitment process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Deliverables: Memoranda of Understanding from districts.

Task 8 – Conduct of Readiness Activities and Summer Training of One MTSS-R Strategy Prior to Year 1 Implementation (2020-2021) The contractor, under a subcontract with a MTSS-R training provider, shall prepare and train 50 elementary schools in the implementation of MTSS-R in grades 1 and 2 in the spring and summer prior to the first implementation year (2020-2021).

Task 9—Support for and Monitoring of Readiness Activities and Summer Training of One MTSS-R Strategy Prior to Year 1 Implementation (2020-2021) The contractor shall work with the MTSS-R training provider to design/develop and implement a logistics and coordination plan to ensure that all MTSS-R readiness activities and trainings are conducted as planned prior to the 2020-2021 school year.

The contractor staff overseeing and logistically supporting training providers shall be separate from the evaluation team. The contractor shall work with the provider and study districts and schools to arrange all MTSS-R readiness, training, and support activities. The contractor shall arrange and pay for all expenses, including any school staff time in MTSS-R readiness, training, and support activities outside staff contract time as well as all coach time in MTSS-R training and implementation activities, and ensure that any necessary staffing and scheduling arrangements are made with the district to support the training and implementation of MTSS-R. The contractor shall provide monthly training updates in the monthly progress reports (Task 10) beginning 18 months after contract award through the completion of the summer training prior to year 1 implementation.

Task 10 -- Reporting Requirements In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the CO and one copy to the COR, simultaneously with the monthly invoice. The Base includes performance reports for months 1 through 34.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, staff use, and changes in personnel and their security designation. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report B. Optional Task Period I [Readiness Activities and Summer Training of a Second MTSS-R Strategy Prior to Year 1 Implementation (2020-2021) Support for the Readiness Activities and Summer Training] Task 11— Selection and Recruitment of an Additional 2-5 Districts and 50 Schools During the selection and recruitment process, the contractor shall frequently update the COR. The contractor shall provide monthly selection and recruitment updates in the monthly progress reports (see Task 10). Consistent with the sample selection criteria developed in consultation with the COR in Task 7, the contractor shall recruit an additional 2-5 study districts and 50 schools to participate in and cooperate with the study. The contractor shall secure final agreements to participate through signed memoranda of understanding so that a total of approximately 12-15 districts and 150 schools are in the study sample. The contractor shall use the recruited study sample to randomly assign schools to the treatment and control groups according to the study design.

The contractor shall complete all recruitment activities no later than 16 months after contract award.

As part of the recruitment process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Deliverables: Memoranda of Understanding from districts.

Task 12 – Conduct of Readiness Activities and Summer Training of a Second MTSS-R Strategy Prior to Year 1 Implementation (2020-2021) The contractor, under a contract with a MTSS-R training provider, shall prepare and train 50 elementary schools in the implementation of a second MTSS-R in grades 1 and 2 in the spring and summer prior to the first implementation year (2020-2021).

Task 13 — Support for and Monitoring of Readiness Activities and Summer Training of a Second MTSS-R Strategy Prior to Year 1 Implementation (2020-2021) The contractor shall work with the MTSS-R training provider to design/develop and implement a logistics and coordination plan to ensure that all MTSS-R readiness activities and trainings of a second MTSS-R strategy are conducted as planned prior to the 2020-2021 school year.

The contractor staff overseeing and logistically supporting training providers shall be separate from the evaluation team. The contractor shall work with the provider and study districts and schools to arrange all MTSS-R readiness, training, and support activities. The contractor shall arrange and pay for all expenses, including any school staff time in MTSS-R readiness, training, and support activities outside staff contract time as well as all coach time in MTSS-R training and implementation activities, and ensure that any necessary staffing and scheduling arrangements are made with the district to support the training and implementation of MTSS-R. The contractor shall provide monthly training updates in the monthly progress reports (Task 10) beginning 18 months after contract award through the completion of the summer training prior to year 1 implementation.

C. Optional Task Period II [Remaining Training and On-going Provider Support of One MTSS-R Strategy to Schools, Support for Provider Activities, and Data Collection across the First Implementation Year (2020-2021)] Task 14 – Conduct of On-going Provider Support of One MTSS-R Strategy to Schools across the First Implementation Year (2020-2021) The contractor, under a contract with a MTSS-R training provider, shall provide on-going support to 50 elementary schools in the implementation of one MTSS-R strategy in grades 1 and 2 in across the first implementation year (2020-2021).

Task 15— Support for and Monitoring of Provider Activities across the First Implementation Year (2020-2021) The contractor staff overseeing and logistically supporting training providers shall be separate from the evaluation team. The contractor shall work with the provider and study districts and schools to arrange all MTSS-R readiness, training, and support activities. The contractor shall arrange and pay for all expenses, including any school staff time in MTSS-R readiness, training, and support activities outside staff contract time as well as all coach time in MTSS-R training and implementation activities, and ensure that any necessary staffing and scheduling arrangements are made with the district to support the training and implementation of MTSS-R. The contractor shall provide monthly training updates in the monthly progress reports (Tasks 10 and 17) beginning 24 months after contract award through the completion of the MTSS-R Year 1 training and implementation.

Task 16— Data Collection First Implementation Year (2020-2021) Using instruments approved by OMB, the contractor shall collect data (listed in Task 6) to estimate the impact of the MTSS-R training on the outcomes of interest and to answer the evaluation questions around one MTSS-R strategy across year 1 implementation.

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf). As part of the monthly reporting requirement (Tasks 10 and 17), the contractor shall provide ongoing information about the data collection activities.

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set. The data monitoring activities shall be in place within 16 months after contract award. During data collection, the contractor shall include with the monthly progress reports updates on the progress of data collection based upon the data monitoring activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program. The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide information on these procedures to the Government upon request. When possible, individuals collecting and/or coding data shall have no knowledge of each school’s treatment condition.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances…