Attachment_A_Draft_PWS.docx

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance

DRAFT Performance Work Statement

IMPACT EVALUATION OF DEPARTMENTALIZED INSTRUCTION

IN ELEMENTARY SCHOOLS

INTRODUCTION

The U.S. Department of Education’s (Department) Institute of Education Sciences (IES) will evaluate the impact of departmentalized instruction in elementary schools on student achievement. More specifically, the evaluation will use a random assignment design to examine the impact of switching from self-contained instruction (teachers teach all subjects to a fraction of students in a particular grade) to departmentalized instruction (teachers teach one subject to all students in a particular grade). The evaluation will focus on math and reading in the upper elementary grades (4 and 5), with an emphasis on schools that serve a high percentage of disadvantaged students and are in need of improvement.

THE EVALUATION

A. Legislative Authority

This evaluation is authorized by Title VIII Section 8601 of the Elementary and Secondary Education Act (ESEA) as amended by the Every Student Succeeds Act, which allows the Department to pool resources across ESEA programs to evaluate the impact of individual federal education programs that currently lack sufficient evaluation dollars or various strategies that cut across a wide range of ESEA programs.

B. Background

There is a growing need to identify school improvement strategies that work. Previous large-scale efforts such as the federal School Improvement Grants program, which provided over $7 billion between 2009 and 2016 to schools that agreed to implement a number of whole-school improvement models and strategies, have not demonstrated evidence of widespread success.[footnoteRef:1] Similarly, while improving the quality of classroom instruction is widely viewed as a key school improvement strategy, much of the evidence on the effectiveness of providing professional development to teachers has been disappointing despite billions of dollars in public investment each year.[footnoteRef:2] The Every Student Succeeds Act (Pub.L. 114-95), which was signed into law in December 2015 as the reauthorization of ESEA, gives states and districts substantial flexibility in deciding how to improve their low-performing schools, provided the improvement strategies are “evidence-based.” This requirement highlights the need to continue searching for improvement strategies that show evidence of effectiveness. [1: Dragoset, L., Thomas, J., Herrmann, M., Deke, J., James-Burdumy, S., Graczewski, C., Boyle, A., Upton, R., Tanenbaum, C., & Giffin, J. (2017). School Improvement Grants: Implementation and Effectiveness (NCEE 2017-4013). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.] [2: Garet, M., Heppen, J., Walters, K., Smith, T., & Yang, R. (2016). Does Content-Focused Teacher Professional Development Work? Findings from Three Institute of Education Sciences Studies (NCEE 2017-4010). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.]

One potentially promising strategy is using departmentalized instruction instead of instruction in self-contained classrooms by one teacher. For example, if there are two fourth-grade teachers in a school, instead of having both teachers teach all subjects (self-contained), one teacher only teaches math while the other teacher only teaches reading (departmentalized). This approach is common in middle and high school but not in elementary school, even though there may be good reasons to specialize. Fractions and rational number content are typically introduced in fourth and fifth grade. Because these topics are foundational for later math courses such as algebra, it is critical for students to receive effective instruction on these topics. However, teaching these topics is particularly challenging since they are complex and since elementary school teachers tend to have a more limited math background.[footnoteRef:3] Departmentalization thus provides a potential avenue to address this challenge because in each school, some teachers may have a comparative advantage over their fellow teachers in teaching certain subjects. Departmentalization may be particularly appealing because it is likely a low-cost strategy that aims to better allocate existing teaching resources, rather than infuse more resources. [3: Garet, M., Heppen, J., Walters, K., Parkinson, J., Smith, T., Song, M., Garrett, R., Yang, R., & Borman, G. (2016). Focusing on Mathematical Knowledge: The Impact of Content-Intensive Teacher Professional Development (NCEE 2016-4010). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.]

Even though a number of elementary schools in the nation have begun switching from traditional self-contained classrooms to departmentalized instruction, there is little convincing evidence on whether one model is more or less effective than the other.[footnoteRef:4] To date, there has been only one random assignment study of departmentalized instruction, which found small negative impacts on reading and math achievement for participating public elementary schools in Houston, Texas.[footnoteRef:5] The impacts were more negative for special education students and students with younger teachers, although it is not clear why. Further evaluation is needed to see if these findings extend to other contexts and to explore the possible mechanisms behind these impacts. [4: Two recent nonexperimental studies include: (1) Taylor-Buckner, N. (2014). The Effects of Elementary Departmentalization on Mathematics Proficiency. Doctoral Dissertation, Columbia University. (2) Strohl, A., Schmertzing, L., Schmertzing, R., & Hsiao, E. (2014). Comparison of Self-Contained and Departmentalized Elementary Teachers’ Perceptions of Classroom Structure and Job Satisfaction. Journal of Studies in Education. 4(1), 109-127. ] [5: Fryer, R. (2016). The ‘Pupil’ Factory: Specialization and the Production of Human Capital in Schools. NBER Working Paper 22205. http://www.nber.org/papers/w22205.]

Findings from the Department’s evaluation will provide relevant information for educators and administrators considering how best to deploy their existing team of teachers to maximize student learning. The evaluation findings will also help state policymakers make an informed decision about whether an improvement strategy such as departmentalized instruction is worth recommending for their low-performing elementary schools.

C. Key Evaluation Questions

The primary research question for this evaluation is as follows:

· What is the impact of switching from self-contained instruction to departmentalized instruction in upper elementary school (grades 4 and 5) on student achievement in math and reading?

The evaluation has an optional task that if exercised will answer the following additional research question:

· What is the impact of switching from self-contained instruction to departmentalized instruction in upper elementary school (grades 4 and 5) on instructional quality, such as the quantity and quality of student-teacher interactions?

D. Evaluation Approach

The primary purpose of the Department’s evaluation is to assess the impact on student achievement of switching from self-contained instruction to departmentalized instruction. To estimate the causal impacts of this strategy, the evaluation will use a random assignment design. Participating elementary schools will be randomized within each district to either switch from self-contained to departmentalized instruction for two years in fourth and fifth grade (treatment condition) or to continue self-contained instruction (control condition).

The evaluation sample will prioritize low-performing elementary schools that serve a high percentage of disadvantaged students, so that the findings are relevant for schools where there is a pressing need to find effective improvement strategies.

Data collection will consist of administrative data on student achievement and any other relevant outcomes that are available from existing data sources, such as attendance and discipline (at baseline and at the end of each intervention year); administrative data on student, teacher, and school characteristics (at baseline); and a teacher survey (at the end of the first intervention year). There is also an optional task that if exercised will involve collecting data on instructional quality from classroom observations (in the spring of the first intervention year).

E. Design Issues

Several considerations need to be taken into account in designing this evaluation, including:

· Assigning teachers to subjects, selecting schools, and securing support from relevant stakeholders. The process for deciding which teachers will teach which subject in each school is a critical component of the departmentalization strategy. This process should be empirically based with clear and transparent assignment rules. The preferred approach is to use value-added measures, but other measures may also warrant consideration. Although the focus of the evaluation is on math and reading, the evaluation may need to account for teacher assignments to subjects other than math and reading, and determine whether it is necessary to impose requirements on the number of participating teachers in each school. It will be important to select schools where there are likely to be gains to specialization. Such schools will need to have teachers with a well-documented and substantial comparative advantage over their fellow teachers (i.e., one teacher is relatively much stronger in teaching math while the other is relatively much stronger in teaching reading). Since the departmentalization strategy involves altering the subjects that teachers will teach, close attention must be paid to securing any necessary approvals from key stakeholders, such as principals, teachers, and teacher unions. Teachers must also have sufficient notice of what condition they are assigned to, so that they have time to prepare for any changes in what they will teach. Careful consideration of the sequencing of these activities as part of the recruitment process, randomization, and setup of the departmentalization intervention will be vital to successfully carrying out the evaluation.

· Outcomes other than math and reading achievement. While the primary outcome of interest for the evaluation is math and reading achievement, other outcomes may be of intrinsic interest or relevant to understanding the mechanisms for the student achievement impacts. The hypothesis is that departmentalized instruction may improve student achievement because teachers can provide relatively higher-quality instruction in the subject that they specialize in. Consideration is needed on whether there are existing instruments that can yield valid and reliable measures of these potential changes in instructional quality. While departmentalization may increase the total number of students that a given teacher impacts, it may also reduce the amount of time that a given teacher interacts with each student. This may have consequences for other aspects of instruction or related student-teacher relationships, which may then impact achievement or other types of outcomes such as attendance and disciplinary incidents. Consideration is needed on whether there are cost-efficient ways to validly and reliably assess potential impacts of departmentalization on these non-achievement outcomes.

· Analytic and attrition considerations. The evaluation will focus on measuring the impact of departmentalization on the cohort of students that are in fourth grade during the first year of departmentalization. The evaluation will follow this cohort into fifth grade during the second year of departmentalization. Because this is a longitudinal cohort, attention to relevant analytical issues such as attrition and mobility is particularly important. Fifth-grade students in treatment schools during the first year of departmentalization will receive departmentalized instruction, but their outcomes will not be measured. Similarly, fourth-grade students in treatment schools during the second year of departmentalization will receive departmentalized instruction, but their outcomes will not be measured.

· The need to develop and finalize evaluation plans in a short timeframe. To be ready for implementation during the 2018-19 school year, it will be necessary for evaluation plans to be detailed and finalized in a very short timeframe.

· Power Analyses and random assignment. To determine the appropriate sample size for the evaluation, it will be necessary to: (1) determine the most appropriate unit of random assignment for this type of intervention; (2) determine a defensible target minimum detectable effect size for this type of intervention; (3) carry out a power analysis with assumptions that are reasonable for the expected context of the evaluation. The Department currently anticipates that approximately 200 schools will be randomly assigned in order to detect a 0.1 standard deviation impact on student achievement.

F. Period of Performance

The period of performance will consist of an 18-month base period and two option periods (of varying terms) that would result in a 48-month period contract in total.

The Base Period will include finalizing the design of the evaluation, recruiting the sample of schools, developing the data collection instruments and Office of Management and Budget (OMB) forms clearance package, and a Technical Working Group (TWG) meeting. Based on successful recruitment of schools and an acceptable plan for delegating teaching assignments in schools that are assigned to departmentalize, the Department will decide whether or not to exercise Option Period I by September 2018.

Option Period I is 24 months and overlaps with the base period by 7 months. Option Period I will include random assignment of schools to treatment and control conditions, securing teaching assignments for schools in the treatment condition, supporting the implementation of departmentalized instruction for two school years, and collecting baseline and outcomes data. Based on the successful conduct of these activities, the Department will decide whether or not to exercise Option Period II by September 2019.

Option Period II is 20 months and overlaps with Option Period I by 7 months. Option Period II will include analyses of data collected for the evaluation, a TWG meeting to discuss findings, a report, and a data file.

SCOPE OF WORK

A. Base Period (All Tasks are Cost Reimbursement)

Task 1—Kick-off Meeting with the Department and Base Period Progress Reports

1.1 Kick-off Meeting: The contractor’s project director and up to three key project staff members shall meet with the Department’s Contracting Officer (CO), Contract Specialist (CS), Contracting Officer’s Representative (COR), and other appropriate Department staff within 1 week after contract award. This meeting shall be held at the Department’s offices in Washington, DC. The contractor shall submit an agenda to the COR at least 2 working days before the meeting that includes discussion of the base period tasks. The COR will review and provide feedback within 1 working day. In the meeting, the contractor shall be prepared to discuss the scheduling of activities, to identify areas of concern, to suggest ways of addressing potential problems, and to review other issues related to the contract. Within 1 week after the meeting, the contractor shall submit to the COR a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed as part of the contract.

1.2 Monthly Progress Reports: The contractor shall prepare monthly progress reports due to the COR no later than the 15th of each month. In these reports, the contractor shall summarize the major activities and accomplishments for the prior calendar month. In addition, the contractor shall provide information for each project task regarding significant events and/or findings, problems encountered, and staff use. The contractor shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, and identify and discuss significant deviations from the management plan. If there are deviations, the contractor shall describe the plan for resolving the problems and shall identify and discuss any decisions needed from the COR. The COR will provide a security personnel spreadsheet that the contractor shall update each month and attach to every monthly progress report. In this spreadsheet, the contractor shall provide all requested information, including all contractor staff members who currently or previously worked on the contract, their position risk levels, and the status of any security clearances required.

1.3 Monthly Manpower Reports: The contractor shall prepare monthly manpower reports due to the COR no later than the 15th of each month. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the prior calendar month, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming calendar month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Task 2—Technical Working Group (TWG) Formation

The contractor shall form a group of experts (the TWG) who will provide advice and guidance on the evaluation design, specific research questions that the evaluation will address, implementation of departmentalized instruction, data collection plans, and analysis plans.

2.1 Bios for Recommended Participants: The contractor shall recruit a team of six TWG members whose expertise areas include (but are not limited to): teacher assignment policies, teacher value-added methods, and experimental evaluation methods. Within 2 weeks after contract award, the contractor shall submit a list of potential TWG members that includes a brief description of each proposed expert’s relevant credentials and area of expertise, and details how the expert’s credentials and expertise align with the evaluation’s needs. The COR will review and approve the list and/or provide comments within 1 week of receipt of the list.

2.2 Letters of Commitment: After the COR approves the proposed TWG members, the contractor shall contact each proposed member to request a signed letter of commitment. The contractor shall submit signed letters of commitment to the COR no later than 6 weeks after the contract has been awarded.

Task 3—Forms Clearance Package

The contractor shall prepare all forms and documents necessary for OMB clearance and shall coordinate its own Institutional Review Board (IRB) approval.

3.1 Draft OMB Package: The contractor shall prepare a draft OMB forms clearance package that includes all required forms, calculations of participant burden, and data collection materials and instruments. The required forms are the Information Collection Submission Worksheet, the Paperwork Reduction Act Submission Worksheet (Part II: Information Collection Detail), Supporting Statement Part A, and Supporting Statement Part B. The package shall incorporate: (1) a description of the evaluation’s general purpose and potential value to the field of education, (2) the evaluation’s primary research questions, (3) a description of the intervention and evaluation design, (4) a description of and justification for any incentives, (5) a description indicating how data will be tabulated and analyzed, and (6) copies of all data collection instruments.

The first draft of the OMB package shall be submitted to the COR no later than 15 weeks after contract award. The COR will provide feedback within one week, and the contractor shall make any revisions necessary to prepare the OMB package for initial submission 18 weeks after contract award.

3.2 Revised OMB Package: Based on comments received after internal and external review of the initially submitted OMB package -- gathered during the first 60 calendar days of the clearance process -- the contractor shall make all needed revisions and submit a revised OMB package to the COR within 2 weeks of the end of the first 60-day comment period. The contractor shall be prepared to make one additional round of revisions to the OMB package within 2 weeks of the end of the second 60-day OMB review period, as required. The contractor shall build a minimum of 120 calendar days into the schedule for the entire clearance process.

3.3 Documentation of IRB Approval: The contractor shall coordinate its own IRB approval for the data collection, ensuring that all aspects of the evaluation have been approved. The contractor shall submit documentation of IRB approval to the COR no later than 36 weeks after the contract award.

Task 4—Evaluation Design

The contractor shall refine and finalize the evaluation design such that the result is a comprehensive plan for how all aspects of the evaluation will be conducted.

4.1 Draft Design Report: The contractor shall prepare a draft report that includes each of the following components:

(a) Description of the intervention and evaluation research questions

(b) Randomization plan and power analysis: The report shall describe how schools will be randomly assigned to treatment and control conditions, as well as how teaching assignments within schools will be determined for schools assigned to the treatment condition. The power analysis shall outline the key contrasts for the evaluation, what the appropriate, policy-relevant minimum detectable effect sizes are, and what sample sizes are needed.

(c) Recruitment plan: The report shall provide detailed plans for ensuring that the recruited evaluation sample is sufficiently large to meet the target minimum detectable effect size. The report shall also provide detailed plans for identifying and recruiting schools that have teachers with sufficiently large relative differences in how effective they are at teaching reading and math. The report shall discuss strategies to ensure that the resulting evaluation sample is policy relevant and includes low-performing elementary schools with a high percentage of disadvantaged students. The report shall detail how the contractor will secure any necessary agreements from districts, schools, teachers, teacher unions, and any other relevant stakeholders. The report shall describe what steps the contractor will take to ensure that attrition and mobility of teachers, students, and schools are minimized, to the extent possible.

(d) Data collection plan: The report shall detail what outcome data will be collected from administrative records, when the data will be collected, and how the data will be collected to ensure that the evaluation’s research questions can be adequately answered. The report shall describe the purpose of the teacher survey, as well as the timeline and procedures for administering it to ensure high response rates and resulting data that will help address the evaluation’s research questions. The report shall detail plans for collecting baseline data from administrative records for the purposes of examining baseline equivalence and subgroup analyses. The report shall include as attachments drafts of all instruments discussed in the data collection plan.

(e) Analysis approach: The report shall include a basic plan for what statistical analyses the contractor will conduct using the data to answer the evaluation’s research questions, including appropriate estimating equations. The plan shall focus on questions related to impacts on outcomes, but also include a discussion of how the contractor will use the teacher survey for purposes such as describing implementation and exploring possible mechanisms for the observed impacts.

The contractor shall electronically submit the draft design report to the COR no later than 6 weeks after the contract has been awarded. The COR will provide feedback within 2 weeks.

4.2 Materials for Sharing Design Plans with TWG and Documenting the TWG’s Feedback: The draft design report (Task 4.1) shall serve as the basis for the PowerPoint slide presentation that the contractor shall prepare in order to solicit guidance from the TWG. At least 3 weeks before the first TWG meeting (Task 5), the contractor shall submit to the COR the following: (1) A set of PowerPoint slides that include proposed discussion questions for the TWG; and (2) A meeting agenda. Within 1 week after the first TWG meeting, the contractor shall submit a written summary memo to the COR detailing the TWG’s key recommendations, remaining issues, and the contractor’s proposed actions to address the TWG’s recommendations. The COR will provide feedback within 1 week.

4.3 Final Design Report: The contractor shall provide a final design report that is a revision of the draft design report, which fully incorporates all TWG, OMB, and COR feedback from Tasks 3.1, 3.2, 4.1, and 4.2. The contractor shall submit the final design report to the COR no later than 48 weeks after contract award.

Task 5—Technical Working Group (TWG) Meeting #1

The contractor shall convene one full-day meeting of the TWG no later than 14 weeks after the contract has been awarded. The meeting shall focus on the implementation strategy for departmentalized instruction, as well as the evaluation design, data collection, and analysis plans. This meeting shall be held at the Department’s offices in Washington, DC, and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build a coffee break and a lunch break into the agenda.

Task 6—Recruit Evaluation Sample

The contractor shall implement the recruitment plan developed in Task 4, and ensure that a sufficient sample size (with the appropriate characteristics defined in Task 4) is identified and recruited to participate in the evaluation.

6.1 Recruitment Materials: For recruitment purposes, the contractor shall prepare a short brochure describing the evaluation that is suitable for distribution to a non-technical audience, including district administrators, principals, and teachers. The contractor shall also include a recruitment letter geared towards districts that include general information on the evaluation, as well as specific information on the importance of the evaluation, its purposes, products, timelines, and the expected benefits from participation.

The contractor shall submit a draft brochure, letter, and all other notification materials to the COR for review no later than 8 weeks after contract award. The COR will provide feedback within 1 week, and the contractor shall make any revisions necessary in order to begin outreach to prospective districts. The contractor shall send the materials to districts/schools and follow-up with the necessary telephone calls and/or site visits to secure participation in the evaluation.

6.2 Memoranda of Understanding (MOU): The contractor shall establish and sign a formal MOU with each participating school district. The MOU shall indicate the roles, responsibilities, and expectations of each party, including but not be limited to the following topics: the intervention; the proposed evaluation design and work plan; random assignment procedures; implementation of the intervention; data needs; compensation to offset financial burdens associated with participation in the evaluation. The MOU shall include a specific agreement regarding the sharing of administrative data and the contractor’s role in teacher assignments within schools assigned to the treatment group. All MOUs shall be signed by the contractor and the lead district staff no later than August 1, 2018.

The contractor shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants.

6.3 Recruitment Progress Reports: While the contractor is actively communicating with districts to recruit the evaluation sample, the contractor shall submit weekly brief updates to the COR that details progress, problems, and plans to address any problems.

B. Option Period I (All Tasks are Cost Reimbursement)

Task 7—Option Period I Progress Reports

7.1 Monthly Progress Reports: The contractor shall submit monthly progress reports due to the COR no later than the 15th of each month. The reports shall use the same format and content as Task 1.2. During the months in Option Period I that overlap with the Base Period, the contractor shall only submit one monthly progress report that combines the reporting for both periods.

7.2 Monthly Manpower Reports: The contractor shall submit monthly manpower reports due to the COR no later than the 15th of each month. The reports shall use the same format and content as Task 1.3. During the months in Option Period I that overlap with the Base Period, the contractor shall only submit one monthly manpower report that combines the reporting for both periods.

Task 8—Randomize Sample and Monitor Treatment Implementation

The contractor shall randomly assign schools who are participating in the evaluation into the treatment or control condition. The contractor shall then ensure that study conditions are clearly communicated to schools and that teaching assignments are made according to the recruitment plan and MOUs with districts specified in Tasks 4 and 6. The contractor shall work with the districts and schools to ensure faithful adherence to the study conditions over the course of the 2018-19 and 2019-20 school years. The contractor shall work with the districts and schools to ensure, to the extent possible, that attrition and mobility do not adversely affect the integrity of the evaluation design.

8.1 Implementation Progress Reports: During the two implementation school years (2018-19 and 2019-20), the contractor shall include brief updates to the COR in the monthly reports under Task 7. These reports should detail progress, problems, and plans to address any problems.

Task 9—Data Collection

The contractor shall follow the data collection plan described in Task 4, including administrative records related to baseline information, as well as administrative records on outcomes of interest. In the course of monitoring implementation of departmentalized instruction as part of Task 8, the contractor shall ensure that they collect whatever information they need to adequately describe implementation. The contractor shall ensure that the teacher survey is administered efficiently and with high response rates so that the data can be used to inform the evaluation research questions.

9.1 Data Collection Progress Reports: During the two years of data collection (2018-19 and 2019-20), the contractor shall include brief updates to the COR in the monthly reports under Task 7. These reports should detail progress, problems, and plans to address any problems.

C. Option Period II (All Tasks are Cost Reimbursement)

Task 10—Option Period II Progress Reports

10.1 Monthly Progress Reports: The contractor shall submit monthly progress reports due to the COR no later than the 15th of each month. The reports shall use the same format and content as Task 1.2. During the months in Option Period I that overlap with Option Period II, the contractor shall only submit one monthly progress report that combines the reporting for both periods.

10.2 Monthly Manpower Reports: The contractor shall submit monthly manpower reports due to the COR no later than the 15th of each month. The reports shall use the same format and content as Task 1.3. During the months in Option Period I that overlap with Option Period II, the contractor shall only submit one monthly manpower report that combines the reporting for both periods.

Task 11—Analyze Data and Produce Findings Tables

The contractor shall prepare a detailed data analysis plan that clearly addresses all of the key evaluation research questions. The contractor shall then implement the data analysis plan using data collected in Task 9, and produce a set of findings tables that answer all of the key evaluation research questions. The tables shall all be consistent with NCES standards. For example, there shall be, no reporting of small cell sizes. (See the following link for more detail on NCEE standards: http://nces.ed.gov/statprog/2012/).

11.1 Draft Analysis Plan: The contractor shall prepare a detailed data analysis plan that documents the intended analyses to be conducted that will answer all key evaluation research questions. The contractor shall ensure that the data analysis plan references power analyses, documents any planned regression analyses (including a list of dependent and independent variables), proposed sensitivity analyses, proposed subgroup analyses, and table shells. The contractor shall submit the draft analysis plan no later than 125 weeks after contract award. The COR will review and provide comments within 2 weeks.

11.2 Final Analysis Plan: The contractor shall revise the draft analysis plan based on COR comments and submit a final version to the COR no later than 3 weeks after COR comments were received.

11.3 Draft Key Findings Memo: The contractor shall prepare populated tables that show the evaluation’s findings for each research question. The contractor shall also include a concise set of bullet points that summarize the main findings in a logical way. The contractor shall submit the key findings tables and bullets to the COR no later than 148 weeks after contract award. The COR will provide feedback on the draft key findings memo within 1 week.

11.4 Department Briefing: The contractor shall conduct a briefing for the COR and selected Department staff based on the draft key findings memo (Task 11.3). The meeting shall take place in the Department’s facilities in Washington, DC. The briefing presentation slides shall succinctly summarize the evaluation research questions, design, analyses, and findings. These slides shall be submitted to the COR no later than 152 weeks after contract award. The COR will provide feedback on these slides within 1 week. The contractor shall revise the slides and use them for the briefing, which shall take place no later than 154 weeks after contract award.

11.5 Materials for Sharing Findings with TWG and Documenting the TWG’s Feedback: The draft key findings memo (Task 11.3) shall serve as the basis for the PowerPoint slide presentation that the contractor shall prepare in order to solicit guidance from the TWG. At least 3 weeks before the TWG meeting, the contractor shall submit to the COR the following: (1) A set of PowerPoint slides that include proposed discussion questions for the TWG; and (2) A meeting agenda. Within 1 week after the first TWG meeting, the contractor shall submit a written summary memo to the COR detailing the TWG’s key recommendations, remaining issues, and the contractor’s proposed actions to address the TWG’s recommendations. The COR will provide feedback within 1 week.

11.6 Revised Key Findings Memo: The contractor shall revise the draft key findings memo (Task 11.3) to fully incorporate all TWG, Department, and COR feedback from Tasks 11.3, 11.4, and 11.5. The contractor shall submit the revised key findings memo to the COR within 4 weeks after TWG Meeting #2 (Task 12).

Task 12—Technical Working Group (TWG) Meeting #2

The contractor shall convene one full-day meeting of the TWG no later than 158 weeks after the contract has been awarded. The meeting shall focus on the draft key findings memo (Task 11.3), specifically the evaluation findings. This meeting shall be held at the Department’s offices in Washington, DC, and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break into the agenda.

Task 13—Report and Data Files

The contractor shall prepare a report and executive summary that clearly describes the evaluation, the intervention, and presents findings that answer the key evaluation research questions. The contractor shall ensure that the report is consistent with NCES and IES peer-review standards (see http://nces.ed.gov/statprog/2012/ and http://ies.ed.gov/director/sro/peer_review/report_review.asp). The contractor shall prepare a restricted-use data file that allows other researchers to reproduce the evaluation’s findings and/or conduct additional analyses. This restricted-use data shall satisfy risk disclosure requirements and meet the approval of IES’ Disclosure Review Board (DRB). The contractor shall be prepared to present a briefing of the report findings for Department staff and/or other interested parties if asked to do so.

13.1 Draft Report: The contractor shall ensure that the report is written and organized so that it is accessible to policymakers and research-savvy practitioners, in addition to academic researchers. The contractor shall include a stand-alone executive summary of the evaluation’s findings. The report is expected to be a maximum of 50 pages, excluding the executive summary and technical appendices. The contractor shall submit a draft report no later than 164 weeks after contract award. The COR will review and provide comments within 2 weeks.

13.2 Revised Draft Report: The contractor shall submit a revised draft report ready for IES’ Standards and Review Office (SRO) review no later than 4 weeks after COR comments are received for the draft report (Task 13.1). The COR will submit this revised draft report to SRO.

13.3 SRO-Approved Report: Within approximately 2 months, the Department will provide comments to the contractor from SRO for the revised draft report (Task 13.2). The contractor shall make final revisions based on SRO comments. The contractor shall revise the report until all SRO comments are addressed (current experience is an average of 3 rounds of successively less-extensive comments) and ensure that the report clears SRO no later than 184 weeks after contract award.

13.4 Final Report: The contractor shall provide a web-ready, 508-compliant PDF version of the SRO-approved report (including executive summary) (Task 13.3) to the COR no later than 3 weeks after the report is cleared by SRO.

13.5 Disclosure Analysis Plan: The contractor shall submit a risk disclosure analysis plan for the restricted-use data file to the COR for submission to IES’ DRB no later than 174 weeks after contract award.

13.6 Restricted-Use Data File: Once the codebook and risk disclosure analysis plan are approved, the contractor shall prepare and submit to the COR a restricted-use data file consistent with NCES standards (https://nces.ed.gov/statprog/style.asp). ASCII, STATA, or SAS file formats are acceptable. The contractor shall provide documentation for the file, including variable labels, category labels, weights (if relevant), and how missing data were addressed. The contractor shall put these files onto a CD and submit the restricted-use files via FedEx to the COR within 192 weeks after contract award.

D. Optional Task 14—Classroom Instruction Outcome (Cost Reimbursement)

The contractor shall determine, with the advice of the TWG and consent of the COR, the appropriate classroom observation instrument to use. The contractor shall collect the classroom observation data during the spring of the first intervention year. The contractor shall analyze the classroom data and estimate impacts of the intervention on the classroom instructional outcome.

14.1 Classroom Observation Plan: The contractor shall propose plans for collecting classroom instructional data using an appropriate observation instrument during the spring of the first intervention year in fourth-grade classrooms. The contractor shall justify how the proposed observation instrument will reliably and validly measure instructional quality. The plans shall provide details on the instrument, procedures for scheduling the observations, sample selection, timing, procedures for ensuring complete, high-quality data are collected, processes to ensure that there are a sufficient number of trained coders, and procedures for analyzing the resulting data. The contractor shall integrate these plans with the evaluation design in Task 4 and the forms clearance package in Task 3, following the same timelines for those tasks.

14.2 Collect Classroom Observation Data: The contractor shall collect the classroom observation data according to the approved plans from Task 14.1. The Department anticipates that the observation data will be collected via video, which the contactor shall purchase the cameras for. The contractor shall ensure that coders and observation data collectors are fully trained and certified, as appropriate. The contractor shall use the same approach and follow the same timelines described in the Task 9 data collection.

Version date: March 30, 2017 Attachment A- PWS

14.3 Analyze and Report Classroom Observation Data: The contractor shall clean, code, and analyze the classroom observation data according to the approved plans from Task 14.1. The contractor shall incorporate the findings from these analyses into the key findings memo (Task 11) and evaluation report (Task 13). The observation scores shall also be incorporated into the restricted-use data file (Task 13), to the extent possible, to allow other researchers to reproduce the evaluation’s findings and/or conduct additional analyses. The contractor shall follow the same timelines for those tasks.

PERFORMANCE WORK STATEMENT 5

IV. SCHEDULE OF DELIVERABLES

The contractor shall submit all deliverables to the Department’s Contracting Officer (CO), Contract Specialist (CS), and Contracting Officer’s Representative (COR) as specified in the delivery schedule. Unless otherwise specified in the scope of work:

· The contractor shall submit deliverables electronically via e-mail.

· The COR will notify the contractor within three working days of receiving a deliverable that it is acceptable or that changes are required.

· If revisions are necessary, the contractor shall make them and submit the revised deliverable to the COR within two working days.

· If a second round of revisions are necessary, the COR will notify the contractor within three working days. The contractor shall make the revisions and resubmit the final deliverable within two working days. The contractor shall ensure that deliverables require no more than two rounds of revisions.

A. Base Period (Months 1-18)

B. Option Period I (Months 12-35)

C. Option Period II (Months 29-48)

D. Optional Task

Compliance with The Department’s IT Security Policy

The contractor, and all subcontractors, shall comply with the Department of Education’s IT security policy requirements, specifically those set forth in the ‘Handbook for Information Assurance Security Policy (OCIO-01)’, and other applicable procedures and guidance. The contractor, and all subcontractors, shall develop and implement management, operational and technical security controls to assure required levels of protection for information systems. The contractor, and all subcontractors, shall further comply with all applicable Federal IT security requirements including, but not limited to, the Federal Information Security Modernization Act (FISMA) of 2014, Office of Management and Budget (OMB) Circular A-130 Appendix III, Homeland Security Presidential Directives (HSPD), the National Institute of Standards and Technology (NIST) standards and guidance, and the Federal Risk and Authorization Management Program (FedRAMP) requirements and guidance.

In accordance with OMB Memorandums M-17-06, and M-15-13, and M-08-23, and with the NIST SP 800-44, all Federal websites and web services must be accessible through a secure connection (HTTPS only, with HSTS), and e-mail applications must have SMTP enable. The use of HTTPS is encouraged on intranets, but not explicitly required.

These security requirements include, but are not limited to, the successful Security Authorization (SA) of the system (includes commercially owned and operated systems managed by the commercial vendor and its subcontractors, supporting Department programs, contracts, and projects); obtaining a full Authority to Operate (ATO) before being granted operational status; performance of annual self-assessments of security controls; annual Contingency Plan testing; performance of periodic vulnerability scans; updating all information system security documentation as changes occur; and other continuous monitoring activities, which may include, mapping, penetration and other intrusive scanning. Full and unfettered access for the Department’s third party Managed Security Services Provider (MSSP) must be granted to access all computers and networks used for this system. Additionally, when there is a significant change to the system’s security posture, the system (Federal and commercial prime- and sub- contractors included) must have a new SA, with all required activities to obtain a new ATO, signed by the Authorizing Official (AO). If a cloud solution will be used, then an ED-issued, FedRAMP-Compliant ATO is a Federal and a Departmental requirement, and it is recommended that one be obtained.

System security controls shall be designed and implemented consistent with the most current, finalized version of the NIST SP 800-53, ‘Recommended Security Controls for Federal Information Systems and Organizations.’ All NIST SP 800-53 controls must be tested / assessed no less than every 3 years, according to federal and Department policy. The risk impact level of the system will be determined via the completion of the Department's inventory form and shall meet the accurate depiction of security categorization as outlined in Federal Information Publishing Standards (FIPS) 199, ‘Standards for Security Categorization of Federal Information and Information Systems.’

System security documentation shall be developed to record and support the implementation of the security controls for the system. This documentation shall be maintained for the life of the system. The contractor, and all subcontractors, shall review and update the system security documentation at least annually and after significant changes to the system, to ensure the relevance and accurate…