Attachment_A_-_Draft_PWS.doc

U.S. Department of Education

Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

DRAFT Performance Work Statement

DESIGN AND FEASIBILITY OF AN IMPACT STUDY OF MAGNET SCHOOLS

I. INTRODUCTION

The Institute of Education Sciences (IES) at the U.S. Department of Education (Department) intends to develop design options and assess the feasibility of conducting an impact study of magnet schools, particularly those awarded grants in FY 2016 and 2017 through the federal Magnet Schools Assistance Program. If the activities and deliverables under the Base Period of the contract suggest such an evaluation is feasible, including interest in participating by a sufficient number of schools that use lotteries, the Department will decide whether to exercise work to conduct the evaluation under two options (Exhibit 1). The decision to execute Option Period 1, which supports data collection for the 2018-2019 and 2019-2020 school-years, will be made by November 1, 2018. The decision to execute Option Period 2, which supports data collection for the 2020-2021 and 2021-2022 school-years as well as the development of the final report, will be made by March 1, 2020. The program office responsible for this requirement will be the National Center for Education Evaluation and Regional Assistance (NCEE) at IES.

Exhibit 1. Contract Periods of Performance

A. Legislative Authority

This impact evaluation of magnet schools is authorized by Part B section 8601 of the Elementary and Secondary Education Act of 1965 (ESEA) as amended by the Every Student Succeeds Act (ESSA) which allows the Department to pool resources across ESEA programs in order to fund rigorous evaluations of individual Federal education programs that currently lack sufficient evaluation dollars or to evaluate the impact of various strategies that cut across a wide range of ESEA programs. The Magnet Schools Assistance Program (MSAP), itself, is authorized under Title IV, Part D of the ESEA, as amended by ESSA, and provides grants to local educational agencies (LEAs) and consortia of LEAs to support magnet schools under an approved, required or voluntary, desegregation plan.

B. Background

Magnet schools are an important component of public school choice, as well as a strategy used by districts with the aim of improving student achievement and school diversity. Magnet schools – or programs within schools - typically implement a distinctive theme or instructional method and recruit at least some students from outside the school’s neighborhood to encourage a broader range of student backgrounds within the school. The theory behind magnets is that all students will benefit academically from the school’s thematic focus and, potentially, from exposure to a diverse student body.

Magnet schools hold a prominent place in the history of education reforms in the United States and were the earliest form of school choice. Since the 1970’s magnet schools have been critical to school districts’ efforts to carry out desegregation plans by providing parents with the option of sending their children to schools with attractive offerings outside their own neighborhoods. The establishment of magnet schools has also been designed to encourage parents to keep their children in a district’s public school system rather than placing them in private schools or relocating to more suburban school districts . Magnet schools are now one of several school choice options, such as charters, that provide parents the opportunity to select a school to meet their children’s educational needs and interests, though the options differ in their governance and oversight.

Participation in magnet schools has increased two-fold since 2000. Approximately 2.5 million students currently attend magnet schools in the US, about the same as the attendance in charters . The Office of Innovation and Improvement's Magnet Schools Assistance Program (MSAP) at the US Department of Education has provided grants to school districts to support magnet programs since 1985. The most recent goals of the MSAP program, specified in the most recent reauthorization of the Elementary and Secondary Education Act, the Every Student Succeeds Act, include improving the diversity of schools serving substantial proportions of minority students, providing all students with the opportunity to meet challenging State academic standards, and utilizing innovative education methods and practices that promote diversity and increase choice in public schools and public educational programs

Despite the growth in the number of students attending magnet schools, there is limited evidence about their effectiveness. Few studies use the types of rigorous methods that allow for drawing conclusions about impacts with confidence. And the studies that do use these methods – random assignment through school lotteries – have been conducted in in a single state (Bifulco et al., 2009) or a single city (Crain et al., 1999) and yielded mixed results The most recent MSAP grant competitions provide a unique opportunity to study the effects of magnet schools across the country. The FY 2016 and 2017 MSAP grants prioritize the creation and replication of evidence-based magnet programs. Also, opportunities for random assignment will likely exist because the grant notices in those years included a competitive preference priority based on the extent to which the grant applicants proposed to select students to attend the magnet schools by methods such as lotteries.

C. Overview of the Design and Feasibility Phase The design and feasibility study (Base Period) will determine if it is possible to conduct an impact evaluation of magnet schools, focusing on MSAP districts funded in FY 2016 and 2017. Two key deliverables shall be prepared: (1) a memo assessing the number and characteristics of schools that meet the study’s criteria and the implications for evaluation design, and (2) an IES brief describing the schools’ recruitment and admission practices, to inform ongoing program improvement.

Specifically, during the feasibility phase of the study the contractor shall address the following questions:

· How many schools are (1) conducting lotteries for admission that yield reasonable-sized treatment and control groups, (2) in districts with data systems that link students and their achievement scores across schools and school years, and (3) in districts willing to provide the necessary data on the conduct of the lotteries as well as student outcomes for use in the evaluation?

· Is the number of schools that meet these criteria and the information available on students attending these schools, sufficient to permit the detection of meaningful effects on student achievement and/or other outcomes? What outcomes should be emphasized, given IES’ desire to rely primarily on administrative data?

· How do all of the districts and schools funded through the FY 2016 and 2017 MSAP grants recruit students to attend the magnet schools and select those who are to be admitted?

A number of challenges may affect the study design. The contractor shall consider these issues in the design and conduct of the evaluation.

· Statistical power. The contractor shall determine the appropriate sample size to achieve a targeted effect size of .10 for student achievement as well as the effect size that might be expected given the number of schools likely to participate in an evaluation and those schools’ typical student applications/admission. In performing power calculations, the contractor shall take into account expected variation in the size and features of school lotteries.

· District and school recruitment. Some districts and schools may be reluctant to share their data with a contractor, even if districts have the required data for participation in the evaluation, either because of the staff burden of providing data for an evaluation, or because of concerns about maintaining confidentiality of individuals’ information. While Department regulations require grantees to cooperate with a federally-designated program evaluation, the contractor shall implement a reasonable approach to recruiting a sufficient number of districts and schools to participate in the evaluation, including the schools attended by students who are not offered admission to the magnet schools.

· School selection. Due to the limited number of FY 2016 MSAP grantees and expected FY 2017 grantees, it will likely not be possible to purposefully select schools according to school characteristics (e.g. instructional approaches or themes). However, it is extremely important that the study is policy relevant and the intervention being tested is clear.

· Appropriate grade level focus and outcomes. It is possible the magnet schools that meet the study’s criteria include elementary, middle, and / or high school grades. The contractor shall consider approaches for the selection of relevant outcomes (derived from administrative data) as well as, potentially, the trade-off of pooling outcomes across school levels.

· Variation in achievement measures. It is likely that the districts and schools recruited for the study will have different measures of student achievement and the availability of testing data may vary by grade level. The contractor shall implement reasonable approaches for pooling different measures of academic achievement in order to draw inferences regarding the impact of magnet schools.

· Treatment-control contamination. Many school districts have multiple magnets and other choice schools. There is the possibility that students who are not offered admission to a participating magnet through its lottery may end-up being assigned to a different magnet or choice school versus a traditional neighborhood school. A strong evaluation design must consider the experiences of students assigned to the control group and find ways to select districts and schools where the likelihood of control group students winding up in other magnet or choice schools is minimal and/or consider analytic approaches to address this contamination.

The schedule for the feasibility phase is ambitious due to the need to screen grantees and develop the information necessary for IES to make the decision to execute Option Period 1 by November 1, 2018. The approach to the design and feasibility phase includes the following key activities.

· Attain data collection approval. The contractor shall obtain the Office of Management and Budget (OMB) approval for the data collection activities in both the design/feasibility study phase and the full evaluation.

· Screen districts and schools and prepare design and feasibility memo. Shortly after OMB approval is attained, the contractor shall contact the FY 2016 and 2017 MSAP grantee districts to determine the eligibility and willingness of districts and schools to participate in the evaluation, if conducted. The contractor shall prepare a memo that summarizes these data as well as provides design options for the evaluation.

· Survey principals about student recruitment and selection and prepare descriptive brief. The contractor shall develop and administer a principal survey, focused on how the FY 2016 and 2017 MSAP grantees recruit students to the magnet schools and determine who is offered admission. Based on these survey data, the contractor shall prepare a brief describing these practices.

· Develop a principal survey on school organization and instruction. The contractor shall develop a principal survey during the design and feasibility phase that would gather information on the characteristics of the magnet schools as well as the schools attended by students who are not offered admission to the magnet schools. The survey will be included in the OMB package, but only administered if the full evaluation is executed.

D. Overview of the Full Evaluation

The evaluation, if exercised through two option periods, will assess and report on the effectiveness of MSAP magnet schools.

Specifically, the contractor shall address the following research questions:

· What is the impact of the magnet programs on relevant student outcomes (achievement and/or other relevant measures of student success such as persistence in school or graduation)?

· What is the impact of the magnet program on the characteristics of the schools that the students attend, including whether they are higher performing or more diverse?

· To what extent is there a relationship between school characteristics, including diversity, and student outcomes?

The approach to the evaluation phase includes the following key activities.

· Obtain agreement to cooperate from eligible districts and schools. The contractor shall finalize memoranda of understanding from districts and schools that specify the roles and responsibilities of the districts and schools as well as the contractor [Option Period 1].

· Collect lottery and outcome data for 4 school years (2018-2019 through 2021-2022). The contractor shall collect data on the participating magnet schools’ admissions lotteries for students seeking to enroll for the first time in school years 2018-2019 and 2019-2020 (two cohorts); these students will constitute the evaluation sample. The contractor shall also obtain administrative records for the evaluation sample, both students attending the participating magnet schools as well as those who attending other schools in the district for the 2018-2019 and 2019-2020 school years [Option Period 1] as well the 2020-2021 and 2021-2022 school years [Option Period 2]. Finally, the contractor shall administer a survey of principals in both sets of schools, limiting the number of control group schools to a maximum of 10.

· Prepare final report. The contractor shall develop one impact report that analyzes the data for all 4 school years [Option Period 2].

II. Scope of Work

This section specifies the tasks and subtasks that the contractor shall perform. All deliverables under these tasks shall be submitted to the Department’s Contracting Officer’s Representative (COR), Contracting Specialist (CS), and Contracting Officer (CO) electronically unless otherwise noted. Unless indicated otherwise, the COR will notify the contractor within two weeks of receipt of any changes required in the deliverable, and the contractor shall have up to two weeks to make the necessary revisions and submit a final deliverable to the Department unless otherwise noted.

The Contracting Officer’s Representative (COR) reserves the right to inspect the contractor’s work while in process. The contractor shall submit to the COR any progress report or other written documentation of work to date that the COR requests to facilitate that inspection, according to contract clause 52.246-5.

All tasks are Cost-Reimbursement.

E. Base Period

Task 1—Opening Meeting with the Department

The contractor’s project director and up to four project staff shall meet with the COR and other appropriate Department staff within 5 business days after the effective date of the contract award to discuss details regarding the tasks outlined in the contractor’s final proposal revision, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, DC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the evaluation.

Deliverables:

Opening Meeting summary

Task 2– Prepare OMB Package and Develop Data Collection Forms

The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments for the full study. The preparation of the data collection forms occurs as part of the base contract. However, data collection for Option Period 1 and Option Period 2 will only occur if the option period(s) are executed. The package shall include a description of the evaluation design, site recruitment, and data collection instruments, as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions. In addition, the contractor shall ensure that the forms clearance package includes: (1) the evaluation mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the evaluation and its potential use for policy decisions.

The contractor shall submit a draft forms clearance package to the COR no later than 6 weeks after contract award. The COR will review within 2 weeks. The contractor shall submit an acceptable revised forms clearance package for original submission no later than 2 weeks after COR comments. The COR will submit the accepted forms clearance package for clearance. Based on comments from internal and external clearance (after the first 60 day clearance process), the contractor shall make all needed revisions and submit a revised forms OMB clearance package for submission to OMB 2 weeks after internal and external comments. The COR will then submit the final revised forms clearance package to OMB.

Data collection forms for the design and feasibility phase (base contract) shall consist of:

· Structured district and school screener – This structured protocol will be used to screen districts and schools receiving FY 2016 and 2017 MSAP funding to determine their eligibility and willingness to participate in an evaluation. The contractor shall use these conversations with district officials to identify over-subscribed magnet schools that have utilized or could utilize lotteries to assign students to openings in the school. ED estimates that all FY 2016 and 2017 grantees will need to be contacted during the feasibility phase of the study in order to identify districts and schools that could be included in the evaluation. Depending on the how the lotteries are conducted, the district may be able to provide the necessary information on the lotteries or the schools themselves may be the more appropriate respondents . The contractor shall use these conversations with districts and schools to determine their willingness to participate in the evaluation, if conducted. These interviews shall be conducted soon after OMB approval is attained.

· School survey on student recruitment and selection - This principal survey will provide data on how the MSAP 2016 and 2017 grantees recruit students to the magnet schools and determine which students are offered admission to the magnet school. This survey shall be administered in the Fall/Winter of 2018.

Data collection forms for the full evaluation (Option Period 1 and Period 2) shall consist of:

· School survey on school organization and instruction - This principal survey will provide data on the characteristics of the magnet schools as well as the schools attended by students who are not offered admission to the magnet schools, including the schools’ curriculum focus, admissions process, programs available to students, organization of instruction, faculty and staff characteristics, facilities, and safety. This survey will be administered in the spring of each data collection year (2018-2019 and 2019-2020 school years [Option Period 1]; 2020-2021 and 2021-2022 school years [Option Period 2]).

· Lottery operator records extraction. These records include application information and lottery results. These records are updated on a frequent basis as students who were waitlisted for a magnet school may be offered admission to a magnet school on a rolling basis. These records will need to be collected prior to and across the first two data collection year (2018-2019 and 2019-2020 school years [Option Period 1]. However, records on the first data collection year (2018-2019 school year) will need to be retrospectively-collected due to the timing of the execution of Option Period 1.

· District records data extraction – These will include data on student characteristics, school enrollment, test scores, and other relevant data such as attendance, persistence and graduation for students admitted to the magnet through lotteries as well as students who are not offered admission to the magnet schools as well as data on the schools (and school staff) that these students attend. These records will need to be collected for each data collection year (2018-2019 and 2019-2020 school years [Option Period 1]; 2020-2021 and 2021-2022 school years [Option Period 2]). Baseline data for the 2018-2019 school year (2017-2018 data) will likely need to be retrospectively-collected due to the timing of the execution of Option Period 1.

In developing the forms, the contractor shall make use of or refine existing instruments to the extent possible and applicable. Tested instruments that could provide a broader context for the study findings are available and the timeline for the development of this OMB package is unavoidably short.

The contractor shall pilot test each of the instruments with no more than nine respondents per instrument in no less than two schools to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective. Piloting is to be conducted prior to the final 60 days in the OMB submission process. The contractor shall ensure that instruments are revised as needed based on pilot test results.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables:

Draft original submission OMB clearance package

Original submission OMB clearance package (first 60 day OMB submission)

Revised OMB clearance package (second 60 day OMB submission)

Final OMB clearance package (cleared package)

Documentation of IRB approvals

Task 3 –Data Collection for the Design and Feasibility Phase

Upon receiving IRB and OMB approval for data collection, and using instruments approved by OMB, the contractor shall contact all FY 2016 and 2017 MSAP grantees to determine their eligibility and willingness to participate in an evaluation. In the Fall/Winter of 2018, the contractor shall survey all 2016 and 2017 grantee schools about their recruitment and admissions practices.

Data Collection Quality

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf). To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set. The data monitoring activities shall be in place within 7 months after contract award. During data collection, the contractor shall include with the monthly progress reports updates on the progress of data collection based upon the data monitoring activities.

To ensure that the data collection and interview staff have the skills necessary to perform their duties, the contractor shall design a comprehensive training program. The training shall document the minimum performance standards that the data collectors or interviewers must meet, the criteria by which the contractor shall ascertain whether data collectors and interviewers meet the performance standards, and how often performance will be evaluated.

Data Collection Security

The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants.

The contractor shall be required to establish procedures at its sites and at subcontractor sites to ensure confidentiality of data. Their systems shall be required to provide reasonable assurance that information identifying individuals is in files physically separated from other research data. The contractor and subcontractors shall be required to maintain security of the complete set of all master data files and documentation. Access to individually identifiable data shall be strictly controlled. At each site all data shall be kept in locked file cabinets during nonworking hours, and work on hardcopy data shall take place in a single room, except for data entry. Physical security of electronic data shall also be maintained. Security features that protect project data shall include password-protected accounts that authorize users to use the contractor’s and subcontractors’ information systems but only to access specific network directories and network software; user rights and directory and file attributes that limit those who can use particular directories and files and determine how they can use them; and, additional security features that the network administrators establish for projects as needed. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) such data shall comply with the requirements of the confidentiality standards in section 183 of the ESRA (20 U.S.C. 9573).

In safeguarding personally identifiable information (PII), the contractor and its subcontractors also shall be subject to ED’s requirements contained in ED’s Handbook for the Protection of Sensitive But Unclassified Information, OCIO-15, and ED’s policy that the transmission of sensitive but unclassified information, including PII, through an e-mail requires that the contents be password protected in a ZIP file.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request. When possible, the contractor shall ensure that individuals collecting and/or coding data do not have knowledge of the study participant’s treatment condition.

In accord with OMB Memorandum M-05-04, and with the NIST SP 800-44, all publicly accessible Federal websites and web services must have HTTPS enabled, and shall only provide service over a secure connection, and e-mail applications must have SMTP enabled.

All documents created or data collected during the contract shall be subject to records retention requirements as mandated by the Federal Records Act of 1950, as amended (44 U.S.C. 3101-3107).IT Systems

Systems developed for this contract shall be hosted in the IES Data Center. To be hosted in the IES Data Center, all websites and web applications must be developed using ASP.NET (.NET Framework 3.5, 4.5, 4.6, and MVC 3 are currently supported) and SQL Server 2012 Standard Edition on Windows 2012 Servers and comply with IES's web development standards. A copy of the standards is available upon request. Other platforms and databases are not authorized to run in the IES Data Center. All third party software needs to be approved by the Department. Any third party software must be licensed appropriately or be freely available through open source licensing. The contractor should not assume new software will be approved. All software developed under this contract should be transferred to IES free of any restrictions, licenses, or fees in accordance with the Rights and Data Clause contained in the contract. All IT security requirements listed in Appendix A. Compliance with ED IT Security Policy are covered by the current IES Data Center security authorization. By hosting this system in the IES Data Center the contractor will be in compliance with the IT security requirements listed in this contract.

Task 4—Technical Working Group for the Evaluation

4.1 Establish Technical Working Group (TWG). The role of the TWG will be to advise the contractor and the Department on the conduct of the evaluation including, but not limited to: evaluation design, instrumentation, data collection and analysis, and draft evaluation reports. The TWG shall be comprised of approximately 10 individuals, and include experts in magnet schools, the conduct of lotteries used for school admission, and evaluation methodology. The contractor shall submit a list of proposed TWG members no later than 6 months after contract award. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the evaluation. Within four week of the COR approval of the the proposed TWG members, the contractor shall contact each proposed member and submit letters of commitment to the COR and CS for approval by the Contracting Officer (CO).

4.2 Convene Technical Working Group. The contractor shall convene a meeting of the TWG at the Department’s offices in Washington, DC in the base period. The TWG meeting shall be held no later than October 24th, 2018 and shall focus on design and feasibility issues, data collection, and data analysis. The design and feasibility memo (prepared in Task 5), which describes the extent of the lottery and student records data available for the evaluation, provides the pros and cons of conducting the full evaluation with various samples of schools and outcomes, and provides plans for the full study, will be used as material for informing the TWG of the evaluation design prior to the TWG meeting.

After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S. Department of Education facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

If TWG members recommend any significant deviation from the proposal, evaluation design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the evaluation based solely upon the recommendation of TWG members without the written approval of the COR. The contractor shall also use the TWG members to review products, as appropriate, when group meetings are not required.

Deliverables:

List of proposed TWG members Letters of commitment and consulting rates Summary of TWG meeting

Task 5 – Design and Feasibility Memo (Cost Reimbursement Task)

The contractor shall prepare a design and feasibility memo that describes (1) the extent of the lottery and student records data available from the FY 2016 and 2017 MSAP grantee districts, (2) the number of districts and schools with the necessary data that are willing to participate in the evaluation, (3) barriers to participation cited by those expressing disinterest, (4) the pros and cons of conducting the full evaluation with various samples of schools and outcome measures, and (5) the policy relevance, evaluation methodology, data collection, and analysis plans for the full study including the aggregation of outcomes across schools and outcomes. The draft design and feasibility memo shall be submitted by August 21, 2018. The COR will review the draft design and feasibility memo within 2 weeks. The contractor shall make revisions and submit a revised design and feasibility memo within 2 weeks after receiving review comments from the COR on the draft design and feasibility memo. The revised design and feasibility memo shall be of appropriate detail and quality to submit to the technical working group (TWG) for review. The contractor shall revise the revised design and feasibility memo based on the feedback from the TWG, in consultation with IES, and submit a final design and feasibility memo 4 weeks after the TWG meeting.

Deliverables:

Draft design and feasibility memo Revised design and feasibility memo

Final design and feasibility memo Task 6 – Descriptive Brief on FY 2016 and 2017 MSAP Grantee Recruitment and Admissions The contractor shall prepare a short brief (approximately 15 pages) describing the FY 2016 and 2017 MSAP grantees’ recruitment and admission practices. This brief will incorporate the data from the school survey on student recruitment and selection administered in Fall/Winter of 2018. The contractor shall submit a draft brief no later than 20 months after contract award. The COR will review and provide comments within two weeks. The contractor shall submit a revised draft brief ready for peer review within 4 weeks after receiving review comments from the COR. The Department will submit for internal peer review. Internal peer review is appropriate for this type of brief because it will only contain simple cross-tabs and descriptive statistics. Within approximately three weeks, the Department will provide comments to the contractor from the IES internal peer review. The contractor shall make final revisions based on IES peer review comments and submit a revised final brief within one month. The contractor shall revise the brief until all peer review comments are addressed and the brief clears peer review (current experience is two rounds of review comments). The contractor shall provide a web-ready version of the brief to the COR within two weeks after the brief is cleared by peer review.

Any and all documents produced under this work statement which are to be posted on the web shall be compliant with the Section 508 accessibility standards for individuals with disabilities posted at 36 CFR 1194.22, and shall be tested for accessibility by the Department’s Assistive Technology Program.

Deliverables:

Draft descriptive brief on MSAP grantee recruitment and admissions

Revised draft brief, SRO-ready on MSAP grantee recruitment and admissions

Revised Final brief on MSAP grantee recruitment and admissions

Print and web-ready brief on MSAP grantee recruitment and admissions

Task 7- Reporting Requirements In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contracting Officer (CO) and one copy to the COR, simultaneously with the monthly invoice which is submitted via IPP. The base phase includes performance reports for months 1 through 24.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive work and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems. The reports shall be accompanied by a list of all staff assigned to the project, their personnel position as it relates to the Department’s Contractor Employee Personnel Security Screening, the date that each staff was added to the contract as well as the date the paperwork for the screening was completed.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

F. Option Period 1

Task 8—Finalize Agreement to Cooperate from Eligible Districts and Schools

As part of the recruitment process, the contractor shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants.

The contractor shall finalize cooperation of districts and schools in the evaluation in consultation with the COR. The contractor shall submit memoranda of understanding for those districts and schools that will participate in the evaluation to the COR within six weeks after Option Period 1 is executed. The contractor shall submit to the COR progress reports on district and school recruitment for the study as part of the monthly progress reports as soon as Option Period 1 is executed.

Deliverables:

Memoranda of understanding from districts

Task 9–Data Collection for Option Period 1

Upon the execution of Option Period 1 and using instruments approved by OMB, the contractor shall conduct the data collection activities (specified in Task 2) for the first two data collection years (2018-2019 and 2019-2020 school years). This data collection includes surveying the magnet schools as well as the schools attended by students who are not offered admission to the magnet schools in the spring of each school year as well as collecting lottery operator records and student records data for these years. Data on the lotteries held prior the 2018-2019 school year will need to be retrospectively-collected due to the timing of the execution of Option Period 1.

To ensure that data collected are of high quality and adhere to the Department’s security process, the contractor shall use all procedures outlined in Task 3 above. During data collection, the contractor shall include status updates on that activity in the monthly progress reports based upon the data monitoring activities.

Task 10- Reporting Requirements With the exception of the months for which the Base Period and Option Period 1 overlap (months 16-24) and monthly reporting will occur under Task 7, the contractor shall submit one copy of the following performance reports each month to the Contracting Officer (CO) and one copy to the COR, simultaneously with the monthly invoice. (months 25 through 40) Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive work and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems. The reports shall be accompanied by a list of all staff assigned to the project, their personnel position as it relates to the Department’s Contractor Employee Personnel Security Screening, the date that each staff was added to the contract as well as the date the paperwork for the screening was completed.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

G. Option Period 2 Task 11–Data Collection for Option Period 2 Upon the execution of Option Period 2 and using instruments approved by OMB, the contractor shall conduct the data collection activities (specified in Task 2) for the last two data collection years (2020-2021 and 2021-2022 school years). This data collection includes surveying the magnet schools as well as the schools attended by students who are not offered admission to the magnet schools in the spring of each year as well as collecting student records data for these years.

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Rask 3 above. During data collection, the contractor shall include with the monthly progress reports updates on the progress of data collection based upon the data monitoring activities.

Task 12—Convene Technical Working Group for the Evaluation The contractor shall elicit written feedback from up to 5 TWG members on the revised analysis plan developed in Task 13. After the TWG comments are received, the contractor shall submit a written summary of the comments to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR.

The contractor shall convene a meeting of the TWG to discuss the impact findings. After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. Meeting sites shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S. Department of Education facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

The contractor shall provide copies of all reviews of products by the panel to the COR within five working days of receipt by the contractor. If TWG members recommend any significant deviation from the data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the evaluation based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review products, as appropriate, when group meetings are not required.

Deliverables:

Summary of TWG feedback on analysis plan

Summary of TWG meeting Task 13—Data Analysis, Report Preparation, and Restricted Use File The contractor shall conduct data analyses and prepare one report that clearly addresses the key evaluation questions. The contractor shall ensure that that study report is consistent with IES peer review standards.

13.1 Data Analysis Plan

The contractor shall prepare a detailed draft data analysis plan that documents the intended analyses to be conducted and included in the report. The contractor shall submit the draft analysis plan within 50 months after contract award. The COR will review and provide comments within 2 weeks. The contractor shall revise based on the COR’s comments and submit a revised analysis plan within 1 month of the COR’s comments. The contractor shall revise the analysis plan based on feedback received from the TWG, in consultation with the COR, and submit a final analysis plan within 1 month of the TWG comments.

13.2 Study Report

The contractor shall ensure that the study report is written and organized, so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers. The contractor shall include a stand-alone executive summary of the study’s findings.

The contractor shall submit a draft report on the findings no later than 70 months after contract award. The COR will review and provide comments within three weeks. The contractor shall submit a revised draft report ready for submission to the Standard’s and Review Office (SRO) within six weeks of ED comments. ED will submit the draft report to the SRO. Within approximately 2 months, ED will provide comments to the contractor from the SRO. The contractor shall make final revisions based on the SRO comments and submit a revised final first report within one month. The contractor shall revise the report until all peer review comments are addressed and the report clears the SRO review (current experience is an average of 3 rounds of successively less extensive comments). The contractor shall provide a web-ready and a print ready version of the report to the COR within 2 weeks after the report is cleared by the SRO.

13.3 Data File

The contractor shall prepare a restricted use data file for the report. The file shall contain all data collected for the evaluation with all direct personal identifiers removed. The contractor shall provide documentation for the files including variable labels, category labels, weights, and how missing data were addressed. In addition, the contractor shall submit documentation accompanying the data to include the unit of observation for each data file (e.g., child, classroom, school, etc.).

For each file, the contractor shall also provide IES with a risk disclosure analysis for submission to National Center for Education Statistics (NCES). Once the codebook and risk disclosure analysis are approved, the contractor shall submit to the COR the restricted use data file consistent with NCES standards . ASCII or SAS file formats are acceptable. The contractor shall submit the restricted use file for the report to the COR by 79 months after contract award.

Deliverables:

Draft analysis plan Revised analysis plan

Final analysis plan

Draft report

Revised draft report, SRO-ready Final report Print and web-ready report Restricted use data file with documentation report

Task 14- Reporting Requirements In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the Contracting Officer (CO) and one copy to the COR, simultaneously with the monthly invoice. The Base Period includes performance reports for months 41 through 80.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive work and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems. The reports shall be accompanied by a list of all staff assigned to the project, their personnel position as it relates to the Department’s Contractor Employee Personnel Security Screening, the date that each staff was added to the contract as well as the date the paperwork for the screening was completed.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month.