ATTACH_B_Draft_i3_PWS_FY_2014-2016_Cohorts.pdf

ATTACHMENT B – i3 Performance Work Statement (PWS)

Draft Performance Work Statement

EVALUATION OF INVESTING IN INNOVATION

FISCAL YEAR (FY) 2014-2016 COHORTS

National Center for Education Evaluation

Institute of Education Sciences

U.S. Department of Education

I. INTRODUCTION

The Institute of Education Sciences (IES) at the U.S. Department of Education (ED) intends to provide technical assistance to the local evaluators of the 2014 cohort of

Investing in Innovation (i3) Grantees and to produce a report summarizing the status of these evaluations through one year of the grants.

i3 Grantees are required to fund an independent evaluation. This contract will provide technical assistance on the conduct of these evaluations to maximize their strength (with a goal of meeting What Works Clearinghouse evidence standards) and the quality of their implementation data and performance feedback. The contract will also assess the strength of the evaluations and conduct a cross-site analysis of the results of the evaluations.

The three existing Evaluation of i3 contracts (ED-IES-10-C-0064, ED-IES-13-C-0005, and ED-IES-14-C-0007), which are described below in the Project Design Issues section

(Section D), cover the 2010 – 2013 cohorts of i3 grantees and cannot be expanded to include future cohorts.

A. Investing in Innovation Authorizing Legislation and Purpose

Investing in Innovation was authorized under the American Recovery and Reinvestment

Act of 2009 (ARRA), Section 14007, Title XIV (Public Law (P.L.) 111-5), as amended by section 307 of Division D of P.L. 111- 117 (H.R. 3288), the Consolidated

Appropriations Act, 2010.

The American Recovery and Reinvestment Act included $100 billion in new federal money for education. A substantial portion of this money was intended to replace lost state revenue, but much of it is designed to drive innovation at all levels of education, improve school performance, and reduce achievement gaps. ARRA authorized up to

$650 million in FY 2010 to establish an Innovation Fund. Subsequent appropriations authorized $150 million for additional grants under this program in FY 2011 and in FY

2012, and $142 million in FY 2013 and FY 2014.

The purpose of i3 is to support entities with a record of improving student achievement

(local educational agencies (LEAs) and nonprofits in partnership with LEAs or a consortium of schools) in order to expand the implementation of, and investment in, evidence-based practices, strategies, and programs and to serve as models. Another purpose of this fund is to identify and document best practices that can be shared and taken to scale based on demonstrated success.

The i3 program includes three types of competitive grants: Scale-up, Validation, and

Development. Scale-up and Validation grants require prior evidence of effectiveness, while Development grants support innovative strategies.

All i3 grantees are required to fund an independent local evaluation. Scale-up and

Validation grantees are expected to conduct a study that, if well-implemented would meet

What Works Clearinghouse (WWC) evidence standards without reservations.

More information on i3 is available on the Department’s website at:

http://www.ed.gov/programs/innovation/index.html.

B. Period of Performance

The contract will include a 14-month Base Period covering the first year of work with the

FY 2014 cohort; and five successive, 14-month Option Periods to extend the technical assistance and reporting work through the completion of the FY 2014 cohort. Each of these option periods will overlap with the previous and following period by 2 months.

The contract will include six additional 14-month Option Periods to conduct this work with the FY 2015 cohort and six additional 14-month Option Periods to conduct this work with the FY 2016 cohort. Within the cohorts, each of the option periods will overlap with the previous and following period by 2 months.

While there will be a base period and 17 option periods, all periods overlap with each other by at least 2 months and as many as 14 months.

If all of the option periods are exercised, the total length of the contract will be 98 months.

C. Key Questions and Activities

The key evaluation questions are:

To what extent are the i3 independent local evaluations well-designed and well-implemented

What are the results for different categories of key i3-funded practices, strategies, and programs?

As defined in the attached document, “National Evaluation of i3 Analysis and Reporting Plan Version

1.0,” produced under the first contract (EDIES10C0064).

Since one of the purposes of i3 is to identify and document best practices that can be shared and taken to scale based on demonstrated success, each grantee is required to fund a local independent evaluation. Scale-up and Validation applications will be judged on the extent to which the proposed methods of evaluation include a study that, if well-implemented, would meet What Works Clearinghouse evidence standards without reservations. While Development applications will not explicitly be judged on the extent to which the proposed evaluation methods would meet WWC evidence standards, these applicants will also be judged on the quality of their evaluation plans.

The contractor will provide technical assistance to local evaluators to maximize the strength of the design and implementation of the independent evaluations. There are several i3 Government Performance and Results Act (GPRA) program performance measures (see the table below) related to the local independent evaluations and to the quality of implementation and performance data they collect (all the i3 performance measures can be found at: http://www2.ed.gov/programs/innovation/index.html). There are measures for Scale-up and Validation grantees that involve the extent to which evaluations are well-designed and well-implemented. This means that the evaluation meets the What Works Clearinghouse evidence standards, with or without reservations.

The contractor will assist i3 Scale-up and Validation grantees in meeting these standards.

For Development evaluations not using a design eligible for WWC review, the contractor will assess the evaluation according to the standards of “promise” in the document

“National Evaluation of i3 Analysis and Reporting Plan Version 1.0.”

Scale-Up Grants Validation Grants

Development Grants

Short-term performance measures The percentage of programs, practices, or strategies supported by a Scale-up grant with ongoing well-designed and independent evaluations that will provide evidence of their effectiveness at improving student outcomes at scale

The percentage of programs, practices, or strategies supported by a Validation grant with ongoing well-designed and independent evaluations that will provide evidence of their effectiveness at improving student outcomes

The percentage of programs, practices, or strategies supported by a Development grant with ongoing evaluations that provide evidence of their promise for improving student outcomes

The percentage of programs, practices, or strategies supported by a Scale-up grant with ongoing evaluations that are providing high-quality implementation data and performance feedback that allow for periodic assessment of progress toward achieving intended outcomes

The percentage of programs, practices, or strategies supported by a Validation grant with ongoing evaluations that are providing high-quality implementation data and performance feedback that allow for periodic assessment of progress toward achieving intended outcomes

The percentage of programs, practices, or strategies supported by a Development grant with ongoing evaluations that are providing high-quality implementation data and performance feedback that allow for periodic assessment of progress toward achieving intended outcomes

Long-term performance measures The percentage of programs, practices, or strategies supported

The percentage of programs, practices, or strategies supported

The percentage of programs, practices, or strategies supported

See http://ies.ed.gov/ncee/wwc/references/idocviewer/Doc.aspx?docId=19&tocId=4).

by a Scale-up grant that implement a completed well-designed, well-implemented and independent evaluation that provides evidence of their effectiveness at improving student outcomes at scale by a Validation grant that implement a completed well-designed, well-implemented and independent evaluation that provides evidence of their effectiveness at improving student outcomes by a Development grant with a completed evaluation that provides evidence of their promise for improving student outcomes

The percentage of programs, practices, or strategies supported by a Scale-up grant with a completed well-designed, well-implemented and independent evaluation that provides information about the key elements and the approach of the project so as to facilitate replication or testing in other settings

The percentage of programs, practices, or strategies supported by a Validation grant with a completed well-designed, well-implemented and independent evaluation that provides information about the key elements and the approach of the project so as to facilitate replication or testing in other settings

The percentage of programs, practices, or strategies supported by a Development grant with a completed evaluation that provides information about the key elements and approach of the project so as to facilitate further development, replication, or testing in other settings

The contractor (using staff that has not been involved in providing technical assistance) will characterize whether or not the evaluations meet the standards for a well-designed and well-implemented study (or a study showing promise). The contractor will also summarize the results of the i3 evaluations, including the use of meta-analysis within key areas of practices, strategies, and programs where feasible.

D. Project Design Issues

ED awarded a contract to Abt Associates in September of 2010 (EDIES10C0064) to conduct the technical assistance and analysis and reporting activities described in this

PWS for the 2010 cohort of i3 grantees (this contract was modified to include the 2011 cohort in July 2012). ED awarded a second contract to Abt Associates in December of

2012 (EDIES13C0005) to conduct the technical assistance and analysis and reporting activities described in this PWS for the 2012 cohort of i3 grantees. ED awarded a third contract to Abt Associates in December of 2013 (EDIES14C0007) to conduct the technical assistance and analysis and reporting activities described in this PWS for the

2013 cohort of i3 grantees.

The FY 2014 contract will continue this work for the FY 2014 (and through options through the FY 2015 and FY 2016 cohorts). It is important for the FY 2014 contractor to fully understand the work of the current three contracts to maintain high quality, consistent technical assistance, analysis, and reporting across i3 cohorts. Critical documents completed under the first contract (EDIES10C0064) are attached: National

Evaluation of i3 Analysis and Reporting Plan Version 1.0, Evaluation Design Summary

Template, Contrast Tool, and Customized Technical Assistance Plan template.

This contract will be awarded on a similar schedule as the 2014 i3 grants (by December

31, 2014). Thus, the contractor shall review the funded applications and use that review to inform the preparation of a technical assistance and analysis plan after award (see Task

2-Project Planning).

Discussed below are a number of areas of uncertainty and variation in the grants that will affect the plan. Other issues may arise as the project is carried out.

Wide scope of the i3 program o Unlike some ED programs that focus on a specific content area (for example, reading), i3 will fund innovations in a wide array of areas. The absolute priorities for the FY 2014 i3 grant competition are

Improving the Effectiveness of Teachers or Principals, Improving Low-Performing Schools, Improving Science, Technology, Engineering, and Mathematics

(STEM) Education, Implementing Internationally Benchmarked, College- and Career-

Ready Elementary and Secondary Academic Standards, Serving Rural Communities.

o The grant competition includes competitive preference priorities in a wide range of areas as well:

Improving Cost-Effectiveness and Productivity, Enabling Broad Adoption of Effective Practices, Supporting Novice i3 Applicants.

o The program will fund innovations for students from birth through grade

12 (including an invitational priority for supporting high-quality early learning).

o The program includes a number of possible outcome measures. Projects are to substantially and measurably improve student achievement or student growth, close achievement gaps, decrease dropout rates, increase high school graduation rates, or increase college enrollment and completion rates.

Variety of evaluation designs o Scale-up and Validation grantees are expected to fund randomized controlled trials or quasi-experimental designs.

o Development grantees are not required to use any particular evaluation design.

Variability of local evaluators’ qualifications and experience across grantees

Variation in grant length (36 – 60 months)

E. Conflicts of Interest

The contractor shall maintain independence between staff providing technical assistance and staff making assessments about the strength of the evaluations and conducting the cross-site analysis and reporting of findings. The contractor shall also ensure that none of their staff will serve as an i3 local evaluator or that there is a firewall between any staff who will serve as an i3 local evaluator and the staff working on this contract.

Applicants must identify one of these priorities for their project to address.

Note that ED could change the absolute priorities used for later competitions covered by the option periods of this contract.

F. Preliminary Assumptions

It is critical for technical assistance to be available early in the i3 grant implementation to ensure the strongest local evaluations possible. However, this timing does create some challenges for the contract, in that information about the grantees will only be available after contract award. Below, ED provides priorities and preliminary assumptions regarding the technical assistance and analysis plans for this contract. It may be necessary to modify the contract once details are available about funded applications

(number of grantees, types of interventions, types of evaluation designs, qualifications of evaluators) that could affect the types or amount of technical assistance required or the amount and types of analyses possible. As such, the contractor shall finalize the technical assistance and analysis plan (with COR review and approval) through the work of Task 2

(Project Planning).

ED’s expectation is that all i3 grantees, regardless of grant type or study design, will receive proactive, regular technical assistance. Should prioritization become necessary, the contractor will negotiate proposed changes to the technical assistance plan with ED.

There were 49 grantees in the 2010 cohort, 23 in the 2011 cohort, 20 in the 2012 cohort, and 25 in the 2013 cohort. The funding level for the FY 2014 competition is similar

($142 million) to the levels in 2011 and 2012 ($150 million) and 2013 ($142 million).

ED’s preliminary assumptions about reporting are to provide OII with information on the progress of the evaluations (including information they need to report on their GPRA performance measures) annually (during the base period and each option period), and a final report that also provides cross-site summary tables of findings for each cohort (in the 5th option period for the FY 2014 cohort, the 11th option period for the FY 2015 cohort, and the 17th option period for the FY 2016 cohort).

II. SCOPE OF WORK

This section specifies the tasks and subtasks that the contractor shall perform to fully execute all PWS requirements outlined in Sections A through R outlined below.

A. Base Contract (Year One of the FY 14 Cohort)

Task 1—Communicate with ED (Fixed-Price Task)

1.1 Kick-off Meeting. The contractor’s project director and key project staff (up to two) shall meet with the COR and other appropriate ED staff in Washington, DC within ten working days of the effective date of the contract award to discuss details regarding the tasks outlined in the proposal and other issues related to the contract.

The contractor shall provide an agenda for the meeting and come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within one week of the meeting, the contractor shall submit to the COR a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed, as well as any proposed updates to the plans and schedule.

Deliverables: Memo summarizing the kick-off meeting

1.2 Periodic Telephone Calls. The contractor shall participate in regular telephone calls with the COR to discuss the progress of the local evaluations and the contract, including any anticipated problems and proposed solutions to those problems. ED assumes calls every other week (during months 1 through 12 of the base contract period).

Within two working days of each call, the contractor shall submit to the COR a brief summary of the key issues and concerns discussed along with the contractor’s understanding of the COR’s technical direction.

Deliverables: Summaries of telephone calls

1.3 Quarterly Meetings. The contractor shall participate in quarterly meetings with the

COR and the OII program staff to discuss the progress of the local evaluations and any problems and proposed solutions to those problems. ED will provide an agenda for these meetings at least one week in advance, and will invite the contractor to make additions to the agenda. ED plans for half of the meetings to be conducted by conference call.

Within two working days of each meeting, the contractor shall submit to the COR a brief summary of the key issues and concerns discussed as well as the contractor’s understanding of next steps.

Deliverables: Summaries of meetings

1.4 Acquire Current Contract Files and Other Knowledge. As described earlier, the work in this contract has already been ongoing for earlier cohorts of i3 grantees through three existing contracts (EDIES10C0064, EDIES13C0005, and EDIES14C0007).

Early in the FY 2014 contract, it will be critical for the contractor to acquire knowledge of the technical assistance and analysis and reporting guidance already in place from the current contractor to ensure work continues seamlessly (without delays and without inconsistencies).

Within two weeks after the effective date of this contract, the contractor shall submit to ED a plan for reviewing all necessary files and information from the current contracts. In the plan, the contractor shall discuss the types of information to be obtained, such as the data files, data collection instruments, and existing written guidance to ensure consistency between the current contract and the FY 2014 contract

(for example, the final report for the FY 2014 cohort must use the same analysis plan and assessment criteria as used in the current contracts, and the current registry of technical assistance resources must be maintained).

Deliverables: i3 Contract Reconnaissance and Continuity Plan

Task 2— Project Planning (Cost Reimbursement Task)

As mentioned in Section C (Project Design Issues), the 2014 cohort of i3 grants will be awarded on a similar schedule to this contract. Since the quality of the designs proposed and the qualifications and experience of the proposed local evaluators must drive the technical assistance, and the proposed designs and types of interventions proposed must drive the analysis plan, the contractor shall review the funded applications in order to develop the project technical assistance and analysis plan.

the contractor will negotiate proposed changes to the technical assistance plan with ED.

There were 49 grantees in the 2010 cohort, 23 in the 2011 cohort, 20 in the 2012 cohort, performance measures) annually (during the base period and each option period), and a final report that also provides cross-site summary tables of findings for each cohort (in

The contractor shall review the funded i3 applications to summarize their characteristics and to assess the evaluation technical assistance needs and analytic possibilities for i3 grantees. The contractor will submit a memo to the COR within eight weeks of the effective date of the contract that summarizes the types of interventions grantees will implement and test, the proposed evaluation designs, and the areas on which technical assistance is needed. Based on the review of the funded applications, the contractor shall submit a draft technical assistance and analysis plan to the COR within 14 weeks of the effective date of the contract. The COR will review the draft plan and provide any comments within two weeks of receipt. The contractor shall make revisions in response to COR comments and submit a revised plan to the COR within 18 weeks of the effective date of the contract. The contractor shall request, in writing, COR approval for any subsequent changes to the approved plan. If approved, the contractor shall then revise the approved final plan document to incorporate the approved changes.

In the technical assistance plan, the contractor shall summarize the strengths and weaknesses of the i3 local evaluation plans, include draft feedback for each of the local evaluators, discuss the types and frequency of technical assistance to be provided to local evaluators (for example, sessions at annual grantee meetings held by the Office of

Innovation and Improvement (OII), monthly calls, maintenance of the current registry of technical assistance resources), describe how well the draft designs conform to the i3 criteria developed under contract EDIES10C0064 and/or what additional information is necessary to make that determination, and discuss how technical assistance staff will be trained to provide consistent feedback and other assistance. Upon receipt of COR approval, the contractor shall disseminate the grantee-specific evaluation feedback to each grantee and local evaluator.

In the analysis plan, the contractor shall document the contents to be included in the year one GPRA report. The contractor shall document proposed methods for assessing the extent to which the local independent evaluations are well-designed and being well-implemented, and for summarizing the progress of the evaluations.

Deliverables: Application summary memo

Draft technical assistance and GPRA report analysis plan

Revised technical assistance and GPRA report analysis plan

Task 3 – Provide Technical Assistance to i3 Independent Local Evaluators (Cost

Reimbursement Task)

Following the COR-approved technical assistance plan (Task 2-Project Planning), the contractor shall provide technical assistance to i3 grantees’ independent local evaluators on the design and implementation of their evaluations during the first year of the grants

(months 1 through 12 of the base contract period). The contractor shall submit a biweekly update on the technical assistance (focusing on any known risks to the evaluations) to the COR at least one workday prior to each biweekly call (see Task 1.2).

Deliverables: Biweekly reports on technical assistance

Task 4—Analysis and Reporting (Cost Reimbursement Task)

4.1 Produce Year One GPRA report for FY 2014 cohort. The purpose of the year one

GPRA report is to update ED on the progress of the local independent evaluations, including the extent to which the evaluations are well-designed and being well-implemented for the purpose of supporting ED with GPRA reporting. This report will not go through the IES peer review process, but will undergo review by at least the COR and OII and will potentially be posted on the ED website.

The contractor shall write and organize the report so that they are accessible to policy makers and research-savvy practitioners rather than for academic researchers and suitable for posting on the OII i3 website. Should ED choose to post these reports on the website, OII will be responsible for posting the reports.

In order to ensure objectivity to analysis and reporting, the contractor shall develop procedures to maintain independence between the staff that provide technical assistance to the evaluators and the staff that assess the extent to which evaluations are well-designed and well-implemented, analyze, and report the findings.

The contractor shall submit to the COR a draft of the year one GPRA report within 52 weeks of the effective date of the contract. The COR will review and provide any comments within two weeks of receipt. The contractor shall make revisions in response to COR comments and submit to the COR a revised year one GPRA report within 56 weeks of the effective date of the contract. ED will review and provide any comments within two weeks of receipt. The contractor shall make revisions in response to ED comments and submit the final year one GPRA report to the COR within 60 weeks of the effective date of the contract.

Deliverables: Draft Year One GPRA Report

Revised Year One GPRA Report

Final Year One GPRA Report

Task 5—Reporting Requirements (Fixed-Price Task)

The contractor shall prepare monthly reports and submit the reports to the COR and the

CS within ten workdays after the end of each calendar month of months 1 through 12 of the base contract period and shall be submitted simultaneously with the submission of the contractor’s voucher charging ED for the work completed during the month for which the performance is being reported. The following reports are required:

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. In the monthly progress reports, the contractor shall summarize the major activities and accomplishments for the reporting period. In addition, the reports shall provide information for each project task and subtask regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the contract, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member (including staff from subcontractors) the hours charged by task and subtask. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures; segregating project costs by individual and by task and subtask, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly reports

B. Option Period 1 (Year Two of the FY 14 Cohort)

ED will make a decision about exercising Option Period 1 by September 19, 2015. If exercised, the 14-month Option Period 1 will begin by the 13 th month (expected to be

December 19, 2015) and end after the 26 th month (expected to be February 5, 2017), overlapping with the base period by 2 months.

Task 6—Communicate with ED (Fixed-Price Task)

6.1 Periodic Telephone Calls. The contractor shall participate in regular telephone calls with the COR to discuss the progress of the local evaluations and the contract, including any anticipated problems and proposed solutions to those problems. ED assumes calls every other week (during months 13 through 24 of Option Period 1).

6.2 Quarterly Meetings. The contractor shall participate in quarterly meetings with the COR and the OII program staff to discuss the progress of the local evaluations and any problems and proposed solutions to those problems. ED will provide an agenda for these meetings at least one week in advance, and will invite the contractor to make additions to the agenda. ED plans for half of the meetings to be conducted by conference call.

Task 7 – Provide Technical Assistance to i3 Independent Local Evaluators (Cost on the design and implementation of their evaluations during the second year of the grants (months 13 through 24 of Option Period 1). The contractor shall submit a biweekly update on the technical assistance (focusing on any known risks to the evaluations) to the COR at least one workday prior to each biweekly call (see Task 6.1).

Task 8—Analysis and Reporting (Cost Reimbursement Task)

8.1 Produce Year Two GPRA report for FY 2014 cohort. The purpose of the year two including the extent to which the evaluations are well-designed and being well-implemented for the purpose of supporting ED with GPRA reporting. This report will

The contractor shall submit to the COR a draft of the year two GPRA report within 104 comments within two weeks of receipt. The contractor shall make revisions in response to COR comments and submit to the COR a revised year two GPRA report within 108 weeks of the effective date of the contract. ED will review and provide any comments within two weeks of receipt. The contractor shall make revisions in response to ED comments and submit the final year two GPRA report to the COR within 112 weeks of the effective date of the contract.

Deliverables: Draft Year Two GPRA Report

Revised Year Two GPRA Report

Final Year Two GPRA Report

Task 9—Reporting Requirements (Fixed-Price Task)

CS within ten workdays after the end of each calendar month of months 13 through 24 of

Option Period 1 and shall be submitted simultaneously with the submission of the monthly progress reports, the contractor shall summarize the major activities and accomplishments for the reporting period. In addition, the reports shall provide information for each project task and subtask regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the contract, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member (including staff from subcontractors) the hours charged by task and subtask. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures; segregating project costs by

C. Option Period 2 (Year Three of the FY 14 Cohort)

ED will make a decision about exercising Option Period 2 by September 18, 2016. If exercised, the 14-month Option Period 2 will begin by the 25 th month (expected to be

December 18, 2016) and end after the 38 th month (expected to be February 3, 2018), overlapping with Option Period 1 by 2 months.

Task 10—Communicate with ED (Fixed-Price Task)

10.1 Periodic Telephone Calls. The contractor shall participate in regular telephone calls every other week (during months 25 through 36 of Option Period 2).

10.2 Quarterly Meetings. The contractor shall participate in quarterly meetings with the

COR and the OII program staff to discuss the progress of the local evaluations and any problems and proposed solutions to those problems. ED will provide an agenda for these meetings at least one week in advance, and will invite the contractor to make additions to the agenda. ED plans for half of the meetings to be conducted by conference call.

Within two working days of each meeting, the contractor shall submit to the COR a

Task 11 – Provide Technical Assistance to i3 Independent Local Evaluators (Cost on the design and implementation of their evaluations during the third year of the grants

(months 25 through 36 of Option Period 2). The contractor shall submit a biweekly update on the technical assistance (focusing on any known risks to the evaluations) to the

COR at least one workday prior to each biweekly call (see Task 10.1).

Task 12—Analysis and Reporting (Cost Reimbursement Task)

12.1 Produce Year Three GPRA report for FY 2014 cohort. The purpose of the year three including the extent to which the evaluations are well-designed and being well-implemented for the purpose of supporting ED with GPRA reporting. This report will

The contractor shall submit to the COR a draft of the year three GPRA report within 156 comments within two weeks of receipt. The contractor shall make revisions in response to COR comments and submit to the COR a revised year three GPRA report within 160 weeks of the effective date of the contract. ED will review and provide any comments within two weeks of receipt. The contractor shall make revisions in response to ED comments and submit the final year three GPRA report to the COR within 164 weeks of

Deliverables: Draft Year Three GPRA Report

Revised Year Three GPRA Report

Final Year Three GPRA Report

Task 13—Reporting Requirements (Fixed-Price Task)

CS within ten workdays after the end of each calendar month of months 25 through 36 of

Option Period 2 and shall be submitted simultaneously with the submission of the monthly progress reports, the contractor shall summarize the major activities and accomplishments for the reporting period. In addition, the reports shall provide information for each project task and subtask regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the contract, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member (including staff from subcontractors) the hours charged by task and subtask. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures; segregating project costs by

D. Option Period 3 (Year Four of the FY 14 Cohort)

ED will make a decision about exercising Option Period 3 by September 16, 2017. If exercised, the 14-month Option Period 3 will begin by the 37 th month (expected to be

December 16, 2017) and end after the 50 th month (expected to be February 2, 2019), overlapping with Option Period 2 by 2 months.

Task 14—Communicate with ED (Fixed-Price Task)

14.1 Periodic Telephone Calls. The contractor shall participate in regular telephone calls with the COR to discuss the progress of the local evaluations and the contract, including any anticipated problems and proposed solutions to those problems. ED assumes calls every other week (during months 37 through 48 of Option Period 3).

14.2 Quarterly Meetings. The contractor shall participate in quarterly meetings with the problems and proposed solutions to those problems. ED will provide an agenda for these

Task 15 – Provide Technical Assistance to i3 Independent Local Evaluators (Cost on the design and implementation of their evaluations during the fourth year of the grants

(months 37 through 48 of Option Period 3). The contractor shall submit a biweekly

COR at least one workday prior to each biweekly call (see Task 14.1).

Task 16—Refine Analysis Plan (Fixed-Price Task)

For the final report for the FY 2014 cohort, the contractor shall prepare a detailed data analysis plan that documents the intended analyses to be conducted and included in the reports, as well as an annotated outline for the reports’ contents and organization. In the data analysis plan, the contractor shall document proposed methods for assessing the extent to which the local independent evaluations are well-designed and being well-implemented, and for summarizing the progress of implementation of the practice, strategy, or program being tested as well as the progress of the evaluations. In addition to the preceding items, the contractor shall include in the analysis plan a discussion of the proposed methods for presenting a summary of findings to date across sites in table format as well as table shells (including a discussion of meta-analysis within key areas of practices, strategies, and programs where feasible).

The contractor shall provide the draft analysis plan for the final reports within 162 weeks of the effective date of the contract. The COR will review and provide any comments within four weeks. The contractor shall make revisions in response to COR comments and submit a revised analysis plan to the COR within 168 weeks of the effective date of the contract.

The contractor shall request, in writing, COR approval for any subsequent changes to the approved analysis plan. If approved, the contractor shall then revise the approved analysis plan document to incorporate the approved changes.

Deliverables: Draft Analysis Plan for FY 2014 cohort Final Report

Revised Analysis Plan for FY 2014 cohort Final Report

Task 17 – Prepare Data Collection Forms and Office of Management and Budget

(OMB) Data Collection Clearance Package (Fixed-Price Task)

17.1 Obtain Institutional Review Board (IRB) Approval. If necessary, the contractor shall ensure that all aspects of the evaluation have been approved by the appropriate

IRBs, including its own IRB. Documentation of all IRB approvals shall be submitted to the COR no later than 52 weeks after the effective date of the contract.

17.2 Prepare data collection forms and Obtain OMB Data Collection Clearance. The contractor shall design data collection instruments to address the key evaluation questions. The contractor shall prepare an OMB data collection clearance package that describes the study design and data collection instruments and activities, and includes the necessary forms required for OMB approval. The contractor shall ensure that the OMB data collection clearance package justifies the necessity for collecting the data and comprehensively responds to each required item in the instructions. In addition, the contractor shall ensure that the OMB data collection clearance package includes: (1) the study mandate and objectives; (2) the research questions; (3) a crosswalk between the research questions and the information to be collected; (4) instructions to study participants; (5) consent forms if needed; (6) description of and justification for incentives; (7) discussion of questions likely to be deemed “sensitive” by OMB; (8) limitations of the study and its potential use for policy decisions; and (9) table shells indicating how data will be tabulated and analyzed. In addition, the contractor shall include the anticipated response rates.

The contractor shall submit the draft OMB data collection clearance package to the

COR within 162 weeks of the effective date of the contract. The COR will review the draft and provide any comments within four weeks of receipt. The contractor shall make revisions in response to COR comments and submit the revised OMB data collection clearance package to the COR within 168 weeks of the effective date of the contract. When acceptable, the COR will submit the OMB data collection clearance package for clearance.

The contractor shall submit the OMB data collection clearance package mindful that the OMB clearance process requires a minimum of 120 days and usually takes at least five months from ED’s submission of the package (the revised package due within

168 weeks of the effective date of the contract) into the clearance process to receipt of

OMB clearance.

The contractor shall be prepared to make revisions to the OMB data collection clearance package, to address questions that arise during the clearance process, and to submit to the COR any necessary modifications to the cleared OMB data collection clearance package such as extension requests.

Deliverables: Documentation of IRB approvals

Draft OMB data collection clearance package

Revised OMB data collection clearance package

Task 18 – Data Collection (Cost Reimbursement Task)

Using instruments and plans approved by the COR and OMB, the contractor shall collect data necessary to answer the key evaluation questions.

The contractor, as appropriate, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the

Federal common rule or ED’s final regulations on the protection of human research participants.

The contractor shall be required to establish procedures at its sites and at subcontractor sites to ensure confidentiality of data. Their systems shall be required to provide reasonable assurance that information identifying individuals is in files physically separated from other research data. The contractor and subcontractors shall be required to maintain security of the complete set of all master data files and documentation.

Access to individually identifiable data shall be strictly controlled. At each site all data shall be kept in locked file cabinets during nonworking hours, and work on hardcopy data shall take place in a single room, except for data entry. Physical security of electronic data shall also be maintained. Security features that protect project data shall include password-protected accounts that authorize users to use the contractor’s and subcontractors’ information systems but only to access specific network directories and network software; user rights and directory and file attributes that limit those who can use particular directories and files and determine how they can use them; and, additional security features that the network administrators establish for projects as needed.

Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) such data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act (20 U.S.C. 9573).

In safeguarding personally identifiable information (PII), the contractor and its subcontractors also shall be subject to ED’s requirements contained in ED’s Handbook for the Protection of Sensitive But Unclassified Information, OCIO-15, and ED’s policy that the transmission of sensitive but unclassified information, including PII, through an e-mail requires that the contents be password protected in a ZIP file.

18.1 Train data collection staff. To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency, the contractor shall develop a training plan for data collectors. The types of data collection will be determined through Task 2 (Project Planning) and could be as simple as collecting existing reports of the grantees' local evaluations or as complex as assisting the local evaluators in obtaining student achievement data from districts and schools. The contractor shall base the data collector training plan on the COR and OMB-approved data collection plan (Task 2-Project Planning and Task 17-Prepare Data Collection

Forms and OMB Data Collection Clearance Package). The contractor shall submit the training plan to the COR within 158 weeks of the effective date of the contract.

The contractor shall train all persons who collect, code, or otherwise have access to raw data in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request.

Deliverables: Training Plan

18.2 Collect data. Following COR and OMB-approved plans (Task 2-Project Planning and Task 17-Prepare Data Collection Forms and OMB Data Collection Clearance

Package), the contractor shall collect data (and/or local evaluation products for data extraction) to address the evaluation questions. The contractor shall submit biweekly updates on the status of the data collection to the COR during each period of data collection (expected to be December 2017 through March 2018).

Deliverables: Biweekly progress reports on data collection during periods of data collection

Task 19—Analysis and Reporting (Cost Reimbursement Task)

19.1 Produce Year Four GPRA report for FY 2014 cohort. The purpose of the year four including the extent to which the evaluations are well-designed and being well-implemented for the purpose of supporting ED with GPRA reporting. This report will

The contractor shall submit to the COR a draft of the year four GPRA report within 208 comments within two weeks of receipt. The contractor shall make revisions in response to COR comments and submit to the COR a revised year four GPRA report within 212 weeks of the effective date of the contract. ED will review and provide any comments within two weeks of receipt. The contractor shall make revisions in response to ED comments and submit the final year four GPRA report to the COR within 216 weeks of

Deliverables: Draft Year Four GPRA Report

Revised Year Four GPRA Report

Final Year Four GPRA Report

Task 20—Reporting Requirements (Fixed-Price Task)

CS within ten workdays after the end of each calendar month of months 37 through 48 of

Option Period 3 and shall be submitted simultaneously with the submission of the monthly progress reports, the contractor shall summarize the major activities and accomplishments for the reporting period. In addition, the reports shall provide information for each project task and subtask regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the contract, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member (including staff from subcontractors) the hours charged by task and subtask. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures; segregating project costs by

E. Option Period 4 (Year Five of the FY 2014 Cohort)

ED will make a decision about exercising Option Period 4 by September 15, 2018. If exercised, the 14-month Option Period 4 will begin by the 49 th month (expected to be

December 15, 2018) and end after the 62 nd month (expected to be February 1, 2020), overlapping with Option Period 3 by 2 months.

Task 21—Communicate with ED (Fixed-Price Task)

21.1 Periodic Telephone Calls. The contractor shall participate in regular telephone calls with the COR to discuss the progress of the local evaluations and the contract, assumes calls every other week (during months 49 through 60 of Option Period 4).

21.2 Quarterly Meetings. The contractor shall participate in quarterly meetings with the problems and proposed solutions to those problems. ED will provide an agenda for these

Task 22 – Provide Technical Assistance to i3 Independent Local Evaluators (Cost on the design and implementation of their evaluations during the fifth year of the grants

(months 49 through 60 of Option Period 4). The contractor shall submit a biweekly

COR at least one workday prior to each biweekly call (see Task 21.1).

Task 23 – Data Collection (Cost Reimbursement Task)

20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the reasonable assurance that information identifying individuals is in files physically separated from other research data. The contractor and subcontractors shall be required to maintain security of the complete set of all master data files and documentation.

Access to individually identifiable data shall be strictly controlled. At each site all data shall be kept in locked file cabinets during nonworking hours, and work on hardcopy data shall take place in a single room, except for data entry. Physical security of electronic data shall also be maintained. Security features that protect project data shall include password-protected accounts that authorize users to use the contractor’s and subcontractors’ information systems but only to access specific network directories and network software; user rights and directory and file attributes that limit those who can use particular directories and files and determine how they can use them; and, additional security features that the network administrators establish for projects as needed.

Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) such data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act (20 U.S.C. 9573).

In safeguarding personally identifiable information (PII), the contractor and its subcontractors also shall be subject to ED’s requirements contained in ED’s Handbook for the Protection of Sensitive But Unclassified Information, OCIO-15, and ED’s policy

23.1 Train data collection staff. To ensure that the data collection and coding staff have shall develop a training plan…