Draft_Performance_Work_Statement.pdf

Attachment A – Performance Work Statement

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

FEASIBILITY STUDY FOR AN IMPACT EVALUATION OF

PROMISE NEIGHBORHOODS

Performance Work Statement

August 14, 2017

I. INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (NCEE) of the U.S. Department of Education’s (the Department) Institute of Education Sciences (IES) needs to assess the feasibility of conducting an Impact Evaluation of Promise Neighborhoods. Promise Neighborhoods (the Program) is a discretionary federal grant program authorized by Title IV Section 4624 of the Elementary and Secondary Education Act as amended by the Every Student Succeeds Act (ESSA). The Program competitively awards grants to eligible non-profit organizations, institutions of higher education, and Indian tribes, who typically work in partnership with a number of other local organizations to plan for and implement place-based initiatives that provide a comprehensive system of community, school, and family supports. The ultimate goal is to help ensure that children living in the nation’s most distressed neighborhoods receive a high-quality education and then successfully transition to postsecondary education and into the workforce. The purpose of this feasibility study is to inform a potential future evaluation effort that examines the impacts and implementation of activities funded under the Program. The final proposed study design options must use the most credible scientific methods feasible, and have the potential to provide information about the Program that is useful to policymakers and practitioners.

A. Legislative Authority

An evaluation of the Promise Neighborhoods Program is mandated by Title IV Section 4624 of ESSA, which states that “The Secretary shall reserve not more than 5 percent of the funds made available under section 4601(b)(2)(A) to provide technical assistance and evaluate the implementation and impact of the activities funded under this section, in accordance with section 8601.”

B. Background1

While the Promise Neighborhoods Program was first formally authorized with the passage of ESSA in December 2015, the Department has been awarding Promise Neighborhood grants since Fiscal Year 2010 under the legislative authority of the Fund for the Improvement of Education, Title V Sections 5411-5413 of the Elementary and Secondary Education Act as amended by the No Child Left Behind Act. This Fund supported nationally significant programs to improve the quality of elementary and secondary education at the State and local levels and to help all children meet challenging State academic content and student academic achievement standards.

1 This section includes a brief summary of the Promise Neighborhoods Program. For detailed information, see:

https://www2.ed.gov/programs/promiseneighborhoods/index.html.

https://www2.ed.gov/programs/promiseneighborhoods/index.html

The Department invested a total of $363 million in Promise Neighborhood grants between 2010 and 2016, with 64 grants awarded to support approximately 50 unique Promise Neighborhoods for up to five years. The grantees are non-profit organizations, institutions of higher education, or Indian tribes. While each grantee is ultimately responsible for planning and implementing their Promise Neighborhood, they typically work in partnership with a number of other organizations (e.g., schools, social services providers, mental health providers) to provide pipeline services to eligible families. Some organizations initially received a planning grant and then subsequently received an implementation grant, which is why there are more grants (64) than unique Promise Neighborhoods served (50). The Department is investing approximately $73 million more in Fiscal Year 2017, which will include the first cohort of grantees under the ESSA-authorized Promise Neighborhoods Program.

The purpose of the Program is to provide the nation’s most distressed neighborhoods with high-quality pipeline services. Grantees must demonstrate through a needs assessment that the neighborhood they propose to serve is distressed. Indicators may include a large number of low-performing schools, and high rates of poverty, unemployment, truancy, drug abuse, teen pregnancy, childhood obesity, academic failure, crime, and juvenile delinquency or incarceration. Grantees must also have a plan to deliver a continuum of integrated supports as children move from birth through postsecondary education and into the workforce. These supports should be evidence-based to the extent possible. While the pipeline services are meant to be comprehensive and coordinated, the Program’s theory of action maintains that strong schools are central to the creation of a successful Promise Neighborhood. The pipeline services that a grantee must offer include activities to support:

• High-quality early childhood education programs

• High-quality school and out-of-school-time programs and strategies

• Transitions between all levels of schooling

• Family and community engagement at school and at home

• Workforce readiness, such as job training, internships, or career counseling

• Social, health, nutrition, and mental health services

• Juvenile crime prevention and rehabilitation programs

Given the comprehensive nature of services that grantees are expected to provide, the Program similarly expects grantees to positively impact a diverse set of outcomes, including:

• Number and percentage of children in kindergarten who demonstrate at the beginning of the program or school year age-appropriate functioning across multiple domains of early learning as determined using developmentally-appropriate early learning measures

• Number and percentage of students at or above grade level according to state mathematics assessments in at least the grades required by ESSA (3rd through 8th grades and once in high school)

• Number and percentage of students at or above grade level according to state English language arts assessments in at least the grades required by ESSA

• Attendance rate of students in 6th, 7th, 8th, and 9th grade as defined by average daily attendance

• Chronic absenteeism rate of students in 6th, 7th, 8th, and 9th grades

• Four-year adjusted cohort graduation rate

• Number and percentage of Promise Neighborhood students who enroll in a two-year or four-year college or university after graduation

• Number and percent of Promise Neighborhood students who graduate from a two-year or four-year college or university or vocational certification completion

• Number and percentage of children who consume five or more servings of fruits and vegetables daily

• Number and percentage of children who feel safe at school and traveling to and from school as measured by a school climate survey

• Student mobility rate

• Number and percentage of parents or family members that read to or encourage their children to read three or more times a week or reported their child read to themselves three or more times a week (birth–8th grade)

• Number and percentage of parents/family members who report talking about the importance of college and career (9th-12th grade)

• Number and percentage of students who have school and home access to broadband internet and a connected computing device

In sum, the structure of the Promise Neighborhoods Program reflects a belief that children who grow up in distressed communities face risks of experiencing negative life outcomes that are compounded, and that counteracting these risks requires a continuum of comprehensive, coordinated supports.

The Program was intended to support the replication and scale-up of place-based initiatives like the Harlem Children’s Zone,2 which provides education and community supports to low-income families in Harlem, New York. Exploiting a lottery that randomly offered interested students the opportunity to attend the Harlem Children’s Zone’s Promise Academy charter school, researchers found that up to six years later, the school had positive impacts on students’ academic outcomes such as math and reading, as well as non-academic outcomes such as teen pregnancy and incarceration.3 These and other researchers have presented evidence suggesting that the positive impacts are mostly being driven by the quality of the charter school and not by the other services and supports provided by the Harlem Children’s Zone.4 However, this evidence is not based on a rigorous lottery design and so should be interpreted with more caution.

Aside from the Harlem Children’s Zone studies, there are no impact studies to date of place-based initiatives funded by the Promise Neighborhoods Program. There have been some case studies of implementation and some impact studies of individual components of an initiative, but no studies that have looked holistically at program impacts on student outcomes.5 Given the limited evidence and that the Department is continuing to invest in the Promise Neighborhoods Program (over $400 million to date); it is important to have more comprehensive evidence on the impacts of these initiatives.

C. Key Research Questions

The purpose of this feasibility study is to generate design options that can provide high-quality and useful evidence on the impact of the Promise Neighborhoods Program on student outcomes. In performing the tasks described in this Performance Work Statement (PWS), the contractor shall propose designs for

2 http://hcz.org/ 3 Dobbie, W., & Fryer, R. (2011). Are High-Quality Schools Enough to Increase Achievement among the Poor?

Evidence from the Harlem Children’s Zone. American Economic Journal: Applied Economics. 3(3), 158-187.

Dobbie, W., & Fryer, R. (2015). The Medium-Term Impacts of High-Achieving Charter Schools. Journal of

Political Economy. 123(5), 985-1037.

4 Dobbie and Fryer, 2011.

Whitehurst, G., & Croft, M. (2010). The Harlem Children’s Zone, Promise Neighborhoods, and the Broader, Bolder

Approach to Education. Washington, DC: Brown Center on Education Policy at Brookings.

5 Horsford, S., & Sampson, C. (2014). Promise Neighborhoods: The Promise and Politics of Community Capacity

Building as Urban School Reform. Urban Education. 49(8), 955-991.

Hulsey, L., Esposito, A., Boller, K., & Osborn, S. (2015). Promise Neighborhoods Case Studies – Final Report.

Princeton, NJ: Mathematica Policy Research.

http://hcz.org/ conducting a full evaluation via a subsequent contract, and thoroughly assess the credibility, feasibility, and tradeoffs of each proposed design.

The design options report for an Impact Evaluation of Promise Neighborhoods shall address the following key questions:

• What are the most policy-relevant impact questions that can be credibly answered about the

Promise Neighborhoods Program? For example, is it the impact of the federal grant money?

Is it the impact of implementing an average program funded by Promise Neighborhood grants? Is it the impact of a particular Promise Neighborhood initiative? Is it the impact of the school-based part of the Program relative to other family or community parts of the Program?

Is it the impact of specific components or activities that are part of a typical Promise Neighborhoods initiative?

• What are the most credible and feasible study designs for each of the relevant impact questions?

• What is the most credible and feasible counterfactual for each of the relevant impact questions? How will an appropriate comparison group be identified? What about recruitment?

• What are the most appropriate and relevant outcomes to collect for each of the relevant impact questions? How will these outcomes be measured, and how often will they be collected?

• What coordination with the Program grantees would be helpful (or necessary) for carrying out each of the proposed study design options, and what are the key required timelines to ensure that such coordination is feasible?

II. PERIOD OF PERFORMANCE

The period of performance will be one 12-month Base Year.

III. SCOPE OF WORK

The specific tasks and subtasks that the contractor shall perform are described below.

Task 1—Kick-Off Meeting with the Department

The contractor’s project director and up to three key project staff shall meet with the Contracting Officer (CO), Contracting Officer’s Representative (COR) and other appropriate Department staff within two weeks of contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, D.C. The contractor shall come prepared to identify any areas of concern not contained in the accepted technical proposal or discussed during contract negotiations, and to suggest ways of responding to these concerns. Within one week following the meeting, the contractor shall submit a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the study. The COR will review the memo within one week. If any changes are needed, the contractor shall make them within two business days.

Deliverables: Kick-Off Meeting Summary

Task 2—Identify and Explore Study Design Options

2.1 Develop and Refine Feasibility Discussion Protocol and Plan: The contractor shall develop and submit a protocol and plan to address each of the key research questions (Section C) no later than five weeks after the Kick-off Meeting (Task 1). The protocol shall include a section to guide discussions with current Promise Neighborhood grantees and relevant partners, as well as a section to guide discussions with districts or schools that reside in the area currently served by the Promise Neighborhood. The protocol shall also be relevant or adaptable for discussions with place-based initiative providers, and districts or schools that may not be currently receiving Promise Neighborhood grants. The general purpose of this protocol shall be to elicit information that will inform study design options, as well as the feasibility of recruitment, data collection, and timelines for an impact evaluation of Promise Neighborhoods. The plan that accompanies the protocol shall explain the contractor’s approach to determining how many and which grantees, non-grantees, schools, and districts to have discussions with.

The plan shall also explain how each of the proposed items in the protocol will inform the key research questions related to feasibility and design options. The COR will review and provide feedback on the draft feasibility plan and protocol within two weeks of submission. Based on COR feedback, the contractor shall refine and submit a final version of the feasibility plan and protocol no later than nine weeks after the kick-off meeting.

2.2 Implement Feasibility Plan and Protocol: After the final feasibility plan and protocol is approved by the COR, the contractor shall implement the plan and protocol. As part of implementation, the contractor shall conduct interviews using this protocol with up to 9 place-based initiative providers and 9 districts or schools. The feasibility plan and protocol shall be fully implemented no later than 17 weeks after the kick-off meeting to ensure sufficient time to prepare the draft study design options report (see Task 3).

Deliverables: Draft Feasibility Plan and Protocol Final Feasibility Plan and Protocol

Task 3—Study Design Options Report

The contractor shall prepare a study design options report that addresses the issues raised in Section B (Background) and Section C (Key Research Questions) of the PWS, drawing in part on the results of implementing the feasibility plan and protocol (Subtask 2.2). The report shall include the pros and cons of each study design option, both in terms of its methodological quality and its practical ease to implement in the field. The report shall also discuss instrumentation, data collection, and analysis issues, particularly as they relate to implementing each of the proposed study design options. The contractor shall provide a draft of the study design options report to the COR 12 weeks prior to the Technical Work Group (TWG) meeting. The COR will review and provide feedback on this draft within two weeks of receipt. The study design options report shall be used as materials for informing the TWG of the study design options prior to the TWG meeting (Task 4). The study design options report shall also be used to prepare presentation materials for the TWG meeting (Task 4). These presentation materials and an agenda shall be submitted to the COR four weeks prior to the TWG meeting. The COR will review and provide feedback on these materials within one week of receipt. The contractor shall make any changes that the COR requires no later than two weeks prior to the TWG meeting.

The contractor shall revise the study design report as appropriate based on the COR’s feedback on the draft report and subsequent to the discussion with the TWG (Task 4). The contractor shall submit this final study design options report no later than 50 weeks after contract award. If TWG members recommend any significant deviations from the study design, data collection, or data analysis options indicated in the study design options report, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study design options report based solely upon the recommendation of TWG members without the written approval of the COR. On an as needed basis, the contractor may have individual TWG members review products to assist with specific issues on which they have expertise. In these cases, there is no need to convene all TWG members in a group meeting to discuss a limited, specific set of issues.

Deliverables: Draft Study Design Options Report TWG Materials Based on Study Design Options Report Final Study Design Options Report

Task 4—Technical Working Group

4.1 Establish Technical Working Group: The role of the TWG will be to advise the contractor and the Department on the conduct of a potential evaluation of Promise Neighborhoods, including but not limited to: study design, instrumentation, data collection, and analysis. The TWG shall be comprised of up to 12 individuals, and must include national experts in: place-based initiatives (particularly those most related to the Promise Neighborhoods Program), specific content areas that are often components of place-based initiatives (e.g., early learning programs, after-school programs, family/community engagement programs, programs to facilitate school transitions, etc.), evaluation methodology (both experimental and quasi-experimental, as well as conducting high-quality impact studies in the context of large federal grant programs). The contractor shall submit a list of proposed TWG members no later than three weeks after the kick-off meeting (Task 1). When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. The list shall also clearly explain the background and expertise of each proposed TWG member, and how it is specifically relevant to the needs of the study. The COR will review the proposed TWG list within three business days of receipt. The contractor shall address any feedback and make any changes that the COR requires no later than three business days after receiving feedback. After COR approval of the proposed TWG members, the contractor shall contact each approved member to secure interest and intention to serve on the TWG.

4.2 Convene Technical Working Group: The contractor shall convene a meeting of the TWG no later than 36 weeks after contract award. The TWG meeting shall focus on the proposed key research questions, the study design options, identification and recruitment feasibility of a relevant counterfactual comparison group, data collection, and analysis issues. The contractor shall submit a written summary of the meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The COR will review the written summary within one week of receipt. The contractor shall address any feedback and make any changes that the COR requires no later than one week after receiving feedback. Proposed actions shall not go forward unless approved by the COR. This one-day meeting shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Honorarium amounts shall be based on the number of hours that TWG members are expected to provide, and the hourly rate of the honorarium should not exceed the current General Schedule (GS)-15, Step 10 hourly rate without justification. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Deliverables: List of proposed TWG members Summary of TWG meeting

Task 5—Reporting Requirements

The contractor shall submit one copy of the following performance reports each month through the Invoice Processing Platform. The COR will review these reports within one week of receipt. Any feedback by the COR shall be addressed by the contractor within 2 business days.

Monthly Progress Report/Exception Reports: The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task or subtask regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the COR.

Monthly Manpower/Expenditure Reports: The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures that segregate project costs by individual and by task or subtask, and specify for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure reports

IV. SCHEDULE OF DELIVERABLES

All deliverables noted below shall be submitted to the Department’s COR, Contract Specialist (CS), and CO electronically, unless otherwise noted. The COR reserves the right to inspect the contractor’s work at any point while in progress.

Tasks and Deliverables Due Dates

Task 1—Kick-Off Meeting with the Department Kick-Off Meeting summary 1 week after kick-off meeting Task 2—Identify and Explore Study Design Options Draft Feasibility Plan and Protocol 5 weeks after kick-off meeting Final Feasibility Plan and Protocol 9 weeks after kick-off meeting Task 3—Study Design Options Report Draft Study Design Options Report 24 weeks after award TWG Materials Based on Study Design Options Report 32 weeks after award Final Study Design Options Report 50 weeks after award Task 4—Technical Working Group List of proposed TWG members 3 weeks after kick-off meeting Summary of TWG meeting 1 week after TWG meeting

Task 5—Reporting Requirements

Monthly progress & manpower/expenditure reports 15th of each month