DRAFT_PN_FSCS_PWS_Clean.doc

ATTACHMENT A

Performance Work Statement

Evaluation of Promise Neighborhoods and Full-Service Community Schools National Center for Education Evaluation

Institute of Education Sciences

U.S. Department of Education

I. INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (NCEE) of the Institute of Education Sciences (IES) at the U.S. Department of Education (Department) intends to award a 3-year contract with multiple option periods to conduct an implementation and impact evaluation of the Promise Neighborhoods and Full-Service Community Schools programs.

This Performance Work Statement includes a base contract with a period of performance of 36 months and three overlapping option periods, which would extend the contract to 50 months.

The base period includes: (1) an implementation and impact evaluation of Promise Neighborhoods, and (2) a feasibility study to identify promising strategies and design approaches to conduct an implementation and impact evaluation of Full-Service Community Schools.

If exercised, a series of option periods would: (1) enhance the rigor of the Promise Neighborhoods impact evaluation, and (2) provide two options for conducting an implementation and impact evaluation of Full-Service Community Schools. The specifics of the optional Full-Service Community Schools evaluation will be based on the feasibility and design work conducted during the contract’s base period, and thus the options in Option #2 and Option #3 are provided as illustrative examples.

Contract Periods of Performance

1 – 36 Evaluation of Promise Neighborhoods using a pre-post design and feasibility/design work for Full-Service Community Schools

6 – 36 Evaluation of Promise Neighborhoods using a comparison group (in conjunction with the evaluation in the Base Period)

10 – 39 Evaluation of Full-Service Community Schools if a difference-in-differences design is selected

10 – 50 Evaluation of Full-Service Community Schools if a randomized controlled trial design is selected

A. Legislative Authority The Promise Neighborhoods and Full-Service Community Schools programs are authorized by Title IV Part F Sections 4624-4625 of the Elementary and Secondary Education Act (ESEA), as amended by the Every Student Succeeds Act (ESSA). ESEA also mandates an evaluation of both programs. Specifically, Title IV Part F Section 4624(I) mandates that the Department conduct an evaluation of the implementation and impact of Promise Neighborhoods activities. Title IV Part F Section 4625(f) mandates that the Department examine the effectiveness of Full-Service Community Schools activities.

B. Background Promise Neighborhoods and Full-Service Community Schools are programs which provide support to grantees to implement comprehensive, coordinated services for children, their families, and community members. These programs are intended to target children attending high-poverty schools in distressed communities. The services provided may include academic, social, health and mental health, family and community engagement, crime prevention and rehabilitation, and workforce readiness services with the goal of improving children’s educational and developmental outcomes, both in the short-term and in the long-term as they transition to college and/or the workforce. The underlying theory is that building a coordinated continuum of educational and developmental supports for children from infancy to college and/or careers (“cradle to career”) combined with family and community supports will be more effective than individual, uncoordinated supports and will lead to improved academic and developmental outcomes.

Promise Neighborhoods and Full-Service Community Schools are both discretionary federal grant programs. The Promise Neighborhoods program competitively awards grants to eligible non-profit organizations, institutions of higher education, and Indian tribes, who typically work in partnership with other local organizations (including schools and school districts) to plan for and/or implement place-based initiatives that provide a comprehensive continuum of supports. The Full-Service Community Schools program competitively awards grants to implement similar place-based initiatives to a consortium of a school district or the Bureau of Indian Education and one or more community-based organizations, non-profit organizations, and other public and private entities.

Promise Neighborhoods and Full-Service Community Schools first awarded grants in 2010. Since that time, there have been 49 one-year planning grants and 22 five-year implementation grants awarded under the Promise Neighborhoods program. The Department has invested $506 million to support those grantees. There have been 47 five-year implementation grants (funds received during the first year may be used for planning, establishing partnerships, and capacity building) awarded under the Full-Service Community Schools program since 2010, with a Department investment of $55 million.

For more information about these two grant programs, see: https://innovation.ed.gov/what-we-do/parental-options/promise-neighborhoods-pn/ and https://innovation.ed.gov/what-we-do/parental-options/full-service-community-schools-program-fscs/ Since both programs’ inceptions, there has not been a national evaluation or a rigorous local evaluation of the impacts of either program.

The programs are modeled in part on the Harlem Children’s Zone, which combines a continuum of community services for children and their families along with charter schools in the Harlem neighborhood of New York City. There have been several rigorous evaluations of the charter school component of Harlem Children’s Zone, which have found positive effects on student achievement.

However, there has been no rigorous evaluation of the Harlem Children’s Zone as a whole.

While there has not been a rigorous impact evaluation of the Full-Service Community Schools or Promise Neighborhoods grant programs, there is some limited evidence that schools which offer comprehensive academic, health, and developmental supports in collaboration with community partners may be positively associated with student achievement, family engagement, behavior, and attendance.

C. Overview of the Full-Service Community Schools Evaluation During the Full-Service Community Schools design and feasibility study (base contract), the contractor shall assess the most feasible and policy relevant options for studying the program’s implementation and impacts. The Department will consider study designs that non-experimentally estimate the effects of the program or experimentally estimate the effects of one or more core features of the program. It is not feasible for grant awards to be randomly assigned.

Specifically, during the feasibility phase of the study the contractor shall address the following research questions:

· What are the most policy relevant impact questions about the Full-Service Community Schools program that can be credibly answered?

· What are the most credible and feasible study designs for each of the relevant impact questions?

· What is the most credible and feasible counterfactual for each of the relevant impact questions? How will an appropriate comparison group be identified? Given the diversity of allowable activities under the Full-Service Community Schools program, how will each of the study designs lead to a credible and useful estimate of the program’s impact? How will each study design leverage information about the implementation of the Full-Service Community Schools program or one of more core features of the program as well as relevant features of any applicable counterfactual condition?

· What are the most appropriate and relevant outcomes to collect for each of the relevant impact questions? How will these outcomes be measured, and how often will outcome data be collected?

· What design and sample size would be needed to detect the most policy relevant effect size?

· What are the most policy relevant subgroup(s) of interest, and how would the inclusion of such subgroup(s) affect the proposed study design, sample size, and analyses?

· What coordination with the program grantees would be helpful (or needed) to carry out each of the proposed study design options, and what are the key timelines to ensure coordination is feasible?

· What are the most credible and feasible approaches to address cost analysis and/or cost-effectiveness analysis?

Several challenges may affect the study design. The offeror shall consider these issues in the development of the proposal as well as in the design and conduct of the evaluation itself (if exercised).

· Data collection. Non-experimental study designs may use data from existing grantees and/or new grantees. The contractor shall propose and justify which grantees to include in such an evaluation, addressing the tradeoffs of collecting data from each group in terms of power, feasibility/ease of collection, ability to measure longer-term outcomes, and any other relevant issues. Note that the implementation and impact samples do not necessarily need to be comprised of the same set of grantees.

· Baseline equivalence and service contrast. For designs that use comparison schools, the comparison and treatment service environments should be similar at baseline and demonstrate enough contrast post-baseline. The contractor shall propose which data and analyses to use to ensure comparability between comparison and treatment schools. The contractor shall discuss the feasibility of collecting these data systematically from both comparison and treatment schools.

· Outcomes. The theory of action relies on grantees implementing a few comprehensive, coordinated services for children, their families, and community members in order to lead to improved academic and development outcomes. Therefore, grantees address several outcomes in both the short-term and the long-term. The contractor shall address which outcomes are feasible and policy relevant to measure in the timeframe of the evaluation.

· Focus and policy relevance of experimental study. Conducting an experimental study of Full-Service Community Schools could involve identifying a sufficient number of grantees with a common, oversubscribed core element(s) of the program. It could also involve implementing a common, core element of the program in non-grantee schools. The contractor shall address how to identify such elements, and the policy relevance of such studies.

· Statistical power. The contractor shall determine what the policy relevant impact sizes are for student achievement and other outcomes, and the appropriate sample size to achieve that target.

Based on the feasibility and design work conducted by the contractor as part of the base period of this contract, the Department may elect to conduct an evaluation of Full-Service Community Schools. The evaluation, if exercised, shall assess the implementation and impacts of Full-Service Community Schools based on the design options identified during the base period. Options #2 and #3 are illustrative examples of a full evaluation.

Option #2 contains a difference-in-differences design that measures changes in outcomes for children in grantee schools before and after grant award compared to changes in outcomes for children in similar comparison schools, using data from existing grantees.

Option #3 contains a randomized controlled trial of schools implementing one or more core, policy-relevant features of the Full-Service Community Schools program. Under this option, the contractor shall be responsible for selecting the intervention model to be evaluated. After the contract award, the contractor shall establish a group of three experts (approved by the Department) in community schools and the Full-Service Community Schools program. In consultation with the expert panel and the Department, the contractor shall develop an open solicitation and selection criteria. Then, the contractor shall publish the solicitation and contact providers to ascertain their interest and ability to participate in a randomized trial of their program. In response to the solicitation, interested providers shall submit a description of their program and the requirements for implementation, along with any evidence of effectiveness. The contractor shall oversee the expert panelists’ review of proposals in order to identify up to 4 providers for inclusion in follow-up presentations of their program to the panelists in Washington DC. After consideration of the Contracting Officer’s Representative (COR) and expert panel’s recommendations, the contractor shall select up to two providers for inclusion in the evaluation.

During the evaluation, the contractor shall address the following research questions:

· What services are offered and by whom? How are services connected and coordinated?

· How many students and their families receive each service?

· What are the implementation challenges, and how are they addressed?

· How do the services compare to those offered prior to the Full-Service Community Schools grant and/or how do the services compare to those offered by comparison schools?

· How much funding to implement Full-Service Community Schools is received from sources other than the federal grant? What is the total cost per year per participant?

· What are the impacts of implementing Full-Service Community Schools, or a core feature of Full-Service Community Schools, on student outcomes such as student achievement? How do the impacts vary by student or other characteristics?

D. Overview of Promise Neighborhoods Evaluation The Promise Neighborhoods evaluation does not contain a design and feasibility study because one has already been conducted (see Attachment X for the design report). The evaluation (base period) will assess the implementation and impacts of Promise Neighborhoods. The contractor shall address the following research questions:

· What services are offered and by whom? How are services connected and coordinated?

· How many students and their families receive each service?

· What are the implementation challenges, and how are they addressed?

· How do the services compare to those offered prior to the Promise Neighborhoods grant and/or how do the services compare to those offered by comparison neighborhoods?

· How much funding to implement Promise Neighborhoods is received from sources other than the federal grant? What is the total cost per year per participant?

· What are the impacts of implementing Promise Neighborhoods on student outcomes such as student achievement? How do the impacts vary by student or other characteristics?

If Option #1 is not exercised, the Promise Neighborhoods evaluation will estimate the impact of Promise Neighborhoods by comparing differences in outcomes in grantee neighborhoods before and after grant award. If Option #1 is exercised, the contractor shall also include a comparison group of neighborhoods in the evaluation design, comparing differences in outcomes before and after grant awards between grantee and comparison neighborhoods. The purpose of including comparison neighborhoods in the evaluation is to improve the chances that the impact estimates reflect the impact of Promise Neighborhoods and not other confounding factors. It essentially transforms the evaluation from a pre-post design to a difference-in-differences design.

Several challenges may affect the study design. The offeror shall consider these issues in the development of the proposal as well as in the design and conduct of the evaluation itself.

· Data collection. The study may use data from existing grantees and/or new grantees. The contractor shall propose and justify which grantees to include in the evaluation, addressing the tradeoffs of collecting data from each group in terms of power, feasibility/ease of collection, ability to measure longer-term outcomes, and any other relevant issues. Note that the implementation and impact samples do not necessarily need to be comprised of the same set of grantees.

· Baseline equivalence and service contrast. If the study design includes a comparison group (i.e., if Option #1 is exercised), the comparison and treatment service environments should be similar at baseline and demonstrate enough contrast post-baseline. The contractor shall propose which data and analyses to use to ensure comparability between comparison and treatment neighborhoods. The contractor shall discuss the feasibility of collecting these data systematically from both comparison and treatment neighborhoods.

· Outcomes. The theory of action relies on grantees implementing several comprehensive, coordinated services for children, their families, and community members in order to lead to improved academic and development outcomes. Therefore, grantees address several outcomes in both the short-term and the long-term. The contractor shall address which outcomes are feasible and policy relevant to measure in the timeframe of the evaluation.

· Statistical power. The contractor shall determine what the policy relevant impact sizes are for student achievement and other outcomes, and the appropriate sample size to achieve that target.

II. SCOPE OF WORK

The Contracting Officer’s Representative (COR) reserves the right to inspect the contractor’s work while in process. The contractor shall submit to the COR any progress report or other written documentation of work to date that the COR requests to facilitate that inspection, according to contract clause 52.246-5 (cost-reimbursement).

This section specifies the tasks and subtasks that the contractor shall perform. All deliverables under these tasks shall be submitted to ED’s Contracting Officer’s Representative (COR), ED’s Contracting Specialist (CS), and ED’s Contracting Officer (CO) electronically unless otherwise noted. Unless indicated otherwise, the COR will notify the contractor within two weeks of receipt of any changes required in the deliverable, and the contractor shall have up to two weeks to make the necessary revisions and submit a final deliverable to ED unless otherwise noted.

A. Base Period The base period contains two distinct phases (the Full-Service Community Schools design and feasibility phase and the Promise Neighborhoods evaluation phase), to be conducted simultaneously.

Task 1—Meeting with ED

1.1 Opening Meeting. The contractor’s project director and up to three key project staff shall meet with the Department’s Contracting Officer (CO), Contracting Officer’s Representative (COR), and other appropriate Department staff within 10 work days after the effective date of contract award to discuss details regarding the tasks outlined in the proposal, study design, scheduling activities, and other issues related to the contract. The Opening Meeting shall address issues related to the Full-Service Community Schools design and feasibility phase and the Promise Neighborhoods evaluation phase. This meeting shall be at the Department’s offices in Washington, DC. The contractor shall come prepared to identify any areas of concern not contained in the accepted technical proposal or discussed during contract negotiations and to suggest ways of responding to these concerns. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting and how each will be addressed in the study. The COR will review the memo within one week. If any changes are needed, the contractor shall make them within two business days.

Deliverable:

Opening Meeting summary memo Full-Service Community Schools Design and Feasibility Phase Task 2—Identify and Explore Full-Service Community Schools Study Design Options

2.1 Develop and Refine Full-Service Community Schools Feasibility Discussion Protocols and Plan: The contractor shall develop and submit protocols and a plan to address each of the key research questions (Section C) no later than five weeks after contract award. The protocols shall include a section to guide discussions with current Full-Service Community Schools grantees (including school districts, schools, and/or Bureau of Indian Education officials and relevant partners). The contractor shall develop separate protocols to guide discussions with each type of grantee (i.e., school districts, schools, Bureau of Indian Education officials, and relevant partners) with overlapping questions where appropriate and relevant. The protocols shall also be relevant or adaptable for discussions with place-based initiative providers, and districts and/or schools that may not be currently receiving Full-Service Community Schools grants. The general purpose of these protocols shall be to elicit information that will inform study design options, as well as the feasibility of recruitment, data collection, and timelines for an impact evaluation of Full-Service Community Schools. The plan that accompanies the protocols shall explain the contractor’s approach to determining how many and which grantees and non-grantees to have discussions with. The plan shall also explain how each of the proposed items in the protocols will inform the key research questions related to feasibility and design options. Finally, the plan shall address which publicly available documents the contractor shall review in order to inform the protocols. The COR will review and provide feedback on the draft feasibility plan and protocols within two weeks of submission. Based on COR feedback, the contractor shall refine and submit a final version of the feasibility plan and protocols no later than eight weeks after contract award.

2.2 Implement Full-Service Community Schools Feasibility Plan and Protocols: After the final feasibility plan and protocols are approved by the COR, the contractor shall implement the plan and protocols. As part of implementation, the contractor shall conduct interviews using the protocols developed in Sub-task 2.2 with up to nine of each type of grantee (i.e., up to nine school districts, up to nine schools, up to nine Bureau of Indian Education officials, and up to nine of their relevant partners). The feasibility plan and protocols shall be fully implemented no later than 20 weeks after contract award to ensure enough time to prepare the draft study design options report (see Task 3).

Deliverables:

Draft Full-Service Community Schools feasibility plan and protocols

Final Full-Service Community Schools feasibility plan and protocols Task 3—Full-Service Community Schools Study Design Options Report The contractor shall prepare a Full-Service Community Schools study design options report that addresses the research questions and study challenges outlined in Section C of the PWS, drawing in part on the results of implementing the feasibility plan and protocols (Sub-task 2.2). The report shall include the pros and cons of each study design option, both in terms of its methodological quality and its practical ease to implement in the field. The report shall also discuss instrumentation, data collection, and analysis issues, particularly as they relate to implementing each of the proposed study design options. The contractor shall provide a draft of the study design options report to the COR 12 weeks prior to the Technical Work Group (TWG) meeting (Task 4). The COR will review and provide feedback on this draft within two weeks of receipt. The study design options report shall be used as materials for informing the TWG of the study design options prior to the TWG meeting. The Full-Service Community Schools study design options report shall also be used to prepare presentation materials for the TWG meeting (Task 4).

The contractor shall revise the Full-Service Community Schools study design report as appropriate based on the COR’s feedback on the draft report and subsequent to the discussion with the TWG (Task 4). The contractor shall submit this final study design options report no later than 40 weeks after contract award. If TWG members recommend any significant deviations from the study design, data collection, or data analysis options indicated in the study design options report, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study design options report based solely upon the recommendation of TWG members without the written approval of the COR. On an as needed basis, the contractor may have individual TWG members review products to assist with specific issues on which they have expertise. In these cases, there is no need to convene all TWG members in a group meeting to discuss a limited, specific set of issues.

Deliverables:

Draft Full-Service Community Schools study design options report

Final Full-Service Community Schools study design options report

Task 4—Full-Service Community Schools Technical Working Group

4.1 Establish Full-Service Community Schools Technical Working Group: The role of the TWG will be to advise the contractor and the Department on the conduct of a potential evaluation of Full-Service Community Schools, including but not limited to: study design, instrumentation, data collection, and analysis. The TWG shall be comprised of up to 10 individuals and must include national experts in: place-based initiatives (particularly those most related to the Full-Service Community Schools program), specific content areas that are often components of place-based initiatives, evaluation methodology (both experimental and quasi-experimental), as well as conducting high-quality impact studies in the context of federal grant programs. The contractor shall submit a list of proposed TWG members no later than four weeks after contract award. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. The list shall also clearly explain the background and expertise of each proposed TWG member, and how it is specifically relevant to the needs of the study. The COR will review the proposed TWG list within one week of receipt. The contractor shall address any feedback and make any changes that the COR requires no later than three business days after receiving feedback. After COR approval of the proposed TWG members, the contractor shall contact each approved member to secure interest and intention to serve on the TWG.

4.2 Convene Full-Service Community Schools Technical Working Group: The contractor shall convene a meeting of the Full-Service Community Schools TWG no later than 36 weeks after contract award. The TWG meeting shall focus on the proposed key research questions, the study design options, identification and recruitment feasibility of a relevant counterfactual comparison group, data collection, and analysis issues. As stated above, The Full-Service Community Schools study design options report (Task 3) shall be used to prepare presentation materials for the TWG meeting. These presentation materials and an agenda shall be submitted to the COR four weeks prior to the TWG meeting. The COR will review and provide feedback on these materials within one week of receipt. The contractor shall make any changes that the COR requires no later than two weeks prior to the TWG meeting.

The meeting shall be one day in length and shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Honorarium amounts shall be based on the number of hours that TWG members are expected to provide, and the hourly rate of the honorarium should not exceed the current General Schedule (GS)-15, Step 10 hourly rate without justification. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

The contractor shall submit a written summary of the meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The COR will review the written summary within one week of receipt. The contractor shall address any feedback and make any changes that the COR requires no later than one week after receiving feedback. Proposed actions shall not go forward unless approved by the COR.

Deliverables:

List of proposed Full-Service Community Schools TWG members

TWG presentation materials (based on Full-Service Community Schools study design options report) and agenda

Summary of Full-Service Community Schools TWG meeting

Promise Neighborhoods Evaluation Phase

Task 5—Promise Neighborhoods Technical Working Group

5.1 Establish Promise Neighborhoods Technical Working Group (TWG). The role of the Promise Neighborhoods TWG will be to advise the contractor and the Department on the conduct of the study including, but not limited to evaluation design, instrumentation, data collection and analysis, and presentation of evaluation findings. The TWG shall be comprised of up to 10 individuals and must include national experts in: place-based initiatives (particularly those most relevant to the Promise Neighborhoods program), specific content areas that are components of place-based initiatives and relevant to the Promise Neighborhoods program, evaluation methodology, as well as conducting high-quality impact studies in the context of federal grant programs. The Promise Neighborhoods evaluation TWG members may overlap with the Full-Service Community Schools TWG members (Task 4) if appropriate.

The contractor shall submit a list of a subset of proposed TWG members (up to 4 individuals) no later than four weeks after contract award. These TWG members will participate in the first TWG meeting and will advise the contractor and the Department on the study design, and, feasibility issues surrounding the use of a comparison group. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. The list shall also clearly explain the background and expertise of each proposed TWG member, and how it is specifically relevant to the needs of the study. The COR will review the proposed TWG list within one week of receipt. The contractor shall address any feedback and make any changes that the COR requires no later than three business days after receiving feedback. After COR approval of the proposed TWG members, the contractor shall contact each approved member to secure interest and intention to serve on the TWG. The contractor shall submit a list of the full set of proposed TWG members no later than 22 months after contract award.

5.2 Convene Promise Neighborhoods Technical Working Group. The contractor shall convene one meeting of the Promise Neighborhoods subset of TWG members and one meeting of the Promise Neighborhoods full set of TWG members over the period of the contract. The first meeting with the subset of TWG members shall be held no later than 22 weeks after the contract award and shall focus on the proposed design for the evaluation, and, feasibility issues surrounding the use of a comparison group. The second meeting with the full set of TWG members shall be held after preliminary data analyses have been conducted. Presentation materials and an agenda shall be submitted to the COR four weeks prior to each TWG meeting. The COR will review and provide feedback on these materials within one week of receipt. The contractor shall make any changes that the COR requires no later than two weeks prior to the TWG meeting.

The first meeting with the subset of TWG members shall be held virtually rather than in-person and shall be no more than a half-day in length. The second meeting with the full set of TWG members shall be one day in length and shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Honorarium amounts shall be based on the number of hours that TWG members are expected to provide, and the hourly rate of the honorarium should not exceed the current General Schedule (GS)-15, Step 10 hourly rate without justification. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

The contractor shall submit a written summary of each meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The COR will review the written summary within one week of receipt. The contractor shall address any feedback and make any changes that the COR requires no later than one week after receiving feedback. Proposed actions shall not go forward unless approved by the COR.

If TWG members recommend any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes based solely upon the recommendation of a TWG member without the written approval of the COR. On an as needed basis, the contractor may have individual TWG members review products to assist with specific issues on which they have expertise. In these cases, there is no need to convene all TWG members in a group meeting to discuss a limited, specific set of issues.

Deliverables:

List of proposed Promise Neighborhoods subset of TWG members

List of proposed Promise Neighborhoods full set of TWG members

TWG presentation materials and agenda

Summary of each Promise Neighborhoods TWG meeting

Task 6—Promise Neighborhoods Study Design Report The contractor shall prepare a Promise Neighborhoods study design report that addresses the research questions and study challenges outlined in Section D of the PWS, revising and building off the existing Promise Neighborhoods design report (see Attachment X) and the contractor’s proposal, as appropriate. The report shall address the study design, instrumentation, data collection, and analysis.

The study design report shall also address the feasibility of including a Promise Neighborhoods comparison group in the data collection (Task 8) and data analyses (Sub-task 9.2). The draft study design report shall include the methods and outcomes of constructing Promise Neighborhoods comparison groups comprised of non-grantees based on: (1) how similar potential comparison neighborhoods are in terms of baseline school and family population characteristics available from extant data, such as the U.S. Census and EDFacts data, and (2) how similar potential comparison neighborhoods are in terms of baseline service environments using information such as web searches and information from the Office of Elementary and Secondary Education on the scores of grantees and non-funded grant applicants. The contractor shall develop a list of recommended comparison neighborhoods based on these criteria and justification for why the comparison neighborhoods are suitably similar to grantees. The draft study design report shall include this list and justification.

The contractor shall consult up to nine grantees and, if appropriate and relevant, up to nine state education agency officials to assess the similarity between the grantees and the recommended comparison neighborhoods. The contractor shall develop and submit a plan to address how many and which grantees and/or state education agency officials they will consult to assess the similarity between the grantees and the recommended comparison neighborhoods. The plan shall also address which publicly available documents the contractor shall review in order to inform the protocols. The contractor shall also develop two protocols: one to guide discussions with those grantees, and one to guide discussions with those state education agency officials. Questions on the two protocols may overlap where appropriate and relevant. The contractor shall submit the draft feasibility plan and protocols no later than 2 weeks after contract award. The COR will review and provide feedback on the draft feasibility plan and protocols within two weeks of submission. Based on COR feedback, the contractor shall refine and submit a final version of the feasibility plan and protocols no later than 6 weeks after contract award. A summary of the consultations with the grantees and/or state education agency officials shall be included in the final Promise Neighborhoods study design report.

The contractor shall submit the draft Promise Neighborhoods study design report to the COR no later than 10 weeks after contract award The COR will review the draft memo within two weeks. The contractor shall make revisions and submit a final Promise Neighborhoods study design report no later than 16 weeks after contract award.

Draft Promise Neighborhoods feasibility plan and protocols

Final Promise Neighborhoods feasibility plan and protocols Draft Promise Neighborhoods study design report

Final Promise Neighborhoods study design report Task 7—Develop Promise Neighborhoods Data Collection Forms and Prepare OMB Package

7.1 Development of Promise Neighborhoods Data Collection Forms. The contractor shall collect data through surveys and extant data. The contractor shall design data collection forms to address the key research questions (Section D).

Data collection forms shall consist of:

· Grantee surveys: Surveys of Promise Neighborhoods grantees will provide information on the services offered prior to the grant and during the grant, the degree to which services are coordinated to form a pipeline, implementation challenges and how they were addressed, and relevant funding issues. The data collected from grantees shall provide information on implementation; the sample may not necessarily align with the grantees included in an impact evaluation.

· School district surveys: Surveys of school districts that partner with Promise Neighborhoods grantees will provide information on the services offered within schools (e.g., what types of services are offered, the degree to which they are coordinated, implementation challenges and how they were addressed, any relevant funding issues) prior to the grant and during the grant. If Option #1 is exercised, comparison group school district surveys will provide information on the service environment counterfactual prior to the Promise Neighborhoods grant and during the grant.

· Extant data forms: These forms will be used to request and extract relevant information, including: (1) web searches to provide information on the service environment prior to and during the grant, (2) state birth records to identify children living in the neighborhoods over time, and (3) student-level state standardized achievement test scores and any other relevant data to estimate the impact of Promise Neighborhoods on educational outcomes. See further discussion of extant data under Sub-task 8.2.

The contractor shall pilot test each of the instruments with no more than nine respondents per instrument to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, that the respondents interpret and respond to the questions as intended, and that the contractor’s procedures and testing systems for administering the instruments are effective. Piloting is to be conducted prior to the final 60 days in the OMB submission process. The contractor shall ensure that instruments are revised as needed based on pilot test results.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full-scale data collection with the IRB approval obtained by districts from their own institutions. Specifically, the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

7.2 Prepare Promise Neighborhoods OMB Package Materials. The contractor shall prepare all relevant forms and justification for a clearance package that includes all data collection instruments. The OMB package shall include a description of the study design and data collection instruments, and include the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions. In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, and (7) limitations of the study and its potential use for policy decisions.

The contractor shall submit an OMB clearance package to the COR no later than 26 weeks after contract award. The COR will review the forms within two weeks. The contractor shall submit the revised OMB clearance package no later than 30 weeks after contract award. The contractor shall submit the OMB package mindful that clearance requires a minimum of 120 days and how this impacts the agreed schedule of deliverables. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process.

Promise Neighborhoods OMB clearance package, including data collection forms

Revised Promise Neighborhoods OMB clearance package, including data

Task 8—Promise Neighborhoods Data Collection

8.1 Administer surveys. Using instruments approved by OMB (Task 7), the contractor shall administer surveys to Promise Neighborhoods grantees and school districts, and if Option #1 is exercised, to comparison neighborhood school districts. The surveys should address the research questions outlined in Section D. The contractor shall administer the surveys once, during Fall 2020.

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (https://nces.ed.gov/statprog/2012/). To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated.

During the survey fielding period, the contractor shall provide weekly progress updates to the COR.

8.2. Collect extant data. The contractor shall also collect extant data, including:

· Web searches. To provide information on the service environment in Promise Neighborhoods prior to and during the grant, the contractor shall conduct web searches. If Option #1 is exercised, the contractor shall also conduct web searches in order to provide information on the service environment in comparison neighborhoods.

· State or district administrative records on students. To estimate the impact of Promise Neighborhoods on educational outcomes, the contractor shall collect student-level administrative data from states or districts for students in the Promise Neighborhoods, and if Option #1 is exercised, in the comparison neighborhoods. These data shall include state standardized achievement test scores and any other relevant student characteristics and outcomes.

· State birth records. The contractor shall also use vital statistics records, typically housed in the state’s department of health, to identify births that occurred in each Promise Neighborhood, and if Option #1 is exercised, in each comparison neighborhood. These data would allow the contractor to identify children living in the neighborhoods over time.

The contractor, as appropriate, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request.

Deliverables:

Survey data collection updates

Task 9—Promise Neighborhoods Data Analysis and Report Preparation

9.1 Promise Neighborhoods Data Analysis Plan. The contractor shall develop a data analysis plan that provides a policy-motivated introduction to the study, describes the study design, identifies study sources, and describes the intended analyses that will be included in the Promise Neighborhoods report (Task 9.2). The data analysis plan shall include table shells, as appropriate, to indicate how the contractor intends to present the data findings in the report.

The contractor shall submit a draft analysis plan to the COR no later than 22 months after contract award. The COR will provide feedback within two weeks. The contractor shall submit a final data analysis plan no later than four weeks following the COR’s comments.

9.2 Promise Neighborhoods Report. The contractor shall conduct data analyses and prepare a report that provides: implementation information on the Promise Neighborhoods grantees included in the study, appropriate service contrast estimates (i.e., comparing services prior to grant award and after grant award, and if Option #1 is exercised, comparing services between the Promise Neighborhoods and the comparison neighborhoods), and impact estimates on relevant outcomes. The contractor shall borrow the introduction, study design description, and data sources from the final analysis plan, rather than developing these components anew, to the extent that it meets the COR’s approval.

The contractor shall make sure that the report is written and organized so that it is accessible to policymakers and practitioners rather than academic researchers. The report length is expected to be no more than 15 pages, with information on the technical aspects of the study included in appendices. The contractor shall also include a one-page stand-alone introduction to the report and contain enough information to form the basis for a press release.

The contractor shall submit a draft Promise Neighborhoods report for COR review no later than 29 months after contract award. The COR will review the draft report within three weeks. The contractor shall make revisions and submit a revised Promise Neighborhoods report, ready for peer review, no later than six weeks following the COR’s comments. The COR will submit the revised report to the Standards and Review Office (SRO) for review. The COR will provide comments from SRO to the contractor within approximately two months. The contractor shall be responsive to SRO’s comments and revise the report until it clears SRO. The contractor shall provide a web-ready (i.e. 508-compliant) version of the report to the COR (the final Promise Neighborhoods report) within two weeks after the report is cleared by SRO.

Draft Promise Neighborhoods analysis plan

Final Promise Neighborhoods analysis plan

Draft Promise Neighborhoods report

Revised Promise Neighborhoods report

Final Promise Neighborhoods report

Task 10—Promise Neighborhoods Restricted-Use Data File The contractor shall prepare one set of restricted-use data files that contain all data collected for the Promise Neighborhoods report, with all direct personal identifiers removed. The contractor shall provide documentation for the files including variable labels, category labels, the unit of observation, weights, and how missing data were addressed.

The contractor shall also submit a risk disclosure analysis for submission to the Disclosure Review Board (DRB). The contractor shall submit the risk disclosure analysis to the COR no later than 36 months after contract award. Within two weeks of DRB approval of the codebooks and risk disclosure analysis, the contractor shall submit to the COR the restricted-use data files consistent with NCES standards.

ASCII file format is acceptable.

Promise Neighborhoods risk disclosure analysis

Promise Neighborhoods restricted-use data files

Task 11—Base Period Reporting Requirements The contractor shall submit one copy of the following performance reports each month through the Invoice Processing Platform. The reports should include relevant information for activities related to both the Full-Service Community Schools design and feasibility phase as well as the Promise Neighborhoods evaluation phase. The COR will review these reports within one week of receipt. Any feedback by the COR shall be addressed by the contractor within two business days.

Monthly Progress Report/Exception Reports: The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task or sub-task regarding significant findings and events, problems encountered, and staff use.