21st_CCLC_-_RFI_12.3.18.pdf

December 3, 2018

AN IMPACT EVALUATION TO INFORM THE

21ST CENTURY COMMUNITY LEARNING

CENTERS PROGRAM

Solicitation No.: 91990018R0003

Request for Information & Vendor Engagement

Facilitated by the Office of the Chief Financial Officer, Contracts and

Acquisitions Management Office

Contents Introduction

Capability Statement and Feedback Submission Instructions

Contact Information

Request for Capability Statements

Vendor Questions

List of Attachments

Draft Performance Work Statement

Introduction

The U.S. Department of Education is conducting market research to determine if sufficient interest and capability exists in the marketplace to support an Impact Evaluation to Inform the 21st Century Community Learning Centers (21st CCLC) Program. This document serves as an invitation for potential offerors to express interest and demonstrate skills, knowledge, and capabilities relevant to this requirement. Additionally, this is an opportunity for potential offerors to provide feedback on the requirement.

Additional Notes: The Department will not pay for the provision of any information nor will it compensate any respondents for providing this information. This notice in no way obligates the Department to award any contract to any interested party. Feedback will not be provided to any interested party who submits information to the Department in response to this notice.

Please note that all solicitation-related documents in this RFI are draft versions. The content of each document may change substantially between release of this RFI and release of the final solicitation. The final solicitation will be governed by the terms, conditions, instructions, and criteria in that document. Offerors should not rely on the contents of this RFI for preparing final proposals. To the extent that information provided in this document is inconsistent with or contradicts information contained in the final solicitation, the solicitation controls.

Capability Statement and Feedback Submission Instructions

Detailed information for the required submission is provided below. Offerors should submit, via email, the following documents to Jonathan.Bettis@ed.gov and Kevin.Wade@ed.gov with a cc to Ryan.Battad@ed.gov no later than December 13, 2018 at 2:00PM EST.

I. One PDF document that contains your Capability Statement (PDF).

II. One PDF document that contains your response to vendor questions.

III. One WORD document that contains any comments, track change revisions, or edits to the PWS (optional/as applicable).

Offerors should submit all three documents in one email. Please note that the Department imposes a 15MB file size limitation. This limitation accounts for potential file size increases associated with Microsoft Cloud services and may be different from earlier Department file size restrictions.

Participation in this vendor outreach effort is entirely voluntary. A response to this RFI is not required in order to respond to any solicitation that the Department issues.

mailto:Jonathan.Bettis@ed.gov mailto:Kevin.Wade@ed.gov mailto:Heather.Watroba@ed.gov

Contact Information Please note that only the individuals listed below are authorized points of contact for this competition. No members of the IES staff will respond to inquiries.

The following members comprise the acquisition team for this procurement:

Primary Point of Contact Jonathan E. Bettis Jonathan.Bettis@ed.gov

(202) 245-6971 Contract Specialist/Contracting Officer

Kevin Wade Kevin.Wade@ed.gov

(202) 245-8067 Contract Specialist

Secondary Point of Contact Ryan Battad Ryan.Battad@ed.gov

(202) 245-6527 Contracting Officer/Team Lead

Dissemination of Procurement Information

Offerors are strongly encouraged to communicate via email. All questions that the Department receives regarding the procurement will be disclosed to the public via www.FBO.gov. All announcements regarding the procurement will be posted to FBO.

mailto:Jonathan.Bettis@ed.gov mailto:Kevin.Wade@ed.gov mailto:Ryan.Battad@ed.gov http://www.fbo.gov/

Request for Capability Statements

Proposed NAICS Information: 541720 – Research and Development in the Social Sciences and Humanities1 Proposed Product Service Code: R410 – Support – Professional: Program

Evaluation/Review/Development

The Department requests capability statements from small businesses in order to evaluate what, if any, set-aside determinations are appropriate for each region.

The Department will not review generic capability statements because generic capability statements tell us nothing about your ability to execute the subject contract. Rather, we will only review capability statements that conform to the specifications provided below.

Format and Instructions for Capability Statements

Capability statements are limited to 7 double-spaced pages (not including the cover page) and must be submitted in PDF format. The Department will not consider material in excess of 7 pages.

1. Cover Page containing the following information:

Name of Company Primary Point of Contact (phone and email) DUNS Number Small Business Status for the applicable NAICS code (if not a small business, please indicate “OTSB” in this field)

2. Substantive Content

The capability statement must describe the collective expertise and experience of your company’s personnel, in each of the following areas:

• Experience and skill in managing, designing and conducting large-scale random assignment studies of interventions designed to improve educational outcomes. Interested parties shall provide a minimum of two examples of personnel’s successful past experience and skill in leading, managing, designing, and conducting all aspects of evaluations (preferably in education settings) that are similar to this procurement in content (e.g., professional development, coaching/mentoring), methodology (e.g., randomized controlled trial), complexity

1 Please note that we welcome feedback on other appropriate NAICS and PSC codes.

(e.g., recruitment effort, intervention features, implementation support), scope (e.g., national), sample size, and data collection modes (e.g., district administrative records, principal, teacher, and grantee surveys).

• Experience and skill in authoring well written, policy relevant publications with clear take-home messages for wide audiences. Interested parties shall provide a minimum of two examples of personnel’s involvement with authoring publications that clearly summarize experimental findings and present them in a manner such that they can be understood and used by a wide audience.

• Substantive knowledge relevant to the Draft 21st CCLC evaluation, including after school programs, the 21st CCLC program, and promising interventions within the context of afterschool programs of homework or tutoring support, support for STEM activities, and support for youth development. Interested parties shall provide a minimum of two examples showing that personnel have relevant knowledge to effectively study and support implementation of potential interventions in these areas, particularly capitalizing on what is known from the research literature.

In addition, the capability statement shall address the following experience and capacity at the corporate level:

• Experience with projects similar in nature and scale to that described in the Draft 21st CCLC evaluation PWS. Interested parties must provide evidence of successful corporate experience with a minimum of two past projects that are similar to that described in the draft PWS, in terms of the specific tasks to be performed, scale, and scope required.

Vendor Questions

Note: all potential offerors, small businesses or otherwise, are encouraged to provide responses to these questions.

(1) What, if any, additional information should the Department provide in order to enable you to accurately estimate the cost of each task/subtask? This includes documents, research materials, more accurate task descriptions, policy information, or other content produced by or utilized by the Department.

(2) Are the NAICS and PSC codes appropriate for this procurement? If not, please justify an alternative.

(3) Can you identify any barriers to competition? If so, please identify and describe how the Department could reduce or eliminate those barriers.

(4) Please explain whether GSA’s OASIS contract vehicle is a suitable alternative to a full and open competition and please identify any other Tier III Category Management contract vehicles in addition to OASIS that may be suitable for this procurement.

(5) Please explain whether this procurement should be set-aside for small businesses.

Alternatively, please explain why this procurement should not be set-aside.

(6) Is there any other information that you wish to convey to the Department?

List of Attachments

• Performance Work Statement

Draft Performance Work Statement

Note: PWS Version Date 11.19.18

U.S. Department of Education Performance Work Statement Institute of Education Sciences

National Center for Education Evaluation

Performance Work Statement

An Impact Evaluation to Inform the 21st Century Community Learning Centers (CCLC) Program

INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (“NCEE”) of the U.S.

Department of Education’s (the “Department”) Institute of Education Sciences (“IES”) requires a contract to conduct an implementation and impact study.

This contract includes a 36-month Base Period and two option periods that would result in a 60-month period of performance in total. The Base Period includes conducting an implementation study of the 21st Century Community Learning Centers (21st CCLC) program, identifying a policy-relevant strategy for an impact study to be conducted within the 21st CCLC program, and initial conduct of the impact study.

Option Period I includes continued intervention implementation and data collection activities for the impact study. Option Period I includes a 22-month period of performance and overlaps with the Base Period by 10 months.

Option Period II includes a technical working group meeting and writing an impact report to be released after obtaining approval from the Standards and Review office of IES. Option Period II includes a 26 month period of performance and overlaps with the Option Period I by 14 months.

Base Option Period I Option Period II

July 2019 - June 2022 Sept. 2021 - June 2023 May 2022 - June 2024

36 months 22 months 26 months Overlaps with base by 10 months Overlaps with Period I by 14 months

Months 1-36 Months 27-48 Months 35-60

A. Legislative Authority

The 21st CCLC program is authorized in the Elementary and Secondary Education Act of 1965 as amended through P.L. 115-64. The 21st CCLC program provides funding for communities to establish or expand activities in community learning centers or expanded learning program activities. Community learning centers provide opportunities for academic enrichment and offer students a broad array of additional services, programs, and activities. Programs and activities range from academic support and enrichment to youth development, career development, and health and wellness activities. The 21st CCLC program was reauthorized in ESSA and was funded at $1.2 billion in 2018.

B. The 21st CCLC program

The 21st CCLC funded afterschool programs tend to offer multiple activities. Based on the 2015- 16 Annual Performance Report (APR) Data Collection System, homework assistance, physical activity, literacy, and Science, Technology, Engineering and Mathematics (STEM) were the activities of most frequent focus. In addition, close to half of the participating students are in grades K-5. It’s also important to note that participation is typically voluntary and states report close to half of the student participants attend the school-year program less than 30 days.

Previous evaluations of afterschool programs include two impact studies by IES, including an impact evaluation of structured academic instruction in reading or math implemented within the afterschool program setting completed in 2009 -https://ies.ed.gov/ncee/projects/evaluation/other_afterschool.asp; and an impact evaluation of the federal discretionary 21st CCLC grant program completed in 2005 -https://ies.ed.gov/ncee/projects/evaluation/other_21clc.asp). A descriptive study of the state-administered grant program in 2005-06 was conducted by the Department’s Policy and Program Studies Service and completed in 2011- https://www2.ed.gov/rschstat/eval/after-school/21st-century-community-learning-centers.pdf.

As part of a recently awarded contract by the 21st CCLC program office, a contractor will conduct a literature review of leading research in the afterschool area to provide information about outcomes that afterschool programs have affected. This effort will also provide information to support revisions of the 21st CCLC’s Government Performance Results Act measures. The Contracting Officer’s Representative (COR) will provide a copy of this work to the contractor awarded this contract a week prior to the kickoff meeting.

B. Key Research Questions

The purpose of the Department’s evaluation is two-fold. One purpose is to provide implementation information on currently funded 21st CLCC grantees. Of particular interest are the characteristics of the students served and the regularity of their participation, programmatic emphasis and variety of activities provided, staff characteristics and the training they receive, and outcomes focused on by the program.

A second goal is to estimate the impact and cost effectiveness of a particularly innovative strategy/practice useful for future 21st CCLC grantees that is implemented, in whole or part, by some current grantees. For the focus of the impact study, there are three potential strategies or practices of interest: 1) training and tools for afterschool staff that support positive youth https://ies.ed.gov/ncee/projects/evaluation/other_afterschool.asp https://ies.ed.gov/ncee/projects/evaluation/other_21clc.asp https://www2.ed.gov/rschstat/eval/after-school/21st-century-community-learning-centers.pdf https://www2.ed.gov/rschstat/eval/after-school/21st-century-community-learning-centers.pdf development outcomes, 2) resources and training for afterschool staff in supporting STEM activities and student math achievement, and 3) promising practices in using tutoring or homework help for students, particularly those who are low-performing, to meet State academic standards.

To address these two goals, the evaluation will include two components. The first component will involve conducting an implementation study. The second component will involve identifying a feasible, policy relevant strategy or practice for the impact study, refining the study design, and carrying out the impact study. The evaluation shall address the following research questions.

Implementation study component:

1. What are the implementation experiences of the 21st CCLC grantees? Which students participate? How are the programs staffed and supported for quality implementation? What participant activities are offered, and what are their primary desired outcomes?

Identification of a promising strategy for the impact study:

2. What are feasible strategies or practices for the focus of an impact study? Which has the potential to provide the most policy relevant information for the 21st CCLC grantees?

3. What is the appropriate study design to assess the impact of the strategy/practice?

Impact study component:

4. What is the effect on outcomes of the studied strategy/practice?

5. What is the cost effectiveness of the studied strategy/practice?

C. Evaluation Approach

The offeror shall consider the following assumptions:

The 21st CCLC program is a state-administered program with about 8500 grantees. The implementation study shall include a grantee survey of a nationally representative sample of grantees (250 grantees).

The impact study shall include approximately 100 afterschool centers.

As part of the feasibility and design phase, the offeror shall visit up to 9 centers (centers currently implementing one of the three potential strategies or practices of interest for the impact study).

The center shall be the unit of random assignment. Centers shall be randomly assigned to treatment (the strategy/practice of focus) or control (control centers do not use the strategy/practice of focus, at least not at the intensity of the treatment centers).

The contractor shall provide training or resources to implement the strategy/practice. The strategy/practice shall be implemented during the 2021-22 school year. Training shall consist of 6 sessions at each center. Each training session also shall include an additional day of observation, either before or after each training session.

Data collection for the impact evaluation component shall be collected in treatment and control centers (unless otherwise noted) and consist of: a director interview; program observations of the treatment and control center activities; training participation forms; 6 observations of center activity and staff-student interactions; survey of a representative sample of students in each of the treatment and control centers (1000 students total); center representative staff survey (300); administrative school records for students participating in the afterschool program.

D. Design Issues

Discussed below are a number of factors that will affect the evaluation design, and successful execution of the study and presentation of study findings. The contractor shall be cognizant of these issues while carrying out the evaluation.

Center Context. Afterschool programs are voluntary, resulting in student participation that may only be two days a week. In addition, recruiting and retaining high-quality staff, particularly from one year to the next, are also a challenge. These challenges will be need to be taken into consideration within the study design, strategy selection, and recruitment of sites to participate in the impact study.

Counterfactual. Afterschool programs are likely to implement strategies or practices of interest, at least to some extent. Thus, the offeror’s proposed evaluation design and data collection shall consider how best to create a service contrast that is policy relevant.

Afterschool staffing. Afterschool staff members are often part-time or are volunteers. Some centers staff some activities with school-day teachers. To be policy relevant, any selected strategy or practice for study and the proposed implementation support will need to take these afterschool staffing realities into consideration.

Power analyses. The contractor shall calculate the appropriate sample size to achieve a targeted effect size of 0.12-0.15. The contractor shall fully explain and justify clustering as well as other power calculation assumptions.

What Works Clearinghouse (WWC) Standards. The contractor shall design and conduct the impact study consistent with WWC standards.

SCOPE OF WORK

This section specifies the tasks and subtasks that the contractor shall perform.

Task 1—Opening Meeting with the Department

The contractor’s project director and two additional key project staff members shall meet with the COR and other appropriate Department staff within ten (10) work days after the effective date of the contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. This meeting shall be at the Department's offices in Washington, DC. The contractor shall provide an agenda for the meeting within one (1) week after award. The contractor shall come to the kickoff meeting prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within one

(1) week following the kickoff meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed in the study.

Deliverables: Meeting Agenda

Memo summarizing meeting

Task 2—Prepare OMB package and Develop Data Collection Forms The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments for the implementation study.

The package shall include a description of the study design and data collection instruments as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions.

In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the study and its potential use for policy decisions.

The contractor shall design data collection instruments to address the key study questions. The contractor shall submit a draft forms clearance package to the COR no later than September 2019. The COR will submit the accepted forms clearance package for clearance. Prior to the final 60 days in the OMB submission process, the contractor shall pilot test the instrument with no more than nine respondents to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective.

Based on comments from internal and external clearance (after the first 60 day clearance process) the contractor shall make all needed revisions and submit a revised forms OMB clearance package for submission to OMB. The contractor also shall ensure that the instrument is revised as needed based on pilot test results. The COR will then submit the accepted revised forms clearance package to OMB for final clearance (60 days).

Data collection for the implementation study shall consist of a survey of a nationally representative sample of 21st CCLC grantees.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables: Original submission OMB clearance package (first 60 days)

Revised OMB clearance package (second 60 days) Final OMB clearance package (cleared package)

Documentation of IRB approvals

Task 3 – Strategy Identification for the Impact Study

The purpose of this task is to recommend a strategy for the impact evaluation component (tasks 7-15) that is likely to provide effectiveness and program improvement information relevant to the 21st CCLC program. The contractor shall identify and learn about promising strategies/practices for consideration in conducting the impact study. In consultation with the COR, the contractor shall recommend at least three possible strategies, with attention to policy relevance and study feasibility.

Deliverables: Memo on recommended strategies for the impact evaluation

Task 4—Technical Working Group

Subtask 4.1 - The contractor shall prepare a study design memo that refines the study design, both implementation and impact components, and incorporates a detailed description of the selected strategy to be included in the impact study. The study design document will be used as materials for informing the TWG of the study design prior to the TWG meeting. A draft of the study design document shall be provided to the COR four (4) weeks prior to the TWG meeting.

Subtask 4.2 - The role of the Technical Working Group (TWG) will be to advise the contractor and the Department on the conduct of the study including, but not limited to: impact study intervention and study design, instrumentation, data collection and analysis, and draft study reports.

The TWG shall be comprised of approximately 8 individuals, and include experts in afterschool programming, youth development, supplemental academic support, as well as experts in evaluation methodology. The contractor shall submit a list of proposed TWG members no later than 2 months after contract award. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. After the COR approves the proposed TWG members, the contractor shall contact each proposed member to secure interest and intention to serve on the TWG.

Convene Technical Working Group. The contractor shall convene a meeting of the TWG in the base contract. The TWG meeting shall be held no later than 12 months after the contract award and shall focus on remaining design issues, data collection, and analysis issues.

After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S. Department of Education facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

If TWG members recommend any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review products, as appropriate, when group meetings are not required.

Deliverables: List of proposed TWG members

Study design memo Summary of TWG meeting

Task 5—Data Collection

Using instruments from Task 2 approved by OMB, the contractor shall administer the spring 2020 grantee survey. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf). As part of the monthly reporting requirement (Task 11), the contractor shall provide ongoing information about the data collection activities.

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data, such as out-of-range variables, missing data, and unusual distributions of scores, so that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set. The data monitoring activities shall be in place within 6 months after contract award. During data collection, the contractor shall include with the monthly progress reports updates on the progress of data collection based upon the data monitoring activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program.

The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. The contractor shall maintain these records and provide information on these procedures to the Government upon request. When possible, individuals collecting and/or coding data shall have no knowledge of each school’s treatment condition.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request.

The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall be responsible for safeguarding any PII that is collected for the purposes of this contract. The contractor must also comply with any relevant requirements of the Privacy Act, 5 U.S.C. 552a; the E-Government Act of 2002, 44 U.S.C. §208; as well as relevant OMB directives, such as OMB M-06-16 and OMB M-17-12. The Contractor shall abide by and follow all Departmental privacy policies, procedures, processes, and standards. All electronically stored sensitive data shall be encrypted at rest and in motion. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act of 2002 (ESRA) (20 U.S.C. 9573).

Task 6- Data Analyses and Implementation Report

The contractor shall conduct data analyses and prepare an implementation report that clearly addresses the implementation evaluation questions.

The contractor shall make sure that the implementation report is written and organized, so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers.

The contractor shall submit a draft implementation report no later than 4 months after data collection (Task 5) is completed. The COR will review and provide comments within two weeks.

The contractor shall submit a revised draft report ready for submission to the Standard’s and Review Office (SRO) within 1 month of ED comments. ED will submit the draft report to the SRO. Within approximately 2 months, ED will provide comments to the contractor from the SRO. The contractor shall make final revisions based on the SRO comments and submit a revised final implementation report within one month. The contractor shall revise the report until all peer review comments are addressed and the report clears the SRO review (current experience is an average of 2-3 rounds of successively less extensive comments). The contractor shall provide a web-ready and a print ready version of the report to the COR within 2 weeks after the report is cleared by the SRO.

Three versions of the report shall be created, including:

• A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

• A four-pager. This document shall convey the key findings and features of the report suitable for ED program staff, reporters, and policymakers (Congress and Hill staffers). It shall include key graphics and highlights.

• A 15-page report. Full technical aspects of the report shall be included in appendices.

Deliverables: Draft Implementation report (based on spring 2020 data)

Revised draft Implementation report, Standards and Review Office (SRO)-ready Final Implementation report Print and web-ready Implementation report

Task 7 —Site Recruitment

The contractor shall recruit 21st CCLC funded afterschool centers for participation in the impact evaluation component during the 2021-22 schoolyear. The study sites shall include 21st CLCC funded centers with capabilities sufficient to support the evaluation’s needs and a willingness to support and participate in the study.

Using sample selection criteria developed in consultation with the COR, the contractor shall recruit study sites to participate in and cooperate with the impact study. The contractor shall secure final agreements to participate through signed memoranda of understanding so that a total of approximately 100 afterschool programs are included in the study sample.

The contractor shall complete all recruitment activities no later than spring 2021.

As part of the recruitment process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Deliverables: Memoranda of Understanding from afterschool centers and the school district serving the participants.

Task 8—Prepare Impact Study OMB package and Develop Data Collection Forms

The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments for the impact study.

The package shall include a description of the study design and data collection instruments as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions.

In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the study and its potential use for policy decisions.

The contractor shall design data collection instruments to address the key impact study questions.

The contractor shall submit a draft forms clearance package to the COR no later than December 2020. The COR will submit the accepted forms clearance package for clearance. The contractor shall pilot test the instrument with no more than nine respondents to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective. Prior to the final 60 days in the OMB submission process, the contractor shall pilot test the instruments with no more than nine respondents to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contactor’s procedures and testing systems for administering the instruments are effective.

Based on comments from internal and external clearance (after the first 60 day clearance process) the contractor shall make all needed revisions and submit a revised forms OMB clearance package for submission to OMB. The contractor also shall ensure that the instruments are revised as needed based on pilot test results. The COR will then submit the accepted revised forms clearance package to OMB for final clearance (60 days).

Data collection for the impact study shall consist of:

• Training Activity and Participation Forms: The contractor shall develop and use forms to track training participation and activities related to the strategy focused on in the centers selected to implement the strategy treatment.

• Afterschool Director Interview: The contractor shall conduct interviews with the director of centers participating in the impact study. The purpose of the interview is to learn about center goals and programming, staff training, student participation, and challenges.

• Program Observations: The contractor shall conduct systematic observations of staff-student interactions during afterschool programming.

• Student attendance forms: The contractor shall collect student attendance forms from the centers participating in the impact evaluation. The purpose of this data collection is to assess afterschool program dosage and participation patterns.

• Student Survey: The contractor shall administer a student survey in the spring of 2022 to provide information about student characteristics and outcomes.

• Afterschool Staff Survey: The contractor shall administer an afterschool staff survey to provide information about staff characteristics and background and perceptions about afterschool offerings.

• Administrative records: The contractor shall collect information available in school records relevant to the outcomes of the practice or strategy under study (student achievement, student characteristics, school attendance, school suspensions or disciplinary actions).

The contractor shall coordinate its own IRB approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions.

Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables: Original submission OMB clearance package (first 60 days)

Revised OMB clearance package (second 60 days) Final OMB clearance package (cleared package)

Documentation of IRB approvals

Task 9.1— Random Assignment and Strategy Implementation (fall)

Prior to the first afterschool staff summer orientation session in 2021, the contractor shall randomly assign centers to treatment or control condition. The contractor shall work with centers assigned to treatment to implement the strategy, including arranging and providing training. As part of the monthly reporting requirement (Task 11), the contractor shall provide ongoing information about the implementation activities essential to the strategy/practice being well-implemented.

Task 10.1 — Impact Data Collection

Using instruments approved by OMB, the contractor shall collect data to estimate the impact of the intervention on the outcomes of interest and to answer the impact component evaluation questions. The contactor shall also keep track of costs associated with implementing the strategy. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 5 above and provide monthly information about data collection activities as part of the monthly report in Task 11.

Task 11- Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the COR, simultaneously with the monthly invoice. The Base Period includes performance reports for months 1 through 36.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report, including

OPTION PERIOD I

Task 9.2—Strategy Implementation (con’t)

The contractor shall work with centers assigned to treatment to continue to implement the strategy. As part of the monthly reporting requirement (Task 12), the contractor shall provide ongoing information about the implementation activities essential to the strategy/practice being well-implemented.

Task 10.2 — Impact Data Collection (con’t)

Using instruments approved by OMB, the contractor shall continue to collect data to estimate the impact of the intervention on the outcomes of interest and to answer the impact component evaluation questions. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 5 above and provide monthly information about data collection activities as part of the monthly report in Task 12.

Task 12- Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the COR, simultaneously with the monthly invoice. The Option Period I phase includes performance reports for months 37 through 48.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

OPTION PERIOD II

Task 13—Data Analyses and Impact Report

The contractor shall conduct data analyses and prepare a report that clearly addresses the impact evaluation questions.

The contractor shall make sure that the impact report is written and organized, so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers.

The contractor shall submit a draft impact report no later than 4 months after data collection (Tasks 10 and 11) is completed. The COR will review and provide comments within three weeks. The contractor shall submit a revised draft report ready for submission to the Standard’s and Review Office (SRO) within 1 month of ED comments. ED will submit the draft report to the SRO. Within approximately 2 months, ED will provide comments to the contractor from the

SRO. The contractor shall make final revisions based on the SRO comments and submit a revised final impact report within one month. The contractor shall revise the report until all peer review comments are addressed and the report clears the SRO review (current experience is an average of 2-3 rounds of successively less extensive comments). The contractor shall provide a web-ready and a print ready version of the report to the COR within 2 weeks after the report is cleared by the SRO.

Three versions of the report shall be created, including:

• A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

• A four-pager. This document shall convey the key findings and features of the report suitable for ED program staff, reporters, and policymakers (Congress and Hill staffers). It shall include key graphics and highlights.

• A 15-page report. Full technical aspects of the report shall be included in appendices.

Deliverables: Draft Impact report (based on impact evaluation data)

Revised draft Impact report, SRO-ready Final Impact report Print and web-ready Impact report

Task 14—Technical Working Group

Convene Technical Working Group. The contractor shall convene a meeting of the TWG members to discuss the study’s impact findings. The meeting shall be held no later than March 2023. After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members, and any lodging required by the TWG members. Finally, the COR will secure a meeting space at Department facilities.

The contractor shall provide copies of all reviews of products by the panel to the COR within five (5) work days of receipt by the contractor. If TWG members recommend any significant deviation from the data analysis plan, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR.

Deliverables: Summary of TWG meeting

Task 15—Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the COR, simultaneously with the monthly invoice. The option period II phase includes performance reports for months 49 through 60.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions.

If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

INFORMATION TECHNOLOGY AND INFORMATION SECURITY REQUIREMENTS

The contractor shall ensure that all data in a computerized form shall be suitable for data editing and corrective actions consistent with the U.S. Department of Education’s Information Quality Guidelines (http://www.ed.gov/offices/OCIO/info_quality/final_guide.html) in order to provide for quality control of data entry and yielding verified and accurate records.

The contractor shall ensure that all reports—including the final reports prepared under tasks 6 and 13—comply with the appropriate technical standards for compliance with Section 508 of the Rehabilitation Act of 1973 as amended, at 36 CFR 1994, and be subject to conformance testing by the Department’s Assistive Technology Program. All Section 508 violations uncovered during the testing process shall be remediated by the vendor at no charge to the Department.

The Department has determined that the work under this contract does not involve a federal information system and therefore does not require a Security Authorization (SA) or Authority to Operate (ATO) designation by the Department.