21st_CCLC_Eval_-_Amendment_0001_Executed_PDF.pdf

NSN 7540-01-152-8070 STANDARD FORM 30. (Rev. 10-83) Previous Edition unusable Prescribed by GSA FAR (48 CFR) 53.243

Page of Pages AMENDMENT OF SOLICITATION/MODIFICATION OF CONTRACT 1. Contract ID Code

2. AMENDMENT MODIFICATION NO. 3. EFFECTIVE DATE 4. REQUISITION/PURCHASE REQ. NO. 5. PROJECT NO. (if applicable)

6. ISSUED BY CODE 7. ADMINISTERED BY (If other than item 6) CODE

(x) 9A. AMENDMENT OF SOLICITATION NO.

9B. DATED (SEE ITEM 11)

10A. MODIFICATION OF CONTRACT/ORDER NO.

8. NAME AND ADDRESS OF CONTRACTOR (NO., Street, Country, State and ZIP Code)

CODE FACILITY CODE

10B. DATED (SEE ITEM 13)

11. THIS ITEM ONLY APPLIES TO AMENDMENTS OF SOLICITATIONS

The above numbered solicitation is amended as set forth in item 14. The hour and date specified for receipt of offers is extended, is not extended.

Offers must acknowledge receipt of this amendment prior to the hour and date specified in the solicitation or as amended, by one of the following methods:

(a) By completing items 8 and 15, and returning ______ copies of amendment; (b) By acknowledging receipt of this amendment on each copy of the offer submitted;

or (c) By separate letter or telegram which includes a reference to the solicitation and amendment numbers. FAILURE OF YOUR ACKNOWLEDGMENT TO BE RECEIVED AT THE PLACE DESIGNATED FOR THE RECEIPT OR OFFERS PRIOR TO THE HOUR AND DATE SPECIFIED MAY RESULT IN REJECTION OF YOUR OFFER. If by virtue of this amendment your desire to change an offer already submitted, such change may be made by telegram or letter, provided each telegram or letter makes reference to the solicitation and this amendment, and is received prior to the opening hour and date specified.

12. ACCOUNTING AND APPROPRIATION DATA (If required)

13. THIS ITEM ONLY APPLIES TO MODIFICATION OF CONTRACTS/ORDERS. IT MODIFIES THE CONTRACT/ORDER NO. AS DESCRIBED IN ITEM 14.

Check One A. THIS CHANGE ORDER IS ISSUED PURSUANT TO: (Specify authority) THE CHANGES SET FORTH IN ITEM 14 ARE MADE IN THE

CONTRACT ORDER NO. IN ITEM 10A.

B. THE ABOVE NUMBERED CONTRACT/ORDER IS MODIFIED TO REFLECT THE ADMINISTRATIVE CHANGES (such as changes in paying office, appropriation date, etc) SET FORTH IN ITEM 14, PURSUANT TO THE AUTHORITY OF FAR 43.103(b).

C. THIS SUPPLEMENTAL AGREEMENT IS ENTERED INTO PURSUANT TO AUTHORITY OF:

D. OTHER (Specify type of modification and authority)

E. IMPORTANT: Contractor is not, is required to sign this document and return _______ copies to the issuing office.

14. DESCRIPTION OF AMENDMENT/MODIFICATION (Organized by UCF section headings, including solicitation/contract subject matter where feasible)

Except as provided herein, all terms and conditions of the document referenced in item 9A or 10A, as heretofore changed, remains unchanged and in full force and effect.

15A. NAME AND TITLE OF SIGNER (Type or print) 16A. NAME AND TITLE OF CONTRACTING OFFICER (Type or print)

15B. CONTRACTOR/OFFEROR 16B. UNITED STATES OF AMERICA

(Signature of person authorized to sign)

15C. DATE SIGNED

(Signature of Contracting Officer)

16C. DATE SIGNED

Please see attached

1 26

91990019R00030001 FEB 22, 2019

CAMPCP

US Dept of ED-550 12th Street SW - Room 7169 Washington DC 20065

See Block 6

91990019R0003

FEB 12, 2019

See Schedule

✔ 1

Jonathan Bettis Digitally signed by Jonathan Bettis Date: 2019.02.22 08:33:35 -05'00'

Attachment Page

PAGE 2 OF 26 91990019R00030001

Version Date: 2.19.2019

U.S. Department of Education Performance Work Statement Institute of Education Sciences National Center for Education Evaluation

Performance Work Statement

An Impact Evaluation to Inform the 21st Century Community Learning Centers (CCLC) Program

I. INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (“NCEE”) of the U.S. Department of Education’s (the “Department”) Institute of Education Sciences (“IES”) requires a contract to conduct an implementation and impact study.

This contract includes a 36-month Base Period and two option periods that would result in a 60-month period of performance in total. The Base Period includes conducting an implementation study of the 21st Century Community Learning Centers (21st CCLC) program, identifying a policy-relevant strategy for an impact study to be conducted within the 21st CCLC program, and initial conduct of the impact study.

Option Period I includes continued intervention implementation and data collection activities for the impact study. Option Period I includes a 22-month period of performance and overlaps with the Base Period by 10 months.

Option Period II includes a technical working group meeting and writing an impact report to be released after obtaining approval from the Standards and Review office of IES.

Option Period II includes a 26 month period of performance and overlaps with the Option Period I by 14 months.

Base Option Period I Option Period II

July 2019 - June 2022 Sept. 2021 - June 2023 May 2022 - June 2024

36 months 22 months 26 months Overlaps with base by 10 months Overlaps with Period I by 14 months

Months 1-36 Months 27-48 Months 35-60

A. Legislative Authority

The 21st CCLC program is authorized in the Elementary and Secondary Education Act of 1965 as amended through P.L. 115-64. The 21st CCLC program provides funding for communities to establish or expand activities in community learning centers or expanded learning program activities. Community learning centers provide opportunities for

PAGE 3 OF 26 91990019R00030001

Version Date: 2.19.2019 academic enrichment and offer students a broad array of additional services, programs, and activities. Programs and activities range from academic support and enrichment to youth development, career development, and health and wellness activities. The 21st

CCLC program was reauthorized in ESSA and was funded at $1.2 billion in 2018.

B. The 21st CCLC program

The 21st CCLC funded afterschool programs tend to offer multiple activities. Based on the 2015-16 Annual Performance Report (APR) Data Collection System, homework assistance, physical activity, literacy, and Science, Technology, Engineering and Mathematics (STEM) were the activities of most frequent focus. In addition, close to half of the participating students are in grades K-5. It’s also important to note that participation is typically voluntary and states report close to half of the student participants attend the school-year program less than 30 days.

Previous evaluations of afterschool programs include two impact studies by IES, including an impact evaluation of structured academic instruction in reading or math implemented within the afterschool program setting completed in 2009 -https://ies.ed.gov/ncee/projects/evaluation/other_afterschool.asp; and an impact evaluation of the federal discretionary 21st CCLC grant program completed in 2005 -https://ies.ed.gov/ncee/projects/evaluation/other_21clc.asp). A descriptive study of the state-administered grant program in 2005-06 was conducted by the Department’s Policy and Program Studies Service and completed in 2011-https://www2.ed.gov/rschstat/eval/after-school/21st-century-community-learning-centers.pdf.

As part of a recently awarded contract by the 21st CCLC program office, a contractor will conduct a literature review of leading research in the afterschool area to provide information about outcomes that afterschool programs have affected. This effort will also provide information to support revisions of the 21st CCLC’s Government Performance Results Act measures. The Contracting Officer’s Representative (COR) will provide a copy of this work to the contractor awarded this contract a week prior to the kickoff meeting.

B. Key Research Questions

The purpose of the Department’s evaluation is two-fold. One purpose is to provide implementation information on currently funded 21st CLCC grantees. Of particular interest are the characteristics of the students served and the regularity of their participation, programmatic emphasis and variety of activities provided, staff characteristics and the training they receive, and outcomes focused on by the program.

A second goal is to estimate the impact and cost effectiveness of a particularly innovative strategy/practice useful for future 21st CCLC grantees that is implemented, in whole or part, by some current grantees. For the focus of the impact study, there are three potential strategies or practices of interest: 1) training and tools for afterschool staff that support positive youth development outcomes, 2) resources and training for afterschool

PAGE 4 OF 26 91990019R00030001

Version Date: 2.19.2019 staff in supporting STEM activities and student math achievement, and 3) promising practices in using tutoring or homework help for students, particularly those who are low-performing, to meet State academic standards.

To address these two goals, the evaluation will include two components. The first component will involve conducting an implementation study. The second component will involve identifying a feasible, policy relevant strategy or practice for the impact study, refining the study design, and carrying out the impact study. The evaluation shall address the following research questions.

Implementation study component:

1. What are the implementation experiences of centers funded by the 21st

CCLC program? Which students participate? How are the programs staffed and supported for quality implementation? What participant activities are offered, and what are their primary desired outcomes?

Identification of a promising strategy for the impact study:

2. What are feasible strategies or practices for the focus of an impact study?

Which has the potential to provide the most policy relevant information for the 21st CCLC grantees?

3. What is the appropriate study design to assess the impact of the strategy/practice?

Impact study component:

4. What is the effect on outcomes of the studied strategy/practice?

5. What is the cost effectiveness of the studied strategy/practice?

C. Evaluation Approach

The offeror shall consider the following assumptions:

The 21st CCLC program is a state-administered program funding about 8500 centers.

The implementation study shall include a survey of a nationally representative sample of 21st CCLC funded centers (approximately 250 centers).

The impact study shall include approximately 100 afterschool centers.

As part of the feasibility and design phase, the offeror shall visit up to 9 centers (centers currently implementing one of the three potential strategies or practices of interest for the impact study).

The center shall be the unit of random assignment. Centers shall be randomly assigned to treatment (the strategy/practice of focus) or control (control centers do not use the strategy/practice of focus, at least not at the intensity of the treatment centers).

The contractor shall provide training or resources to implement the strategy/practice.

The strategy/practice shall be implemented during the 2021-22 school year. Training shall consist of 6 half-day sessions at each center.

PAGE 5 OF 26 91990019R00030001

Version Date: 2.19.2019

Data collection for the impact evaluation component shall be collected in treatment and control centers (unless otherwise noted) and consist of: one director interview– treatment and control; six observations of center activity and staff-student interactions in programs assigned to treatment (two of these observations shall take place on consecutive days at the beginning of the year; two additional observations on consecutive days shall occur during the year per treatment center; and two observations shall take place on consecutive days at the end of the year); four observations of center activity and staff-student interactions in programs assigned to control (two of these observations on consecutive days shall take place at the beginning and two of these observations on consecutive days shall take place at the end of the year); training participation forms – treatment centers only; survey of a representative sample of students in each of the treatment and control centers – treatment and control (1000 students total); center representative staff survey-treatment and control (300); administrative school records for students participating in the afterschool program – treatment and control.

D. Design Issues

Discussed below are a number of factors that will affect the evaluation design, and successful execution of the study and presentation of study findings. The contractor shall be cognizant of these issues while carrying out the evaluation.

Center Context. Afterschool programs are voluntary, resulting in student participation that may only be two days a week. In addition, recruiting and retaining high-quality staff, particularly from one year to the next, are also a challenge. These challenges will be need to be taken into consideration within the study design, strategy selection, and recruitment of sites to participate in the impact study.

Counterfactual. Afterschool programs are likely to implement strategies or practices of interest, at least to some extent. Thus, the offeror’s proposed evaluation design and data collection shall consider how best to create a service contrast that is policy relevant.

Afterschool staffing. Afterschool staff members are often part-time or are volunteers.

Some centers staff some activities with school-day teachers. To be policy relevant, any selected strategy or practice for study and the proposed implementation support will need to take these afterschool staffing realities into consideration.

Power analyses. The contractor shall calculate the appropriate sample size to achieve a targeted effect size of 0.12-0.15. The contractor shall fully explain and justify clustering as well as other power calculation assumptions.

What Works Clearinghouse (WWC) Standards. The contractor shall design and conduct the impact study consistent with WWC standards.

PAGE 6 OF 26 91990019R00030001

Version Date: 2.19.2019

II. SCOPE OF WORK

This section specifies the tasks and subtasks that the contractor shall perform.

Task 1—Opening Meeting with the Department

The contractor’s project director and two additional key project staff members shall meet with the COR and other appropriate Department staff within ten (10) work days after the effective date of the contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. This meeting shall be at the Department's offices in Washington, DC. The contractor shall provide an agenda for the meeting within one (1) week after award. The contractor shall come to the kickoff meeting prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within one (1) week following the kickoff meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed in the study.

Deliverables: Meeting Agenda Memo summarizing meeting

Task 2—Prepare OMB package and Develop Data Collection Forms The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments for the implementation study.

The package shall include a description of the study design and data collection instruments as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions.

In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the study and its potential use for policy decisions.

The contractor shall design data collection instruments to address the key study questions. The contractor shall submit a draft forms clearance package to the COR no later than September 2019. The COR will submit the accepted forms clearance package for clearance. Prior to the final 60 days in the OMB submission process, the contractor shall pilot test the instrument with no more than nine respondents to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective.

Based on comments from internal and external clearance (after the first 60 day clearance process) the contractor shall make all needed revisions and submit a revised forms OMB

PAGE 7 OF 26 91990019R00030001

Version Date: 2.19.2019 clearance package for submission to OMB. The contractor also shall ensure that the instrument is revised as needed based on pilot test results. The COR will then submit the accepted revised forms clearance package to OMB for final clearance (60 days).

Data collection for the implementation study shall consist of a survey of a nationally representative sample of centers funded by the 21st CCLC program.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables: Original submission OMB clearance package (first 60 days) Revised OMB clearance package (second 60 days) Final OMB clearance package (cleared package) Documentation of IRB approvals

Task 3 – Strategy Identification for the Impact Study

The purpose of this task is to recommend a strategy for the impact evaluation component (tasks 7-15) that is likely to provide effectiveness and program improvement information relevant to the 21st CCLC program. The contractor shall identify and learn about promising strategies/practices for consideration in conducting the impact study. In consultation with the COR, the contractor shall recommend at least three possible strategies, with attention to policy relevance and study feasibility.

Deliverables: Memo on recommended strategies for the impact evaluation

Task 4—Technical Working Group

The role of the Technical Working Group (TWG) will be to advise the contractor and the Department on the conduct of the study including, but not limited to: impact study intervention and study design, instrumentation, data collection and analysis, and draft study reports.

The TWG shall be comprised of approximately 8 individuals, and include experts in afterschool programming, youth development, supplemental academic support, as well as experts in evaluation methodology. The contractor shall submit a list of proposed TWG members no later than 2 months after contract award. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. After the COR approves the proposed TWG members, the contractor shall contact each proposed member to secure interest and intention to serve on the TWG.

PAGE 8 OF 26 91990019R00030001

Version Date: 2.19.2019

Convene Technical Working Group. The contractor shall convene a meeting of the TWG in the base contract. The TWG meeting shall be held no later than 12 months after the contract award and shall focus on remaining design issues, data collection, and analysis issues. The contractor shall prepare a study design memo that refines the study design, both implementation and impact components, and incorporates a detailed description of the selected strategy to be included in the impact study. The study design document will be used as materials for informing the TWG of the study design prior to the TWG meeting. A draft of the study design document shall be provided to the COR four (4) weeks prior to the TWG meeting.

After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S. Department of Education facilities in Washington, DC. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members.

If TWG members recommend any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review products, as appropriate, when group meetings are not required.

Deliverables: List of proposed TWG members Study design memo Summary of TWG meeting

Task 5—Data Collection

Using instruments from Task 2 approved by OMB, the contractor shall administer the spring 2020 center survey. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf). As part of the monthly reporting requirement (Task 11), the contractor shall provide ongoing information about the data collection activities.

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data, such as out-of-range variables, missing data, and unusual distributions of scores, so that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set.

The data monitoring activities shall be in place within 6 months after contract award.

PAGE 9 OF 26 91990019R00030001

Version Date: 2.19.2019

During data collection, the contractor shall include with the monthly progress reports updates on the progress of data collection based upon the data monitoring activities.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program.

The training shall document the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. The contractor shall maintain these records and provide information on these procedures to the Government upon request. When possible, individuals collecting and/or coding data shall have no knowledge of each school’s treatment condition.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request.

The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L.

93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall be responsible for safeguarding any PII that is collected for the purposes of this contract. The contractor must also comply with any relevant requirements of the Privacy Act, 5 U.S.C. 552a; the E-Government Act of 2002, 44 U.S.C. §208; as well as relevant OMB directives, such as OMB M-06-16 and OMB M-17-12. The Contractor shall abide by and follow all Departmental privacy policies, procedures, processes, and standards. All electronically stored sensitive data shall be encrypted at rest and in motion. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act of 2002

(ESRA) (20 U.S.C. 9573).

Task 6- Data Analyses and Implementation Report

The contractor shall conduct data analyses and prepare an implementation report that clearly addresses the implementation evaluation questions.

PAGE 10 OF 26 91990019R00030001

Version Date: 2.19.2019

The contractor shall make sure that the implementation report is written and organized, so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers. Three versions of the report shall be created, including:

A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

A four-pager. This document shall convey the key findings and features of the report suitable for ED program staff, reporters, and policymakers (Congress and Hill staffers). It shall include key graphics and highlights.

A 15-page report. Full technical aspects of the report shall be included in appendices.

Deliverables: Draft Implementation report (based on spring 2020 data) Revised draft Implementation report, Standards and Review Office

(SRO)-ready Final Implementation report Print and web-ready Implementation report

Task 7 —Site Recruitment

The contractor shall recruit 21st CCLC funded afterschool centers for participation in the impact evaluation component during the 2021-22 schoolyear. The study sites shall include 21st CLCC funded centers with capabilities sufficient to support the evaluation’s needs and a willingness to support and participate in the study.

Using sample selection criteria developed in consultation with the COR, the contractor shall recruit study sites to participate in and cooperate with the impact study. The contractor shall secure final agreements to participate through signed memoranda of understanding so that a total of approximately 100 afterschool programs are included in the study sample.

The contractor shall complete all recruitment activities no later than spring 2021.

As part of the recruitment process, the contractor shall, as appropriate, comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects.

Deliverables: Memoranda of Understanding from afterschool centers and the school district serving the participants.

PAGE 11 OF 26 91990019R00030001

Version Date: 2.19.2019

PAGE 12 OF 26 91990019R00030001

Version Date: 2.19.2019

Task 8—Prepare Impact Study OMB package and Develop Data Collection Forms

The contractor shall prepare all relevant OMB forms and justification for a forms clearance package that includes all data collection instruments for the impact study.

The package shall include a description of the study design and data collection instruments as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions.

In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) description of and justification for incentives, (4) description of sampling and recruiting procedures, and (5) limitations of the study and its potential use for policy decisions.

The contractor shall design data collection instruments to address the key impact study questions. The contractor shall submit a draft forms clearance package to the COR no later than December 2020. The COR will submit the accepted forms clearance package for clearance. The contractor shall pilot test the instrument with no more than nine respondents to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures and testing systems for administering the instruments are effective. Prior to the final 60 days in the OMB submission process, the contractor shall pilot test the instruments with no more than nine respondents to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contactor’s procedures and testing systems for administering the instruments are effective.

Based on comments from internal and external clearance (after the first 60 day clearance process) the contractor shall make all needed revisions and submit a revised forms OMB clearance package for submission to OMB. The contractor also shall ensure that the instruments are revised as needed based on pilot test results. The COR will then submit the accepted revised forms clearance package to OMB for final clearance (60 days).

Data collection for the impact study shall consist of:

Training Activity and Participation Forms: The contractor shall develop and use forms to track training participation and activities related to the strategy focused on in the centers selected to implement the strategy treatment.

Afterschool Director Interview: The contractor shall conduct interviews with the director of centers participating in the impact study. The purpose of the interview is to learn about center goals and programming, staff training, student participation, and challenges.

Program Observations: The contractor shall conduct systematic observations of staff-student interactions during afterschool programming.

PAGE 13 OF 26 91990019R00030001

Version Date: 2.19.2019

Student attendance forms: The contractor shall collect student attendance forms from the centers participating in the impact evaluation. The purpose of this data collection is to assess afterschool program dosage and participation patterns.

Student Survey: The contractor shall administer a student survey in the spring of 2022 to provide information about student characteristics and outcomes.

Afterschool Staff Survey: The contractor shall administer an afterschool staff survey to provide information about staff characteristics and background and perceptions about afterschool offerings.

Administrative records: The contractor shall collect information available in school records relevant to the outcomes of the practice or strategy under study (student achievement, student characteristics, school attendance, school suspensions or disciplinary actions).

The contractor shall coordinate its own IRB approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables: Original submission OMB clearance package (first 60 days) Revised OMB clearance package (second 60 days) Final OMB clearance package (cleared package) Documentation of IRB approvals

Task 9.1— Random Assignment and Strategy Implementation (fall)

Prior to the first afterschool staff summer orientation session in 2021, the contractor shall randomly assign centers to treatment or control condition. The contractor shall work with centers assigned to treatment to implement the strategy, including arranging and providing training. As part of the monthly reporting requirement (Task 11), the contractor shall provide ongoing information about the implementation activities essential to the strategy/practice being well-implemented.

Task 10.1 — Impact Data Collection

Using instruments approved by OMB, the contractor shall collect data to estimate the impact of the intervention on the outcomes of interest and to answer the impact component evaluation questions. The contactor shall also keep track of costs associated with implementing the strategy. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 5 above and provide monthly information about data collection activities as part of the monthly report in Task 11.

Task 11- Reporting Requirements

PAGE 14 OF 26 91990019R00030001

Version Date: 2.19.2019

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the COR, simultaneously with the monthly invoice. The Base Period includes performance reports for months 1 through 36.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report, including

OPTION PERIOD I

Task 9.2—Strategy Implementation (con’t)

The contractor shall work with centers assigned to treatment to continue to implement the strategy. As part of the monthly reporting requirement (Task 12), the contractor shall provide ongoing information about the implementation activities essential to the strategy/practice being well-implemented.

Task 10.2 — Impact Data Collection (con’t)

Using instruments approved by OMB, the contractor shall continue to collect data to estimate the impact of the intervention on the outcomes of interest and to answer the impact component evaluation questions. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards (http://nces.ed.gov/pubs92/92021.pdf).

PAGE 15 OF 26 91990019R00030001

Version Date: 2.19.2019

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in Task 5 above and provide monthly information about data collection activities as part of the monthly report in Task 12.

Task 12- Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the COR, simultaneously with the monthly invoice. The Option Period I phase includes performance reports for months 37 through 48.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

OPTION PERIOD II

Task 13—Data Analyses and Impact Report

The contractor shall conduct data analyses and prepare a report that clearly addresses the impact evaluation questions.

The contractor shall make sure that the impact report is written and organized, so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers. Two versions of the report shall be created, including:

A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

PAGE 16 OF 26 91990019R00030001

Version Date: 2.19.2019

A four-pager. This document shall convey the key findings and features of the report suitable for ED program staff, reporters, and policymakers (Congress and Hill staffers). It shall include key graphics and highlights.

A 15-page report. Full technical aspects of the report shall be included in appendices.

Restricted use data file with documentation for study report

The contractor shall prepare a restricted use data file for the study report. The file shall contain all data collected for the evaluation with all direct personal identifiers removed.

The contractor shall provide documentation for the files including variable labels, category labels, weights, and how missing data were addressed. In addition, the contractor shall submit documentation accompanying the data to include the unit of observation for each data file (e.g., child, classroom, school, etc.).

For each file, the contractor shall also provide IES with a risk disclosure analysis for submission to National Center for Education Statistics (NCES). Once the codebook and risk disclosure analysis are approved, the contractor shall submit to the COR the restricted use data file consistent with NCES standards1. ASCII or SAS file formats are acceptable. The contractor shall submit the restricted use file for the report to the COR by 59 months after contract award.

Deliverables: Draft Impact report (based on impact evaluation data) Revised draft Impact report, SRO-ready Final Impact report Print and web-ready Impact report Restricted use data file with documentation for study report (impact and implementation data)

Task 14—Technical Working Group

Convene Technical Working Group. The contractor shall convene a meeting of the TWG members to discuss the study’s impact findings. The meeting shall be held no later than March 2023. After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The contractor shall pay the travel, per diem, and honorarium expenses of the TWG members, and any lodging required by the TWG members.

Finally, the COR will secure a meeting space at Department facilities.

The contractor shall provide copies of all reviews of products by the panel to the COR within five (5) work days of receipt by the contractor. If TWG members recommend any significant deviation from the data analysis plan, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall

1 http://nces.ed.gov/statprog/style.asp

PAGE 17 OF 26 91990019R00030001

Version Date: 2.19.2019 not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR.

Deliverables: Summary of TWG meeting

Task 15—Reporting Requirements

In addition to the deliverables that are required for each task described above, the contractor shall submit one copy of the following performance reports each month to the COR, simultaneously with the monthly invoice. The option period II phase includes performance reports for months 49 through 60.

Monthly Progress Report/Exception Reports. The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure report

III. INFORMATION TECHNOLOGY AND INFORMATION SECURITY

REQUIREMENTS

The contractor shall ensure that all data in a computerized form shall be suitable for data editing and corrective actions consistent with the U.S. Department of Education’s Information Quality Guidelines (http://www.ed.gov/offices/OCIO/info_quality/final_guide.html) in order to provide for quality control of data entry and yielding verified and accurate records.

PAGE 18 OF 26 91990019R00030001

Version Date: 2.19.2019

The contractor shall ensure that all reports—including the final reports prepared under tasks 6 and 13—comply with the appropriate technical standards for compliance with Section 508 of the Rehabilitation Act of 1973 as amended, at 36 CFR 1994, and be subject to conformance testing by the Department’s Assistive Technology Program. All Section 508 violations uncovered during the testing process shall be remediated by the vendor at no charge to the Department.

The Department has determined that the work under this contract does not involve a federal information system and therefore does not require a Security Authorization (SA) or Authority to Operate (ATO) designation by the Department. However, with regard to information obtained in carrying out the work requirements, the contractor, and all subcontractors, shall maintain a level of security that is similar to what is specified by the Federal Information Security Modernization Act (FISMA) of 2014, Office of Management and Budget (OMB) Circular A-130, Managing Information as a Strategic Resource, the most current, finalized version of the National Institute of Standards and Technology (NIST) standards and guidance SP 800-53 and SP 800-53A, and the Federal Risk and Authorization Management Program (FedRAMP) requirements and guidance. Guidance on key components of information security can be found here: https://www.nist.gov/programs-projects/federal-information-security-management-act-fisma-implementation-project

Specifically, all contractors and subcontractors shall ensure that:

1. Their IT product/system is monitored during all hours of operations using entrusted detective/preventive systems;

2. Their IT product/system has current antiviral products installed and operational;

3. Their IT product/system is scanned on a reoccurring basis;

4. Vulnerabilities are remediated in a timely manner on their IT product/system; and

5. Access/view for cyber security situational awareness on their IT product/system is made available to the Department CIRC (cyber incident response capability).

Preparing for and Responding to a Breach of Personally Identifiable Information Per OMB-M-17-12:

• The contractor shall cooperate with and exchange information with agency officials, as determined necessary by the agency, in order to effectively report and manage a suspected or confirmed breach.

• The contractor and subcontractors (at any tier) shall properly encrypt PII in accordance with OMB Circular A-130 and other applicable policies and to comply with any agency-specific policies for protecting PII;

• The contractor shall complete regular Department training for contractors and subcontractors (at any tier) on how to identify and report a breach;

• The contractor and subcontractors (at any tier) shall report a suspected or confirmed

PAGE 19 OF 26 91990019R00030001

Version Date: 2.19.2019 breach in any medium or form, including paper, oral, and electronic, as soon as possible and without unreasonable delay, consistent with the agency's incident management policy and US-CERT notification guidelines;

• The contractor and subcontractors (at any tier) shall maintain capabilities to determine what Federal information was or could have been accessed and by whom, construct a timeline of user activity, determine methods and techniques used to access Federal information, and identify the initial attack vector;

• The contractor shall allow for an inspection, investigation, forensic analysis, and any other action necessary to ensure compliance with this Federal and Department PII Breach Response policies (such as OMB-M-17-12), the Department’s breach response plan, and to assist with responding to a breach;

• The contractor shall identify roles and responsibilities, in accordance with Federal and Department PII Breach Response policies (such as OMB-M-17-12), and the agency's breach response plan; and,

• The contractor shall be aware that a report of a breach shall not, by itself, be interpreted as evidence that the contractor or its subcontractor (at any tier) failed to provide adequate safeguards for PII.

Reporting of data security breaches

If there is a suspected or known breach/disclosure of personally identifiable information (PII) related to this contract due to lost, theft, intercepted transfer, or other reason, the contractor must ensure that this breach is reported to the agency as soon as the contractor has knowledge of it. Per Office of Management and Budget Memorandum M-17-12, Federal agencies have a requirement to report breaches of PII security to a United States Computer Emergency Response Team (US-CERT). IES must notify the department within 30 minutes of discovering the incident (and the agency should not distinguish between suspected or confirmed breaches). The data security plan must be written to reflect this requirement, and the contractor must provide sufficient notification and documentation of the suspected loss, as it is understood at the time of notification to the agency for this requirement to be met. Follow-up reports of the final status of loss events will also be prepared by the contractor within a reasonable period of time as advised by the COR.

IV. SCHEDULE OF DELIVERABLES

All deliverables under these tasks shall be submitted to the Department’s Contracting Officer’s Representative (COR) electronically unless otherwise noted. Unless indicated otherwise, the COR will notify the contractor within two weeks (unless otherwise noted in the Schedule of Deliverables) of receipt of any changes required in the deliverables, and the contractor shall have up to two weeks to make the necessary revisions and submit a final deliverable to the Department.

PAGE 20 OF 26 91990019R00030001

Version Date: 2.19.2019

Deliverable Schedule Date Base Period

Task 1 – Opening Meeting with the Department Agenda Within 10…