Title_III_-_RFI_12.07.18.pdf

December 10, 2018

STUDY OF PRACTICES FOR ENGLISH

LEARNERS: TITLE III IMPLEMENTATION

EVALUATION AND IMPACT FEASIBILITY

STUDY

Solicitation No.: 91990019R0004

Request for Information & Vendor Engagement

Facilitated by the Office of the Chief Financial Officer, Contracts and Acquisitions Management Office

Contents Introduction

Capability Statement and Feedback Submission Instructions

Contact Information

Request for Capability Statements

Vendor Questions

List of Attachments

Draft Performance Work Statement

Introduction

The U.S. Department of Education is conducting market research to determine if sufficient interest and capability exists in the marketplace to support a study of practices for English learners. This document serves as an invitation for potential offerors to express interest and demonstrate skills, knowledge, and capabilities relevant to this requirement. Additionally, this is an opportunity for potential offerors to provide feedback on the requirement.

Additional Notes: The Department will not pay for the provision of any information nor will it compensate any respondents for providing this information. This notice in no way obligates the Department to award any contract to any interested party. Feedback will not be provided to any interested party who submits information to the Department in response to this notice.

Please note that all solicitation-related documents in this RFI are draft versions. The content of each document may change substantially between release of this RFI and release of the final solicitation. The final solicitation will be governed by the terms, conditions, instructions, and criteria in that document. Offerors should not rely on the contents of this RFI for preparing final proposals. To the extent that information provided in this document is inconsistent with or contradicts information contained in the final solicitation, the solicitation controls.

Capability Statement and Feedback Submission Instructions

Detailed information for the required submission is provided below. Offerors should submit, via email, the following documents to Jonathan.Bettis@ed.gov and Catherine.Shade@ed.gov with a cc to Ryan.Battad@ed.gov no later than December 20, 2018 at 2:00PM EST.

I. One PDF document that contains your Capability Statement (PDF).

II. One PDF document that contains your response to vendor questions.

III. One WORD document that contains any comments, track change revisions, or edits to the PWS (optional/as applicable).

Offerors should submit all three documents in one email. Please note that the Department imposes a 15MB file size limitation. This limitation accounts for potential file size increases associated with Microsoft Cloud services and may be different from earlier Department file size restrictions.

Participation in this vendor outreach effort is entirely voluntary. A response to this RFI is not required in order to respond to any solicitation that the Department issues.

mailto:Jonathan.Bettis@ed.gov mailto:Catherine.Shade@ed.gov mailto:Heather.Watroba@ed.gov

Contact Information Please note that only the individuals listed below are authorized points of contact for this competition. No members of the IES staff will respond to inquiries.

The following members comprise the acquisition team for this procurement:

Primary Point of Contact Jonathan E. Bettis Jonathan.Bettis@ed.gov

(202) 245-6971 Contract Specialist/Contracting Officer

Catherine Shade Catherine.Shade@ed.gov

(202) 245-6857 Contract Specialist

Secondary Point of Contact Ryan Battad Ryan.Battad@ed.gov

(202) 245-6527 Contracting Officer/Team Lead

Dissemination of Procurement Information

Offerors are strongly encouraged to communicate via email. All questions that the Department receives regarding the procurement will be disclosed to the public via www.FBO.gov. All announcements regarding the procurement will be posted to FBO.

mailto:Jonathan.Bettis@ed.gov mailto:Kevin.Wade@ed.gov mailto:Ryan.Battad@ed.gov http://www.fbo.gov/

Request for Capability Statements

Proposed NAICS Information: 541720 – Research and Development in the Social Sciences and Humanities1 Proposed Product Service Code: R410 – Support – Professional: Program

Evaluation/Review/Development

The Department requests capability statements from small businesses in order to evaluate what, if any, set-aside determinations are appropriate for each region.

The Department will not review generic capability statements because generic capability statements tell us nothing about your ability to execute the subject contract. Rather, we will only review capability statements that conform to the specifications provided below.

Format and Instructions for Capability Statements

Capability statements are limited to 7 double-spaced pages (not including the cover page) and must be submitted in PDF format. The Department will not consider material in excess of 7 pages.

1. Cover Page containing the following information:

Name of Company Primary Point of Contact (phone and email) DUNS Number Small Business Status for the applicable NAICS code (if not a small business, please indicate “OTSB” in this field)

2. Substantive Content

The capability statement must describe the collective expertise and experience of your company’s personnel, in each of the following areas:

• Experience and skill in assessing the feasibility of, designing, and carrying out large-scale randomized controlled trials of education programs, policies, and practices. Interested parties shall provide a minimum of two examples of personnel’s successful experience and skill in conducting these activities. This includes experience producing impact evaluation design reports that balance rigor, policy relevance, feasibility, and cost effectiveness of design options.

1 Please note that we welcome feedback on other appropriate NAICS and PSC codes.

• Experience and skill in designing, managing, and conducting large-scale implementation evaluations of education programs and practices. Interested parties shall provide a minimum of two examples of personnel’s successful past experience and skill in designing, managing, and conducting all aspects of evaluations in education settings that are similar to this procurement in content (e.g., English language acquisition), scope (e.g., nationally representative), sample size, and data collection modes (e.g., evidence reviews, surveys, site visits).

• Experience and skill in authoring well-written, policy relevant publications with clear take-home messages for wide audiences. Interested parties shall provide a minimum of two examples of personnel’s involvement with authoring publications that clearly summarize implementation findings and present them so that they can be understood and used by a wide audience.

• Substantive knowledge relevant to the draft Study of Practices for English Learners PWS, including language instruction and other educational supports for English learners, factors related to English language acquisition, academic performance, and social emotional learning of English learners, and federal, state, and local policies for serving English learners, including Title III of ESEA.

Interested parties shall provide a minimum of two examples showing that personnel have relevant knowledge to effectively study programs and practices for English learners, particularly capitalizing on what is known from the research literature.

In addition, the capability statement shall address the following experience and capacity at the corporate level:

• Experience with projects similar in nature and scale to that described in the draft Study of Practices for English Learners PWS. Interested parties shall provide evidence of successful corporate experience with a minimum of two past projects that are similar to that described in the draft PWS, in terms of the specific tasks to be performed, and the scale and scope required.

Vendor Questions

Note: all potential offerors, small businesses or otherwise, are encouraged to provide responses to these questions.

(1) What, if any, additional information should the Department provide in order to enable you to accurately estimate the cost of each task/subtask? This includes documents, research materials, more accurate task descriptions, policy information, or other content produced by or utilized by the Department.

(2) Are the NAICS and PSC codes appropriate for this procurement? If not, please justify an alternative.

(3) Can you identify any barriers to competition? If so, please identify and describe how the Department could reduce or eliminate those barriers.

(4) Please explain whether GSA’s OASIS contract vehicle is a suitable alternative to a full and open competition and please identify any other Tier III Category Management contract vehicles in addition to OASIS that may be suitable for this procurement.

(5) Please explain whether this procurement should be set-aside for small businesses.

Alternatively, please explain why this procurement should not be set-aside.

(6) Is there any other information that you wish to convey to the Department?

List of Attachments

• Performance Work Statement

Draft Performance Work Statement

Note: PWS Draft 11.28.18

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

STUDY OF PRACTICES FOR ENGLISH LEARNERS:

TITLE III IMPLEMENTATION EVALUATION AND IMPACT FEASIBILITY STUDY

(SPEL)

Performance Work Statement

INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (NCEE) of the U.S. Department of Education’s (the Department) Institute of Education Sciences (IES) requires a contract for an evaluation to inform policy and practice under Title III, Part A of the Elementary and Secondary Education Act (ESEA).

There are two major components of this requirement: an implementation evaluation of practices for serving English learners both through Title III of the ESEA and more generally, and an impact feasibility study of educational practices to promote the English proficiency, academic achievement, and social-emotional learning of English learners.

The total contract period of performance is 36 months.

Legislative Authority

Grants to support language instruction for ELs and immigrant students are authorized under Title III of the ESEA, as amended by the Every Students Succeeds Act of 2015 (P.L. 114-95). Subsection (a) of Section 8601 of the ESEA authorizes the Department to reserve up to 0.5 percent from the Title III program to carry out evaluation activities.

Background

English learners (ELs) are students with a primary language other than English whose difficulties in speaking, reading, writing, or understanding the English language may deny them the ability to be successful in classrooms where the language of instruction is English, meet academic standards, or participate fully in society. The purpose of the English Language Acquisition, Language Enhancement, and Academic Achievement Act (Title III, Part A of ESEA) is to assist ELs in attaining English proficiency and meeting the same challenging academic standards as all students. Title III was funded at $737.4 million in fiscal year 2018. The Department distributes Title III funding to states based on the number of EL and immigrant students in each state. States must use at least 95 percent of their allotted Title III funds to award subgrants to eligible local educational agencies (LEAs). Subgrantees provide language instruction educational programs for ELs and professional development for educators of ELs to enhance their capacity to implement instructional programs.

States, districts, and policymakers need information about effective instructional programs for ELs.

During the last national evaluation of Title III implementation in 2009–10, 43 percent of Title III district officials reported that a lack of proven curricula and programs for ELs was a moderate or major challenge2. The number of ELs in the U.S. has been increasing. Almost 5 million ELs were enrolled in public elementary and secondary schools in fall of 2015, accounting for 9.5 percent of the public school population in the U.S.3 In the 2017 National Assessment of Educational Progress, the gap in reading assessment scores between ELs and their non-EL peers was 37 points for fourth graders and 43 points for eighth graders. The size of the gap between ELs and non-ELs has not changed appreciably in a decade.4 Ensuring ELs reach proficiency in English and achieve at high levels in academic subjects remains a central goal for the Department.

Title III requires states and districts to select and use effective methods of instruction in programs and activities to help ELs attain English language proficiency and meet academic standards. However, there is limited evidence on effective language instruction curricula for ELs. The most recent national study of Title III implementation was based on data collected in 2009-10. Since changes in the 2015 reauthorization of ESEA could affect how districts and schools serve ELs, it is important to collect current information on state policies and how states and LEAs use Title III funds under the reauthorized law5. In addition, there is limited information available on the instructional programs and supports ELs receive through Title III and other sources to promote English proficiency, academic achievement, and other outcomes including social-emotional learning. This information is important to document implementation as well as to identify widely used and/or potentially promising strategies for further testing.

Key Evaluation Questions

The purpose of the Department’s Study of Practices for English Learners is two-fold. One purpose is to provide information on how states and districts are implementing Title III and the services and supports they provide to meet the educational needs of ELs more generally. Of particular interest are the specific language curricula or instructional strategies used, their content, and the settings in which ELs are served.

The Department is also interested in learning more about other activities and strategies used to enhance or supplement language instruction educational programs for ELs, particularly in the area of social emotional learning. The second purpose of the evaluation is to identify policy-relevant models or strategies for improving EL student outcomes and to develop feasible and policy relevant design options to test the effectiveness of one or more of these strategies for ELs.

To address these two purposes, the contract will include two main components. The first component is an implementation evaluation of Title III and other services for ELs. The second component is an impact feasibility/design study of policy-relevant strategies to improve EL’s English proficiency, academic achievement, and/or social emotional learning.

2 U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, National Evaluation of Title III Implementation—Report on State and Local Implementation, Washington, DC, 2012. https://www2.ed.gov/rschstat/eval/title-iii/state-local-implementation-report.pdf

3 https://nces.ed.gov/programs/digest/d17/tables/dt17_204.20.asp 4 https://www.nationsreportcard.gov/reading_2017/nation/gaps/ 5 The 2017 National Academies of Science, Engineering, and Medicine report Promoting the Educational Success of Children and Youth Learning English: Promising Futures recommended that IES examine variations in state policies and decisions related to ESSA implementation.

https://www2.ed.gov/rschstat/eval/title-iii/state-local-implementation-report.pdf https://nces.ed.gov/programs/digest/d17/tables/dt17_204.20.asp https://www.nationsreportcard.gov/reading_2017/nation/gaps/

In performing the tasks described in this Performance Work Statement (PWS), the contractor shall address, at a minimum, the following questions.

Implementation Evaluation

1. How do SEAs and LEAs use Title III funds? What other resources do SEAs and LEAs use for educational programs and supports for ELs?

Of particular interest is measuring the extent to which SEAs and LEAs use Title III funds for various activities (for example, language instruction educational program curricular materials versus other supports such as social emotional learning or family engagement) and classes of expenditures (for example, staff salaries versus other expenditures). Also of interest is measuring the extent to which SEAs and LEAs use other sources of funding to serve ELs.

2. What are state and district policies and procedures for implementing Title III and serving

ELs more generally? How do states and districts make decisions about EL entry and exit status, English proficiency, and curricula or methods of instruction for language instruction educational programs and other supports for ELs?

SEA implementation topics of interest include: establishing and implementing statewide entrance and exit procedures and assessing students for EL status; selection, monitoring, and evaluation of subgrantees; educator preparation and professional development; technical assistance to subgrantees to identify and implement effective language instruction educational programs and curricula for teaching ELs and to identify or develop and implement measures of English proficiency; and implementation challenges and key information or evidence needs.

LEA implementation topics of interest include: outcome domains of focus for EL services;

assessment of English proficiency against State’s long-term goals; identifying and selecting language instruction educational programs and other supports to serve ELs; educator professional development; parent engagement and information activities; characteristics of schools and teachers providing services to ELs and the students served; and implementation challenges and key information or evidence needs.

3. What services do ELs receive to promote English proficiency, academic achievement, and social emotional learning, through what sources, and in what settings?

Of particular interest is understanding what specific curricula or methods of instruction are used in language instruction educational programs to assist ELs in attaining English proficiency and academic achievement, as well as what specific other strategies are used to enhance or supplement language instruction educational programs.

4. What are the social emotional learning needs of ELs and how do states, districts, and schools identify and support those needs? How well do existing SEL interventions meet these needs?

The connection between social emotional learning (SEL) skills and academic performance of ELs is of particular policy interest for the Office of English Language Acquisition (OELA). The Department requires information on the SEL needs of ELs identified by states and districts, how these needs are identified, the extent to which states and districts use Title III funds for SEL programs geared toward ELs, the extent to which SEL supports are provided to ELs through Title III or other sources of funding, the specific models or strategies used and how they are selected, the extent to which SEL outcomes are measured and how, and challenges to implementing SEL programs under Title III.

Impact Feasibility Study

1. What are the most policy-relevant impact questions related to instruction and support services for ELs? What are the most credible and feasible study designs for each of the relevant impact questions?

2. What practices and instructional strategies have the most promise in improving the English proficiency, academic achievement, and/or social-emotional learning for ELs?

3. What is the target population of greatest interest to the field, as far as instructional staff, districts, schools, and students? What is the most feasible and policy-relevant counterfactual for the study?

4. What are the outcomes of greatest importance for the field, and what are the most credible measures of these outcomes?

5. Would program developers/training providers be interested in participating in a rigorous study, and how could the study work with or within existing programs and school district arrangements?

6. What are the key timelines/supports/incentives required for program developers, school district, and ultimately school and instructional staff participation in the study?

Evaluation Approach

The contractor’s data collection plan shall describe the proposed respondents, target sample sizes, methods, topics/content to be measured, schedule, frequency, and respondent burden. The Department assumes the following proposed approach:

• The implementation evaluation will include data collection (assumed to be surveys informed by a review of state and local plans) from all SEAs and a nationally representative sample of 1,000 LEAs in spring 2020, and data collection (assumed to be interviews and site visits) from at least 9 LEAs in fall 2020 on social emotional learning.

• The impact feasibility study will include an evidence review of up to 10 widely-used and/or promising strategies as well as data collection (assumed to be interviews and site visits) from at least 9 LEAs and program developers in spring 2021 to observe strategies as implemented in classrooms and inform impact design options. Also, the Department intends for the implementation evaluation spring 2020 and fall 2020 data collections to inform the impact feasibility study.

Design Issues

Some factors that will affect the evaluation design and successful conduct of the study and presentation of study findings are discussed below. In the study design, the contractor shall discuss how each of the following issues, as well as any other issues they identify, will be addressed. Other design issues for the impact feasibility study may arise as a result of findings from the implementation evaluation.

• Implementation Evaluation Data Collection Plan. The key evaluation questions cover a wide range of topics. The contractor shall develop a cost effective data collection and sampling plan that confirms whether the proposed levels of data collection in section D (staff at the SEA and LEA levels) are appropriate to address the evaluation questions and proposes additional levels if necessary to address the evaluation questions (for example, if a school level data collection is deemed necessary to address some of the evaluation questions). The contractor’s plan shall also identify within each level who (which staff position) would be the most knowledgeable respondent. To reduce burden on respondents, the contractor shall consider whether existing data (or data already being collected by the Department) may be appropriate to address the evaluation questions.

• Wider context of services for ELs. While the federal Title III program is an important source of funds for serving ELs, the Department is interested in understanding the full picture of educational services and supports that states, districts, and schools provide ELs. The contractor’s study design must measure use of Title III funds as well as provide information on the education programs and supports available to ELs across other funding sources.

• Impact Feasibility Study. The contractor’s final proposed study design options must use the most credible scientific methods feasible, and have the potential to provide information about the Title III program and effective methods for serving ELs that is useful to policymakers and practitioners. As discussed in the section on key evaluation questions, a particular interest is learning more about state and local information and evidence needs for serving ELs. In the impact feasibility study, the contractor’s study design shall balance rigor, policy relevance, feasibility, cost effectiveness, and responsiveness to needs identified by states and districts.

SCOPE OF WORK

This contract will be cost plus fixed fee (CPFF) with a period of performance of 36 months.

The specific tasks and subtasks that the contractor shall perform are described below.

Task 1—Communicate with the Department

Subtask 1.1 Kick-Off Meeting The contractor’s project director and up to two additional key project staff members shall meet with the Contracting Officer (CO), Contracting Officer’s Representative (COR) and other appropriate Department staff within 2 weeks of contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. This meeting shall be held at the Department’s offices in Washington, D.C. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. The contractor shall submit an agenda for the meeting to the COR within one (1) week of contract award. Within one week following the meeting, the contractor shall submit a memo to the COR summarizing the key issues and concerns raised at the meeting, and how each will be addressed in the study.

Deliverables: Kick-Off meeting agenda Kick-Off meeting summary

Subtask 1.2 Ongoing Meetings The contractor shall participate in regular meetings with the COR (and other Department staff, as relevant and appropriate) to update the COR on the progress of contract activities and to discuss any issues or anticipated problems and proposed solutions to those problems. The Department assumes the meetings will take place via conference call or webinar every other week during the first 6 months of the contract, and monthly for the remainder of the contract (the contractor shall also be available for additional calls if requested by the COR).

The contractor shall submit a proposed agenda to the COR no later than two working days before each call. Within three working days of each call, the contractor shall submit to the COR a brief summary of the key issues and concerns discussed along with the contractor’s understanding of the COR’s technical direction and next steps.

Deliverables: Meeting agendas Summaries of telephone calls

Subtask 1.3 Reporting Requirements The contractor shall submit one copy of the following performance reports each month through the Invoice Processing Platform (simultaneously with the monthly invoices).

Monthly Progress Report/Exception Reports: The contractor shall prepare monthly progress reports due at the same time as the monthly invoice. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task or subtask regarding significant findings and events, problems encountered, changes in personnel and their security designation, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from the Department. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports: The contractor shall prepare monthly expenditure reports due at the same time as the monthly invoice. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures; segregating project costs by individual and by task or subtask, and specifying for all travel the locations, duration, and personnel for each trip.

Deliverables: Monthly progress & manpower/expenditure reports

Task 2—Prepare OMB Package and Data Collection Forms and Refine Implementation Evaluation Design

The contractor shall refine the study design submitted in their proposal and use this design to guide the development of the data collection forms and OMB clearance package. The contractor shall design data collection instruments to address the key evaluation questions. The contractor shall make use of or refine existing instruments to the extent possible and applicable. For instruments where validity or reliability are not yet well established, the contractor shall pilot test each of the instruments with no more than nine respondents per instrument to ensure that the questions can be answered in a reasonable amount of time, that the directions are easily followed, and that the contractor’s procedures for administering the instruments are effective. Piloting is to be conducted prior to the final 60 days in the OMB submission process. The contractor shall ensure that instruments are revised as needed based on pilot test results.

The contractor shall prepare all relevant data collection forms and OMB clearance forms and justification components required for the Department to submit the components as a clearance package to OMB under procedures of the Paperwork Reduction Act and 5 CFR 1320. In the package, the contractor shall include a description of the implementation evaluation design and data collection instruments, as well as the necessary forms and sample notification materials required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions.

The contractor shall submit the first draft forms clearance package and data collection instruments to the COR no later than September 13, 2019. The contractor shall revise the package to reflect COR comments and submit a revised package to the COR no later than October 11, 2019. Assuming the revised package is acceptable, the COR will submit it for clearance. Based on comments from internal and external clearance received during the first 60 day public comment period, the contractor shall address all comments, make all needed revisions, and submit a revised package to the COR within one (1) week of receipt of the comments. The contractor shall address all comments and make any needed revisions to the package and forms in response to the second 60-day public comment period and OMB review within one

(1) week of receipt of the comments. The OMB clearance process typically takes a minimum of 5 months, thus OMB clearance is expected to be received in mid-March 2020.

The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field testing, and full scale data collection with the IRB approval obtained by districts from their own institutions. Specifically the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB. The contractor shall submit documentation of IRB approval to the COR by March 13, 2020.

Deliverable: First draft OMB clearance package and data collection instruments

Revised OMB clearance package and data collection instruments (for first 60 day public comment period) Revised OMB clearance package and data collection instruments (for second 60 day public comment period) Final OMB clearance package and data collection instruments (cleared package) Documentation of IRB approvals

Task 3—Technical Working Group (TWG)

Subtask 3.1 Establish Technical Working Group The role of the TWG will be to advise the contractor and the Department on the conduct of the implementation evaluation and the impact feasibility study, including but not limited to: study design, instrumentation, data collection and analysis, and draft study reports. The TWG shall be comprised of up to 10 individuals and include experts in: English learners and their acquisition of English, academic performance, and social emotional learning; evidence-based curricula and strategies in language instruction educational programs; Title III and federal, state, and local policies for serving ELs; and evaluation methodology.

The contractor shall submit a list of proposed TWG members no later than one month after contract award. When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG nominee has a bias or financial interest in the outcome of the study. The contractor shall also clearly explain the background and expertise of each proposed TWG member, and how it is specifically relevant to the needs of the study. After the CO and COR approve the proposed TWG members, the contractor shall contact each approved member to secure interest and intention to serve on the TWG.

Deliverable: List of proposed TWG members

Subtask 3.2 Prepare materials for and Summary of Technical Working Group Meeting 1 The contractor shall use the implementation evaluation design, data collection plan and instruments under task 2 as the basis for materials for informing the TWG about discussion topics prior to the first TWG meeting as well as for the preparation of presentation materials to use during the first TWG meeting.

The contractor shall submit to the COR a draft of these presentation materials and an agenda for the first TWG meeting no later than four weeks prior to the first TWG meeting. The contractor shall revise the materials for the first TWG meeting in response to any COR comments and submit revised materials to the COR no later than one week prior to the first TWG meeting.

The contractor shall submit a written summary of the meeting to the COR within one week after the TWG meeting, detailing suggestions and recommendations along with the contractor’s proposed actions. The contractor shall not move forward with proposed actions unless approved by the COR.

Deliverables: Draft TWG materials and agenda Revised TWG materials and agenda Summary of TWG meeting 1

Subtask 3.3 Convene Technical Working Group—Meeting 1 The contractor shall convene a meeting of the TWG no later than November 2019 to obtain input on implementation evaluation design, particularly the data collection plan and instruments. This meeting shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Subtask 3.4 Prepare materials for and Summary of Technical Working Group Meeting 2 The contractor shall use the preliminary findings from the implementation evaluation spring 2020 data collection under subtask 5.2, the impact feasibility plan and protocol developed under subtask 6.1, and the data collection plan for the SEL component of the implementation evaluation under subtask 4.2 as the basis for materials for informing the TWG about discussion topics prior to the second TWG meeting as well as for the preparation of presentation materials to use during the second TWG meeting.

The contractor shall submit to the COR a draft of these presentation materials and an agenda for the second TWG meeting no later than four weeks prior to the second TWG meeting. The contractor shall revise the materials for the second TWG meeting in response to any COR comments and submit revised materials to the COR no later than one week prior to the second TWG meeting.

The contractor shall submit a written summary of the meeting to the COR within one week after the TWG

Summary of TWG meeting 2

Subtask 3.5 Convene Technical Working Group—Meeting 2 The contractor shall convene a meeting of the TWG no later than October 2020 to obtain input on the preliminary analyses for the implementation evaluation report as well as how the findings should inform the design of the SEL component of the implementation evaluation and the design of the impact feasibility study. This meeting shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Subtask 3.6 Prepare materials for and Summary of Technical Working Group Meeting 3 The contractor shall use the preliminary findings from the SEL component of the implementation evaluation under subtask 5.3 and the impact feasibility results and draft impact study design options developed under task 7 as the basis for materials for informing the TWG about discussion topics prior to the third TWG meeting as well as for the preparation of presentation materials to use during the third TWG meeting.

The contractor shall submit to the COR a draft of these presentation materials and an agenda for the third TWG meeting no later than four weeks prior to the third TWG meeting. The contractor shall revise the materials for the third TWG meeting in response to any COR comments and submit revised materials to the COR no later than one week prior to the third TWG meeting.

The contractor shall submit a written summary of the meeting to the COR within one week after the TWG

Summary of TWG meeting 3

Subtask 3.7 Convene Technical Working Group—Meeting 3 The contractor shall convene a meeting of the TWG no later than October 2021 to discuss the impact feasibility study results and design options and preliminary findings from the SEL component of the implementation evaluation data collection. This TWG meeting shall focus on the policy relevance of the proposed key research questions, the study design options, identification and recruitment feasibility of a relevant counterfactual comparison group, data collection, and analysis issues. This meeting shall be held at the Department’s offices in Washington, D.C., and the contractor shall cover the travel, per diem, and honorarium expenses of the TWG members. Food and beverage shall not be provided for attendees, but the contractor shall build in a coffee break and a lunch break time into the agenda.

Task 4—Implementation Evaluation Data Collection

Subtask 4.1 Conduct Spring 2020 Data Collection

Following the Department-approved data collection plan and using instruments approved by OMB, the contractor shall collect data from the sample selected and recruited for the evaluation to answer the first three implementation evaluation questions. The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards and, if needed, implement a non-response bias analysis (https://nces.ed.gov/statprog/2002/stdtoc.asp). The contractor shall complete the data collection by June 2020. The contractor shall submit biweekly updates on the status of the data collection to the COR during each period of data collection.

https://nces.ed.gov/statprog/2002/stdtoc.asp

To ensure that data collected are of high quality, the contractor shall quickly identify patterns of inaccuracies and anomalies in the data (such as out-of-range variables, missing data, and unusual distributions of scores), such that immediate corrective action can be initiated. All such problems and subsequent corrections must be carefully documented and compiled into a technical report that shall be included with the final archived data set. The contractor shall put these data monitoring activities in place within 6 months after contract award.

To ensure that the data collection and coding staff have the skills necessary to perform their duties, and to ensure consistency of training across time, the contractor shall design a comprehensive training program with training manuals for data collection and data coding. The manuals shall detail the topics covered in the training, the minimum performance standards that the data collectors or coders must meet, the criteria by which the contractor shall ascertain whether data collectors and coders meet the performance standards, how often performance will be evaluated, and how the contractor will verify that the training of data collectors and coders will be consistent across all waves of data collection. The contractor shall submit the training manuals to the COR at least 4 weeks prior to the start of data collection.

All persons who collect, code, or otherwise have access to raw data must be trained in confidentiality issues and shall provide signed assurances of nondisclosure of confidential information. The contractor shall maintain these records and provide them to the Government upon request. When possible, the contractor shall ensure that individuals collecting and/or coding data do not have knowledge of the study participant’s treatment condition. The contractor, and all subcontractors, shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or the Department’s final regulations on the protection of human research participants. The contractor shall also satisfy any state, district, or school policies regarding school participation in external research projects. Contractor and subcontractor employees who “maintain” (collect, maintain, use, or disseminate) data shall comply with the requirements of the confidentiality standards in section 183 of the Education Sciences Reform Act of 2002 (ESRA) (20 U.S.C. 9573).

Deliverables: Data Collection Training Manuals

Data collection updates

Subtask 4.2 Conduct Fall 2020 Data Collection on SEL

The contractor shall update the data collection plan for the SEL component of the implementation evaluation to reflect the findings of the spring 2020 data collection. The contractor shall submit the data collection plan for the SEL component of the implementation evaluation to the COR no later than August 2020.

Following the Department-approved data collection plan and using instruments approved by OMB (if applicable), the contractor shall collect data from the sample selected and recruited for the SEL component of the implementation evaluation to address the fourth implementation evaluation question.

The contractor shall utilize appropriate techniques to obtain response rates of at least 85 percent according to the National Center for Education Statistical Standards and, if needed, implement a non-response bias analysis (https://nces.ed.gov/statprog/2002/stdtoc.asp). The contractor shall complete the data collection by December 2020.

To ensure that data collected are of high quality, the contractor shall use all procedures outlined in subtask

4.1 above and submit biweekly updates on the status of the data collection to the COR during data collection.

https://nces.ed.gov/statprog/2002/stdtoc.asp

Deliverables: SEL data collection plan

Data collection updates

Task 5—Data Analysis and Reports

Subtask 5.1 Refine Analysis Plan The contractor shall prepare a detailed data analysis plan and annotated report outline that documents the intended analyses to be conducted using the data collected in task 4 to address the evaluation questions.

The final plan shall be used to inform reporting of analyses on the data (see subtasks 5.2 and 5.3).

The contractor shall submit the draft data analysis plan for the implementation evaluation report (subtask 5.2) to the COR no later than May 2020. The COR will review the draft analysis plan within two weeks.

The contractor shall make revisions to reflect COR comments and submit a revised analysis plan to the COR no later than July, 2020.

The contractor shall submit the draft data analysis plan for the SEL evaluation brief (subtask 5.3) to the COR no later than January 2021. The COR will review the draft analysis plan within two weeks. The contractor shall make revisions to reflect COR comments and submit the final analysis plan to the COR no later than February 2021.

Deliverables: Draft implementation report analysis plan Revised implementation report analysis plan Draft SEL evaluation brief analysis plan Final SEL evaluation brief analysis plan

Subtask 5.2 Prepare Implementation Evaluation Report

Following the Department-approved analysis plan, the contractor shall conduct data analyses and prepare an implementation evaluation report that clearly addresses the first three implementation evaluation questions.

The contractor shall make sure that the evaluation report is written and organized so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers. The contractor shall create three versions of the implementation report including:

• A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

• A four-pager. This document shall convey the key findings and features of the report suitable for Department program staff, reporters, and policy makers (Congress and Hill staffers). It shall include key graphics and highlights.

• A 15-page report. Full technical aspects of the report shall be included in appendices.

The contractor shall conduct a briefing for Department staff on the results of the analyses that will go into the implementation evaluation report no later than 6 weeks prior to TWG meeting 2. The contractor shall prepare briefing presentation slides and other supporting materials to briefly summarize the study design and analysis plan and then present preliminary findings addressing the implementation evaluation questions based on the analysis. No later than 3 weeks before the briefing, the contractor shall submit draft materials to the COR. The COR will review and comment within 1 week. The contractor shall revise the materials and submit a revised version to the COR for NCEE leadership review no later than 1 week before the briefing.

The contractor shall use the briefing materials and Department feedback received on the preliminary analysis to prepare materials for the TWG meeting 2.

The contractor shall submit the first draft of the implementation report for COR review no later than 6 weeks after TWG meeting 2. The COR will review the first draft within three weeks. The contractor shall incorporate comments and submit a revised draft report to the COR no later than 12 weeks after TWG meeting 2. The Department will expect this draft to be acceptable for IES peer review submission; if the report is not acceptable for peer review submission, the contractor shall submit revisions until the comments are adequately addressed so that the COR can submit the report for peer review. After receipt of initial comments from the peer review (expected February 2021), the contractor shall make appropriate revisions to the report and produce a response memo detailing how the peer review comments were addressed and noting the appropriate page numbers and submit these documents to the COR no later than two weeks after receipt of peer review comments. The Department will expect this revised draft based on IES peer review (and response memo) to be responsive to the peer review comments and if it is not, the contractor shall submit revisions until the documents are responsive to peer review comments based on the COR’s review. After receipt of final comments from IES peer review, the contractor shall revise the report accordingly and submit the final web-ready report to the COR no later than two weeks after receipt of final comments.

Deliverables: Draft briefing materials Final briefing materials First draft implementation report (based on spring 2020 data)

Revised implementation report, SRO-ready Revised implementation report and peer review response memo Final web-ready implementation report

Subtask 5.3 Prepare SEL Evaluation Brief

Following the Department-approved analysis plan, the contractor shall conduct data analyses and prepare an evaluation brief that clearly addresses the fourth implementation evaluation question. The contractor shall make sure that the evaluation brief is written and organized so that it is accessible to policy makers and research-savvy practitioners rather than academic researchers. The contractor shall create three versions of the evaluation brief including:

• A one-pager. This shall be a stand-alone introduction to the brief and contain enough information to form the basis for a press release.

• A four-pager. This document shall convey the key findings and features of the brief suitable for Department program staff, reporters, and policy makers (Congress and Hill staffers). It shall include key graphics and highlights.

• A 15-page report. Full technical aspects of the brief shall be included in appendices.

The contractor shall conduct a briefing for Department staff on the results of the analyses that will go into the evaluation brief no later than 6 weeks prior to TWG meeting 3. The contractor shall prepare briefing presentation slides and other supporting materials to briefly summarize the study design and analysis plan and then present preliminary findings addressing the implementation evaluation questions based on the analysis. No later than 3 weeks before the briefing, the contractor shall submit draft materials to the COR.

The COR will review and comment within 1 week. The contractor shall revise the materials and submit a revised version to the COR for NCEE leadership review no later than 1 week before the briefing.

The contractor shall use the briefing materials and Department feedback received on the preliminary analysis to prepare materials for the TWG meeting 3.

The contractor shall submit the first draft of the evaluation brief for COR review no later than 6 weeks after TWG meeting 3. The COR will review the first draft within three weeks. The contractor shall incorporate comments and submit a revised draft brief to the COR no later than 12 weeks after TWG meeting 3.