DRAFT_Academic_Lanaguage_Interventions_PWS_(May_18 _2015).docx

PWS for ED-IES-15-R-0024 Version Date: May 18, 2015 Performance Work Statement for the

“Impact Evaluation of Academic Language Interventions” Contract

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance

Table of Contents

Introduction

The National Center for Education Evaluation (NCEE) of the U.S. Department of Education’s Institute of Education Sciences (IES) intends to evaluate the impact of academic language interventions on the academic achievement of English Learners (ELs), as well as non-EL disadvantaged students in the same classrooms. Specifically, this contract will be responsible for conducting a large-scale effectiveness study of rigorously tested academic language interventions that have previously shown positive impacts on EL achievement.

This contract includes a 30-month base period and four 18-22 month option periods that would extend the contract to 6 years. The base period includes the development of the study design, the selection of the academic language interventions (based on input from a technical working group of experts, hereafter referred to as “TWG”), sample recruitment, the development of data collection forms for an Office of Management and Budget (OMB) clearance package, random assignment, and training in the academic language interventions for implementation in the 2017-2018 school year. Based on the preliminary results, the Department will decide whether or not to exercise Option Period I by December of 2016. Option period I will be 20 months and begin in March 2017. This option period includes support for the training on the selected academic language interventions, initial program implementation, baseline data collection, as well as a briefing on the progress of training and implementation. Option Period II is 22 months and would overlap with Option Period I by 10 months. Based on documentation of successful year program implementation, the Department will decide whether or not to exercise Option Period II by October 2017. Option Period II includes initial follow-up data collection, the development of a report on implementation, and briefings. Option Period III is 22 months and would overlap with Option Period II by 13 months. Option Period III will begin in October 2018 and includes longer-term data collection and a report on findings from the first follow-up data collection. Option Period IV is 18 months and will begin in March 2020. This option period will include a final report covering the longer-term data collection that conveys the goals and findings of the evaluation for an audience of decision-makers and practitioners.

Legislative Authority

This impact evaluation of academic language interventions is authorized under the direction of two legislative authorities. The first is Title III, Part B, Subpart 2 Section 3221 of the Elementary and Secondary Education Act (ESEA), which is Research, Evaluation, and Dissemination. This section provides direct authority to conduct research and program evaluation activities for the purpose of improving English language instruction programs for limited English proficient children.

In addition, the Consolidated Appropriations Act of 2014 (P.L. 113-67) allows the Department to strengthen impact evaluation work by pooling resources across ESEA programs. Together, these authorities make it possible to fund rigorous evaluations and to evaluate the impact of various strategies that cut across a wide range of ESEA programs. The Department has decided to combine funding from these two legislative authorities to test the effectiveness of interventions that aim to improve the academic achievement of ELs given the dearth of information on effective instructional practices for this population of students. Since academic language interventions are implemented in the context of regular classroom instruction, this study will also estimate the impact of these interventions on non-EL disadvantaged students.

Background

Districts and policymakers need information about effective academic language interventions that have been tested in a variety of settings and demonstrate effects on key subgroups of the EL population. In 2009–10, nearly half the Title III district officials (43 percent) reported that a lack of proven curricula and programs for ELs was a moderate or major challenge[footnoteRef:1]. The estimated number of ELs in public schools was 4.4 million during the 2011-2012 school year, in comparison to 4.1 million during the 2002-2003 school year[footnoteRef:2]. Similarly, the number of school-age children who speak a language other than English at home continues to rise steadily. Between 1979 and 2008, the number of children ages 5-17 who spoke a language other than English at home increased from 3.8 to 10.9 million or from 9 to 21 percent of the population in this age range.[footnoteRef:3] ELs are students with a primary language other than English who have a limited range of speaking, reading, writing, and listening skills in English. This might include students who have been identified and determined by their school as having limited English proficiency (LEP). According to the National Assessment of Educational Progress, fourth grade ELs continue to score significantly lower on reading and math achievement compared to their non-EL counterparts, despite a decline in the size of gap between 1998 and 2005. Therefore, closing the achievement gap for ELs and ensuring they are prepared for school remains to be a central focus across the Department. [1: National Evaluation of Title III Implementation: Report on State and Local Implementation. ] [2: U.S. Department of Education, National Center for Education Statistics. (2014). The Condition of Education 2014 (NCES 2014-083), English Language Learners.] [3: U.S. Department of Education, National Center for Education Statistics. (2010). The Condition of Education 2010 (NCES 2010-028). ]

While Spanish is the native language of the majority of ELs, overall, ELs speak more than 150 languages.[footnoteRef:4] This presents challenges in the classroom as students of various language backgrounds and abilities are faced with the need to master content without the benefit of proficiency in English. It also presents challenges for teachers, many of whom may not be adequately prepared to provide instruction appropriate to and effective for ELs. The need for guidance on instructional practices for ELs has triggered IES funding to test the efficacy of various academic language interventions. For the purpose of this Performance Work Statement (PWS), the term “academic language interventions” refers to interventions focused on improving comprehension processes and language and vocabulary development designed to improve the achievement outcomes for ELs. Academic vocabulary interventions are a specific type of academic language interventions that focus specifically on vocabulary development. Both broader academic language and academic vocabulary interventions are of interest for this contract. [4: http://www.migrationinformation.org/ellinfo/FactSheet_ELL3.pdf ]

To understand the depth of efficacy research on academic language interventions and to identify practices that educators can use, IES assembled a panel of experts to develop a Practice Guide focused on Teaching Academic Content and Literacy to English Learners in Elementary and Middle School (http://ies.ed.gov/ncee/wwc/PracticeGuide.aspx?sid=19). Released in April 2014, this Practice Guide provides four recommendations on instructional practices that are supported by evidence for ELs during reading and content area instruction. Among the four recommendations, one of the most prominent recommendations from the guide is to “Teach a set of academic vocabulary words intensively across several days using a variety of instructional activities.” Teaching academic vocabulary is a technique strongly supported by high quality research, as there are currently six studies that meet What Works Clearinghouse evidence standards and find positive impacts across domains including vocabulary, English language, and reading. Three of these six studies directly tested intensive academic vocabulary instruction as specified in the recommendation and found that it is beneficial to provide intensive instruction on a few select words across several days using a variety of instructional activities. All of the studies were conducted in settings integrating ELs with native English speakers. Taken together, these studies constitute an initial base of efficacy work that can inform evaluation work that tests the impact of academic vocabulary and language interventions.

Key Research Questions

Recent emerging research indicates that academic language interventions are effective at improving a range of outcomes for ELs. Given the success of initial efficacy trials in this area, there are a number of questions that large-scale effectiveness research can begin to address.

· Which academic language interventions are effective at improving outcomes for English learners when taken to scale?

· What are the program impacts on achievement outcomes for English learners? What are the program impacts on achievement outcomes for disadvantaged non-EL students in the same classrooms?

· Is there variation in program impacts?

· Do impacts vary depending on the setting in which the program is implemented?

· Do impacts differ for certain subgroups (e.g., gender, age, and ethnicity)?

· Are program impacts replicated across the sites implementing a particular program model?

· What aspects of program implementation are associated with program impacts? Specifically,

· How effective were sites in maintaining fidelity to the program model?

· What factors were associated with the ability to maintain fidelity?

· To what extent was fidelity to the model associated with program impact?

· What other factors may have influenced program impact (e.g., program strategies, organizational, training supports, and capacity)?

· How did sites respond to implementation challenges?

Evaluation Approach

The evaluation shall examine the impact of various academic language programs at scale in multiple school settings and across districts, with a close examination of the training provided. There are various approaches to the academic language programs that are commercially available. For example, some require the independent selection of a trainer or teacher, and others require that a trainer from the program should train educators. Considering an array of possibilities, the contractor shall develop a solicitation for academic language intervention developers and trainers that specifies that up to three different approaches academic language may be selected for the evaluation. The goal is to test the effectiveness of each of the approaches to see which interventions are effective when operating in multiple different school settings at once.

Academic Language Intervention Selection The contractor shall be responsible for selecting the three academic language interventions to be evaluated in this study. After the contract award, the contractor shall establish a group of five experts in academic vocabulary instruction (approved by the Department). In consultation with the TWG and the Department, the contractor shall develop an open solicitation and selection criteria. Then, the contractor shall publish the solicitation and contact developers to ascertain their interest and ability to participate in a randomized trial of their academic vocabulary intervention. In response to the solicitation, interested providers will submit a description of their program and the requirements for school staff training and implementation, along with any evidence of effectiveness. The contractor shall oversee the TWG’s review of proposals in order to identify up to four providers for inclusion in meetings in Washington, DC. After consideration of the TWG’s recommendations, the contractor (in consultation with the COR) shall select up to three academic language programs for inclusion in the study.

The contractor shall monitor the implementation process and facilitate, as needed, implementation between the training providers and the districts and schools during the evaluation.

Impact Study Design The purpose of the Department’s evaluation is to study the impact of academic language interventions on student academic achievement outcomes. The contractor, with assistance and input from the COR and in consultation with a TWG (Task 4), will select up to three academic language interventions or program models to evaluate at scale. This PWS assumes the following:

The contractor shall, through subcontracts with academic language intervention providers, implement up to three academic language interventions in upper elementary and middle schools with high proportions of ELs[footnoteRef:5]. The study shall recruit and complete data collection from a sample of upper elementary and middle schools and perform random assignment at the school level. The contractor shall consider the feasibility of all classrooms either receiving the same treatment or control condition in one school; or whether an AB design is more appropriate[footnoteRef:6]. The contractor shall develop study design options to randomly assign schools to one of up to three academic language interventions or a business-as-usual classroom. The study should be powered for an effect of 0.15 standard deviations on upper elementary or middle school student achievement. The design options shall specify the exact parameters and define the most appropriate power estimates for the study. [5: While the study will include non-EL disadvantaged students as well as EL students, the Department would like this study to be eligible for review under the What Works Clearinghouse English Language Learners topic area. See the evidence review protocol for interventions for English Language Learners for sample eligibility criteria: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_ell_protocol_v2.2.pdf.] [6: Under this design, schools are randomly assigned to two groups: one in which one set of schools selects different grades to participate in the academic vocabulary intervention (for example, in one school 4th grade gets treatment and 5th grade gets the control and in another school, 4th grade gets the control and 5th grade gets the treatment).]

Data collection shall consist of the collection of student records data, implementation data (e.g., fidelity measures, program goals and objectives, program components, organizational capacities, information about the counterfactual condition, etc.). These data collection efforts shall occur prior to the randomization to conditions and one to two times after the implementation of the program, depending on the given program’s theory of action).

Specific Design Issues

Discussed below are a number of factors that will affect the evaluation design or successful execution of the study. In the study design, the contractor shall discuss, at a minimum, how each of these challenges will be addressed and shall also highlight other key challenges and design considerations. In addition, design issues may arise as the evaluation is carried out that will need to be resolved by the study team.

· Random Assignment of Schools. The contractor shall address the desirability of random assignment by school. Exactly how will the contractor carry it out, and how will any potential drawbacks to this level of random assignment be addressed? The contractor may consider random assignment designs that are more efficient, cost effective, and allow for more treatment conditions, such as AB designs.

· Challenging Nature of Experimental and Long-term Research with ELs. Prominent researchers who have studied EL populations have noted the challenges inherent with studying this population of students. Some of these challenges include the following:

· EL Designation and Status. First, the designation of students as ELs is simultaneously dynamic and developmental as students are constantly acquiring greater English proficiency and their status as ELs changes over time (Francis & Vaughn, 2010). Therefore, it can be difficult to identify ELs as they transition from elementary to middle school.

· Elementary School ELs vs Middle School ELs. The contractor shall consider the pros and cons of conducting the study in elementary grades compared to middle grades. For example, while interventions in the elementary grades may be able to have a greater impact on EL status, before English language is acquired, the initial evidence on efficacy trials for academic language interventions has been on students in grades 4 through 7.

· Specific Outcomes to Collect. The primary outcome of interest is English language acquisition; however, other data shall be collected on other outcomes as necessary. For example, if the selected interventions operate through academic vocabulary instruction in science or social studies classes, extant data related to student record data in those areas will need to be considered and collected. In addition, the contractor shall consider the utility of collecting outcome data based on measures identified in the literature around academic language experimental evaluations. Specifically, it may be the case that standardized assessments may not tap into key constructs relevant for this study (e.g., specific components of vocabulary comprehension and development). The contractor shall consider the cost of developing and administering these measures as appropriate.

· Timing of Data Collection. Many of the interventions that have been tested through efficacy trials rely on an immediate post-test at the close of the intervention, while others collect outcome data twelve weeks to 6 months after the intervention. The contractor shall consider the theory of action behind programs and whether short-and longer-term data collection is necessary and the key variables of interest. The contractor shall be sure to identify these ahead of time to ensure that accurate baseline data is collected on potential outcomes of interest.

· Clustering of ELs within Schools. According to recent data, EL students are generally clustered within selected schools within a given school district, rather than distributed across schools. The selected schools are often affected by factors such as low socioeconomic status, high mobility, and less qualified teachers. Combined, these factors can threaten the implementation of academic language interventions. The contractor shall identify key ways to mitigate potential threats to program implementation. At the same time, the contractor shall consider strategies to obtain a study sample that is relevant to federal education policy and can inform decisions about academic language interventions for both ELs and disadvantaged non-EL students. The contractor shall fully explain any assumptions made.

· Varied EL Composition within Classrooms. While ELs can be clustered within the same schools, EL students are often located in regular classrooms with non-EL students. The contractor shall examine classroom composition to ensure that the academic language interventions can be implemented in classrooms in which at least 50 percent of the students are ELs[footnoteRef:7]. [7: While the study will include non-EL disadvantaged students as well as EL students, the Department would like this study to be eligible for review under the What Works Clearinghouse English Language Learners topic area. See the evidence review protocol for interventions for English Language Learners for sample eligibility criteria: http://ies.ed.gov/ncee/wwc/pdf/reference_resources/wwc_ell_protocol_v2.2.pdf.]

· Academic Language Intervention Training and Implementation. The Department understands that there are various approaches to academic language instruction that have shown positive impacts on academic achievement through rigorous, experimental studies. With the consultation of experts, the contractor shall identify key variations across academic language programs that are offered by different providers, as well as the most useful contrasts between/among different approaches.

· Experience in the Counterfactual. Schools are anticipated to be randomly assigned to academic language interventions or business as usual. A key issue is the nature of other academic language instruction experienced by the control schools, the counterfactual for those not assigned to the treatment/intervention. The Department expects full documentation on the counterfactual to develop and report on evaluation findings that clearly identify “typical” classroom contexts that occur for EL students. The contractor shall discuss the instructional contrast likely to be found and power the study accordingly. This PWS assumes the study will be powered to detect an effect of at least 0.15 standard deviations on school student achievement.

· Identifying Appropriate Pretest Covariates. Previous experimental research with ELs suggests that it can be difficult to assess treatment effects on groups that are highly stratified on key covariates, such as EL status. The contractor shall address this issue to ensure that baseline data is collected on appropriate covariates necessary for the study to meet What Works Clearinghouse (WWC) Standards. Key relevant WWC protocols for reviewing studies that contain ELs include the English Learners Protocol (http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=22) and the Teaching Academic Content and Language to English Learners protocol (http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=248).

I. Scope of Work

This section provides a brief overview of the key substantive tasks of this contract. The work under this contract will be undertaken through a 30-month base period and four option periods that reflect the annual appropriation of research funds. The option periods will overlap in time and include similar tasks but each will produce an independent product of value to the government.

The Contracting Officer’s Representative (COR) reserves the right to inspect the contractor’s work while in progress. The contractor shall submit to the COR any progress report or other written documentation of work to date that the COR requests to facilitate that inspection, in accordance with FAR 52.246-4 (Inspection of Services- Fixed Price) and FAR 52.246-5 (Inspection of Services- cost-reimbursement).

BASE PERIOD

The Base Period extends from September 2015 through March 2018. Central tasks during this period include, but are not limited to, finalizing the study design, selecting interventions for the evaluation, and training schools in the selected academic language interventions prior to the start of the 2017-18 academic year.

Task 1 Project Management (pricing for each subtask indicated below)

The contractor shall manage tasks in an efficient manner that fosters communication with the COR.

Subtask 1.1 Kickoff Meeting (firm-fixed-price) The contractor’s Project Director and Key Project Staff (up to three) shall meet with the Contracting Officer’s Representative (COR) and other appropriate Department staff within two weeks following the effective date of the contract (EDOC). During this Kickoff meeting, Department staff and the contractor shall discuss details regarding the anticipated schedule of activities for the coming year and other issues related to the contract. The meeting shall be at IES’s offices in Washington, D.C. The contractor shall provide an agenda for the meeting within five business days after the EDOC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within two weeks following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting and how each will be addressed.

Deliverables:

· Kickoff Meeting Agenda

· Kickoff Meeting Summary

Subtask 1.2 Regular Conference Calls (firm-fixed-price) The contractor shall participate in a call with the COR once every two weeks. The contractor’s project director shall submit an agenda for each call to the COR (including the status of each of the tasks under the contract) at least one day prior to the call. Outstanding issues or actions requiring follow up by either the COR or the contractor’s project director will be discussed in as-needed conference calls between the contractor’s project director and the COR. The contractor shall create a call summary of discussions that arose during the call and submit the summary to the COR within three days following each call.

Deliverables:

· Call Agenda

· Call Summaries

Subtask 1.3 Periodic in-person Meetings (cost-reimbursement) The project director and up to two additional key staff shall meet with the COR and other appropriate NCEE staff in Washington, D.C. on an as needed basis, in person or by conference call, not to exceed four times per year. The purpose of the meeting is to brief NCEE on the progress of the contract and to discuss issues as they arise. The contractor shall prepare an agenda one week prior to the meeting. Within two weeks following each meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed.

Deliverable:

· Agenda

· Memo summarizing key issues

Subtask 1.4 Develop Work Plan (firm-fixed-price) Based on the information obtained during the Kickoff meeting and discussions with the COR, the contractor shall develop a work plan to guide the tasks to be conducted during the base period. The work plan must, at a minimum, include the steps to be taken to complete the tasks and the subtasks described in this PWS.

Within four weeks of the EDOC, the Contractor shall submit to the COR for comment a draft work plan that incorporates any decisions made during the kickoff meeting. The work plan shall identify potential challenges that may be encountered in carrying out the tasks and provide recommended approaches for addressing them. Two weeks after receiving comments from the COR, the contractor shall submit a revised work plan.

Deliverables:

1. Draft work plan

1. Revised work plan

Subtask 1.5 Develop Communication Plan for Describing the Study (cost-reimbursement) This study will require frequent communication with a wide range of interested parties. The contractor shall propose a plan and mechanisms for on-going communication with key audiences about the technical aspects of this study such as the structure and implementation of the study, as well as the impact and implementation findings. The contractor shall develop content that IES will publish on the existing IES website to provide information to the public on the progress of the academic language evaluation. The contractor shall develop a brief project description for use by the Department that provides an overview of the study and communicates the technical aspects of this study such as the structure and implementation of the study, as well as the impact and implementation findings. Within one month after the EDOC, the contractor shall prepare a draft of a short 2-3 page evaluation profile including study purposes, methods, and timelines suitable for broad distribution and posting to the existing IES website. In addition to the profile, the contractor shall develop accompanying materials to communicate the evaluation, incorporating media such as Shutterstock images, which can be supplied through IES. The contractor shall be prepared to discuss how this evaluation fits into the broader context of research on academic language interventions and instruction for ELs. The contractor shall revise the project summary and submit the revised version within 2 weeks of receiving the COR’s comments.

The contractor shall submit the draft communication plan to the COR within 2 weeks of receiving approval from the Department on the study design in task 3. Based on COR comments and the result of the academic language developer competition in task 5, the contractor shall submit a revised communication plan to the COR within 2 weeks of selecting the interventions under task 5.

Deliverable:

· Draft project summary

· Revised project summary

· Draft communication plan

· Revised communication plan

Task 2 Review of the Evidence Supporting Academic Language Interventions (cost-reimbursement)

The contractor shall review the evidence supporting academic language interventions to help specify design options for the evaluation. The contractor shall identify up to 10 academic language interventions that will inform design options for the study, primarily using programs identified in the 2014 IES Practice Guide as a starting point (see Teaching Academic Content and Literacy to English Learners in Elementary and Middle School (http://ies.ed.gov/ncee/wwc/PracticeGuide.aspx?sid=19). The contractor shall also (1) scan the field for other programs with emerging evidence that are both widely-used and replicable and (2) consider the list of National Institute of Education Research sponsored evaluations of key efficacy trials in this area, which will be provided at the kickoff meeting. Studies of academic language interventions that have not been examined in the Teaching Academic Content and Literacy to English Learners in Elementary and Middle School Practice Guide shall be coded by WWC-certified reviewers to determine if they are supported by a design that meets WWC evidence standards with or without reservations and whether there are positive impacts. The contractor shall ensure that these studies have not been reviewed by the WWC before beginning review and shall submit the reviews as external content to the WWC. These candidate interventions will be the target of the solicitation for academic language programs that will participate in the evaluation.

The contractor shall submit the draft list of academic language interventions for further exploration to the COR within 3 weeks of the EDOC. The contractor shall submit the list of interventions that need to undergo WWC review to the COR within 4 weeks of the EDOC. The contractor shall submit the results of the evidence review to the COR within 9 weeks of the EDOC. The final list of potential interventions shall be submitted to the Department within 10 weeks of the EDOC.

Deliverables:

· Draft list of academic language interventions for further exploration

· List of interventions that need to undergo WWC review

· Final review of studies

· List of up to 10 academic language interventions that can inform evaluation design options

Task 3 Develop an Evaluation Design with Multiple Options (cost-reimbursement)

The contractor shall develop a design report for an impact evaluation of academic language interventions. The contractor shall address each of the research questions as well as issues raised after consulting with a Technical Working Group (described in Task 4) in the design report. The contractor shall also discuss the sampling plan, plans for training and implementation of interventions, the data collection plan, and the preliminary analysis plan. The design report shall provide multiple options for conducting an evaluation with the pros and cons of each option. The contractor shall submit a first draft design report by February 1, 2016. This draft will be the focus of the mini-TWG meeting held under Task 4. Based on the discussion at the mini-TWG meeting and consultation with the COR, the contractor shall revise the design report and submit a second draft design report by August 2016 or within 11 months of the EDOC, whichever is earlier. The COR will review the draft report within three weeks. By January 2017 or within 16 months of the EDOC, whichever is earlier, the contractor shall submit a final design options report that will incorporate the edits specified by the COR.

The contractor shall, in consultation with the COR, develop a strategy for selecting study sites and prioritize potential sites based on criteria that the COR and contractor agree are pertinent to addressing the evaluation objectives. These shall include policy-relevant districts with capabilities sufficient to support the evaluation’s needs and a willingness to participate in the academic language program training and to implement the programs.

The design options report shall also specify the appropriate mechanism for collecting the following types of data:

· Student characteristics. These shall include data on student background, achievement across language-related domains in particular, and across subjects, English language proficiency, English language status, attendance, special education eligibility, and educational placement, as well as other relevant information. These data shall be collected in the spring prior to the implementation of the academic language interventions.

· Student testing data in relevant grades. If deemed necessary by the COR, in consultation with IES leadership and the TWG, the contractor shall individually-administer an adaptive student achievement test for students in the study sample appropriate across the three academic language interventions and consistent with the minimum detectable effect size in the study design. The contractor shall identify tests in language comprehension and vocabulary development, as well as across subjects that are valid and reliable for the purposes of this evaluation with guidance from the IES and the TWG. The contractor shall administer the test according to the theory of action of each program, which could range from immediate post-test to one year after implementation.

· Student testing data. The contractor shall identify appropriate sources and collect extant assessment data for all students in the study sample. The contractor shall identify tests in language comprehension, reading, and vocabulary development that are valid and reliable for the purposes of this evaluation with guidance from the TWG and IES. The contractor shall collect these forms consistent with achieving appropriate effect size. The contractor shall collect baseline data between Fall 2016 and Spring 2017. The contractor shall also consider student testing data that aligns with the theory of action of the program.

· Fidelity of implementation/Documentation of counterfactual experience. The contractor shall obtain data on the provision of training and training dosage, training content and implementation, as well as data to understand the fidelity of implementation and the counterfactual experience.

· School characteristics. The contractor shall obtain data on school staff characteristics, previous academic language intervention experience, current training and implementation, and school climate.

Deliverables:

· 1st Draft design report

· 2nd Draft design report

· Final design report

Task 4 Technical Working Group for the Evaluation (cost-reimbursement)

Subtask 4.1 Establish Technical Working Group (TWG) The contractor shall establish a TWG of experts early in the contract to provide guidance throughout the study. The role of the TWG will be to advise the contractor and the Department on the conduct of the study including, but not limited to: the academic language interventions chosen as candidates for the evaluation, the overall study design, instrumentation, data collection and analysis, and draft study reports. The TWG shall be comprised of at least 8, and no more than 10 individuals, and should include experts in academic language interventions, implementation, as well as experts in evaluation methodology. The contractor shall submit to the COR a list of proposed TWG members at the Kickoff Meeting (Task 1). When preparing the list, the contractor shall evaluate the potential for conflicts of interest by assessing whether each potential TWG member has a bias or financial interest in the outcome of the study. After the COR approves the proposed TWG members, the contractor shall contact each proposed member and, within four weeks of COR approval, submit letters of commitment for the duration of the evaluation, as well as documentation of consulting rates for the proposed TWG members to the COR and Contracting Specialist (CS) for approval by the CO (Contracting Officer).

Deliverables:

· List of proposed TWG members

· Letters of commitment and consulting rates

Subtask 4.2 Convene a “Mini Technical Working Group” The contractor shall convene 1 “mini-TWG” meeting during the base period. The meeting shall be held no later than March 2016 and shall include no more than 4 of the TWG members. This “mini-TWG” shall be responsible for reviewing the scope of the initial evaluation design and weighing in on the list of candidate interventions that will be the focus of the evaluation. Specifically, this group will review the scope of the evaluation questions and help decide key design options (e.g., whether the evaluation will focus on head-to-head comparisons of academic language programs or comparisons of programs to business as usual classroom practices, etc.). The contractor shall submit a list of proposed members (from the list of TWG members approved in subtask 4.1) to the COR within 2 weeks of the effective date of the contract. The COR will review and approve the list of proposed mini TWG members within one week of receiving the list. The contractor shall contact each proposed member and obtain and submit letters of commitment and documentation of consulting rates to the COR within 2 weeks of COR approval of the proposed list. The contractor shall use its conflict of interest policy to have this TWG openly disclose any situations where there could be a perceived conflict of interest with the academic language interventions being selected for the evaluation.

Contract dollars may not be used to pay for food. However, this meeting shall include a working lunch whereby the contractor informs participants that they are expected to either bring their own lunch or to order food and pay for it themselves. With regard to this latter option, the contractor shall identify local eateries with delivery services, provide menus to meeting participants, collect participants’ food orders, arrange for delivery of food, and collect money from participants for their food delivery. This will enable a working lunch, but participants will pay for food themselves. No later than 4 weeks before the meeting, the contractor shall submit to the COR draft materials for the meeting, including an agenda, presentation materials, and other materials for the TWG’s review. The COR will review and comment within 3 days. Final materials shall be fully responsive to COR comments on the draft materials. After COR approval, and no later than 1 week before the meeting, the contractor shall send the approved materials to the TWG members.

After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S. Department of Education facilities in Washington, DC. The contractor shall pay the travel, lodging, per diem, and consulting rate (as negotiated in subtask 4.1) expenses of the TWG members.

In addition, the “mini-TWG” will help plan for the request for proposals (RFP) that will identify developers to participate in the evaluation, as described in Task 5. As part of this work, the contractor shall request that each member of the “mini-TWG” review the contractor’s draft of the academic language RFP and the criteria by which the proposals shall be evaluated. The contractor shall request that TWG members provide recommended changes within 2 weeks of receiving the draft RFP with evaluation criteria.

Deliverables:

· Proposed “mini-TWG” members

· Final “mini” TWG members

· Summary of “mini” TWG meeting

· “Mini TWG” review of RFP and criteria

Subtask 4.3 Convene a Full Technical Working Group The contractor shall convene a one-day meeting of the entire TWG no later than November 2016. The TWG shall focus on remaining design issues, data collection, and analysis issues. Primarily, the TWG will review the 2nd draft design options, which are due to IES by August 2016. Contract dollars may not be used to pay for food. However, this meeting shall include a working lunch whereby the contractor informs participants that they are expected to either bring their own lunch or to order food and pay for it themselves. With regard to this latter option, the contractor shall identify local eateries with delivery services, provide menus to meeting participants, collect participants’ food orders, arrange for delivery of food, and collect money from participants for their food delivery. This will enable a working lunch, but participants will pay for food themselves. No later than 4 weeks before the meeting, the contractor shall submit to the COR draft materials for the meeting, including an agenda, presentation materials, and other materials for the TWG’s review. The COR will review and comment within 1 week. The contractor shall make revisions in response to COR comments and submit final materials to the COR no later than 2 weeks before the meeting. Final materials shall be fully responsive to COR comments on the draft materials. After COR and IES approval, and no later than 1 week before the meeting, the contractor shall send the approved materials to the TWG members.

After the meeting, the contractor shall submit a written summary of the meeting to the COR within one week, detailing suggestions and recommendations along with the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR. The meeting site shall be arranged to minimize travel of Federal government staff members, with preference given to meeting in U. S. Department of Education facilities in Washington, DC. The contractor shall pay the travel, lodging, per diem, and consulting rate (as negotiated in subtask 4.1) expenses of the TWG members.

The contractor shall provide copies of all reviews of products by the TWG to the COR within five work days of receipt by the contractor. If TWG members recommend any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR. The contractor also shall use the TWG members to review products, as appropriate, when group meetings are not required.

Deliverables:

· Draft meeting materials

· Final meeting materials

· Summary of TWG meeting

Task 5 Academic Language Intervention Request for Proposals & Developer Selection (cost-reimbursement)

Subtask 5.1 Advance Announcement The contractor shall prepare a draft advance announcement that proposals will be requested to train schools in the implementation of academic language interventions and the expected timelines for the proposal process. The contractor shall submit the draft announcement to the COR for review within 5 weeks of EDOC. The COR will review within one week after receipt. The contractor shall send the revised advance announcement to academic language training providers, post it on the website(s) of relevant associations, and submit the advance announcement to IES for posting on the study website within 9 weeks of EDOC.

Deliverables:

· Draft announcement

· Revised announcement

Subtask 5.2 Request for Proposals Based on the contractor’s proposal and in consultation with the COR, the contractor shall prepare a draft of the academic language developer request for proposals (RFP) including the timelines for proposals, requirements for the training, and the criteria by which the proposals shall be evaluated. The contractor shall submit a schedule for the solicitation to the COR by January 2016. In the schedule, the contractor shall include a pre-proposal request for intent to bid letters to estimate the potential burden for reviewing proposals.

Within the RFP, the contractor shall include criteria that address the need for proposals from developers of academic language interventions that have studies demonstrating evidence of effectiveness and that meet WWC standards with or without reservations. The RFP shall also specify the maximum amount of funding that will be available to support the training. The contractor shall submit the draft request for proposals to the COR for review no later than February 2016. The COR will review within one week of receiving the draft request for proposals. The contractor shall make revisions and send the request for proposals to the members of the mini TWG formed in Task 4. for review within one week of COR comment. The contractor shall make any necessary revisions within one week of receiving comments from the mini-TWG and submit the final request for proposals to the COR by April 2016. After COR approval, the contractor shall submit the RFP for posting and widely disseminate information about the competition for academic language training providers suitable for this evaluation effort. The contractor shall submit the RFP and evaluation criteria to IES for posting on the study website.

Deliverables:

· Schedule for the solicitation

· Draft RFP

· Final RFP

Subtask 5.3 Review of Proposals and Selection of Interventions Within one week of closing the request for proposals, the contractor shall send proposals to the mini TWG members for review. The contractor shall compile the TWG members’ reviews of proposals and conduct teleconferences with them as necessary in order to develop a ranking of developer proposals for inclusion in the meetings in DC within one week of receiving reviews. The contractor shall arrange a meeting with members of the mini TWG and developers of each of up to four providers recommended by the mini TWG. The contractor shall evaluate the potential for conflicts of interest by assessing whether the mini TWG members have a bias or financial interest in the outcome of the study once the potential academic language developers/interventions are known (the contractor shall replace any mini-TWG members who have a COI with any of the academic language developers). The meeting shall be one or two days and shall include time for each academic language developer to present their training plan and answer any questions from the mini TWG, contractor, and/or the Department. The contractor shall plan to arrange and pay for travel, lodging, and per diem for up to two staff for each academic language developer to travel to DC, present on their training plans, and answer questions that the mini TWG, the contractor, or the Department may have. The contractor shall pay the travel, lodging, per diem, and consulting rate (as negotiated in subtask 4.1) expenses of the mini TWG members.

The contractor shall compile TWG members’ reviews of the presentations/proposals, conduct teleconferences with the TWG reviewers as necessary, and consult with the Department in order to select academic language developers for the evaluation and submit to the COR a written summary of the reviews and recommendation for awards within one week of receiving the mini TWG’s reviews. The contractor shall negotiate with and secure the full cooperation of the recommended academic language developers. The contractor shall submit subcontracts with the selected academic language developers by July 2016 or within 10 months of the EDOC, whichever is earlier.

Deliverable:

· Written summary of the mini TWG’s reviews and recommendations for award

· Contracts with academic language developers Task 6 Prepare OMB Package and Develop Data Collection Forms (cost- reimbursement)

The contractor shall prepare all relevant forms for an OMB package that includes the data collection instruments and methods.

The projected timeline to guide data collection for this evaluation is as follows:

· Academic language intervention implementation year: 2017-18

· Baseline data year: 2016-17 (where baseline data would be available and collected in the fall of the 2017 school year)

· First follow-up: Starting in Spring 2018

· Longer-term follow up data collection, pending promising results from first follow-up: Spring 2019 (where the follow up time periods will differ depending on the developer’s theory of action)

The contractor shall prepare all relevant forms and justification for a forms clearance package that includes all data collection instruments. While the preparation of the data collection forms occurs as part of the base contract, data collection using the forms will only occur if one or more of the Option Periods including data collection (Option Periods I, II, and III) are executed. The package shall include a description of the study design and data collection instruments, as well as the necessary forms required for OMB approval. The contractor shall ensure that the clearance package justifies the data collection activities and comprehensively responds to each required item in the OMB instructions.