IDEA IDIQ PWS 12.6.21 LA.docx

U.S. Department of Education Performance Work Statement Institute of Education Sciences National Center for Education Evaluation

Performance Work Statement

EVALUATING PROGRAMS AND STRATEGIES TO IMPROVE THE OUTCOMES OF STUDENTS WITH DISABILITIES

I. INTRODUCTION

The U.S. Department of Education’s (ED) Institute of Education Sciences (IES) requires one or more contractors to assist IES in conducting evaluations of programs and strategies designed to improve the outcomes of children with disabilities. These studies may use existing data sources and/or new data collection to : (1) describe the implementation and costs of federally-funded programs and strategies that support children with disabilities; (2) describe the needs of children with disabilities, the services that they receive, and their outcomes; (3) identify and describe the core components of important programs and strategies, including the conditions necessary to achieve impact, in particular, for key student populations including those with particular needs; (4) analyze trends in outcomes and potential impacts of federal grants to children with disabilities; (5) examine the effectiveness and cost effectiveness of important programs and strategies that support children with disabilities, including demonstration or low-cost opportunistic impact evaluations; (6) conduct systematic evidence reviews to support potential or planned evaluations. The program office responsible for the evaluations will be the National Center for Education Evaluation and Regional Assistance within IES.

A. Legislative Authority

The latest 2004 reauthorization of the Individuals with Disabilities Education Act (IDEA) provides funding for, and directs IES to, carry out, studies to determine the effectiveness of IDEA in achieving its purposes, develop information on how IDEA could be implemented more effectively, and provide information useful in developing more effective legislation.[footnoteRef:2] [2: See Sec. 664 of Public Law 108-446: H.R.1350 - 108th Congress (2003-2004): Individuals with Disabilities Education Improvement Act of 2004 | Congress.gov | Library of Congress ]

B. Background

Federal policy has long played a key role in the education of the more than 1 in every 10 US children who are identified with a disability, and in efforts to improve their outcomes. Students with disabilities continue to trail their peers in rates of high school graduation (66% vs. to 84%), enrollment in postsecondary education (61% vs. 84%), and later employment (35% vs. 78%), perhaps stemming from gaps in achievement that begin earlier[footnoteRef:3]. The Coronavirus pandemic only heightened attention to these gaps, as students with disabilities typically need specialized instruction and related services and supports that are challenging to provide in the context of remote and hybrid learning. Perhaps now more than ever, it is critical to understand the needs of children with disabilities, the policies and practices carried out to support them, and the programs and strategies that are effective in boosting their academic and longer-term prospects for success. [3: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), Various years, 1996–2017 Mathematics and Reading Assessments.]

II. OBJECTIVES

This contract will be an Indefinite Delivery-Indefinite Quantity (IDIQ) vehicle. The IDIQ structure is intended to help IES design and conduct evaluations of programs and strategies to accelerate the learning of children with disabilities rigorously, quickly, and efficiently in order to make information more actionable for policymakers and practitioners. By strategically awarding task orders to one or more contractors with appropriate content area, data collection, and methodological expertise, IES expects to reduce the time and costs involved in starting up separate studies and to better link across them. Specifically, through this IDIQ IES seeks the following:

· Access to one or more contractors that each possesses wide-ranging knowledge of children with disabilities, their needs, and related policies (federal and local), programs, and strategies intended to improve their outcomes. Examples of such strategies include, but are not limited to, those that focus on providing access to instruction in the general education classroom, supports for social-emotional learning, use of accommodations and related services, use of technology, and supports for family engagement. To be able to respond to emerging evaluation needs quickly, the IDIQ must have experts already familiar with these likely policies, programs, and approaches.

· The ability to tap into broad expertise in evaluation methods, including cutting-edge statistical approaches. Tasks under this IDIQ might require complex sampling for nationally-representative surveys, rigorous analytic strategies for random assignment or regression discontinuity impact evaluations, or predictive analytics and other approaches to using data in innovative ways. A contractor or team with this full set of skills is important because IES may want to “nest” or interconnect studies to increase their relevance – for example, evaluating the effectiveness of a particular strategy in the context of how commonly that strategy is used across the country, based on survey data collected.

· Access to a contractor or team with the capacity to conduct multiple large-scale evaluations at the same time, and to start up evaluation activities within a month of task order awards. This capacity is important in order to meet policymaker needs or successfully interconnect studies.

· Access to a contractor or team with skills in communicating to a wide range of audiences through a variety of different evaluation products. This includes not only IES’ new report format (see, for example, https://ies.ed.gov/ncee/pubs/2021005/), but its shorter data-oriented Snapshots (for example, https://ies.ed.gov/ncee/pubs/2021003/), “how-to” guides when evaluations show strategies are effective, and other products that extend the return-on-investment in the studies.

The IDIQ shall have an ordering period of 60 months. All task orders shall begin within the 60-month ordering period, but may have a duration that extends for up to 60 months beyond the end of the ordering period. Task orders may be issued as Firm Fixed Price, Time and Materials, Cost Reimbursement, or Cost Plus Fixed Fee contracts. Each task order will include one or more tasks related to designing and implementing an evaluation study. The scope of each task order will depend on IES’ needs, including which activities are appropriate for each desired study or stage of the study. For example, some task orders may include only design work, while other task orders may include external data collection activities.

III. SCOPE OF WORK

The scope of work performed under this contract shall be defined by the following tasks and/or subtasks. Each task order specified by IES shall include one or more of the tasks or subtasks, combined in various ways, to accomplish needed evaluation studies.

Task 1—Meetings

1.1 Initial Meeting. The contractor’s project director and two other key project staff members shall meet with the Contracting Officer (CO), Contracting Specialist (CS), the IES Contracting Officer’s Representative (COR) for the IDIQ/task order, and other appropriate ED staff at the start of the IDIQ/task order to discuss details regarding the tasks, scheduling activities, and other issues related to the IDIQ or a particular task order. If public health conditions allow, the meeting will be held in ED’s office in Washington, DC. Otherwise, the meeting will be held virtually via video conference. All video conferences under this IDIQ shall be held via Microsoft Teams, WebEx, or Zoom. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to them.

1.2 Additional Meetings. The contractor shall meet with the COR in Washington, DC (as public health conditions allow; otherwise, meeting to be held virtually via video conference) to discuss the progress of contract activities, including all current task orders, and any issues or anticipated problems and proposed solutions to those problems.

1.3 Study Briefing. The contractor shall prepare for and participate in a briefing for relevant stakeholders, such as ED program staff, to summarize interim or final results from specific evaluations conducted under a task order. The COR shall specify whether the briefing shall be in person in Washington, DC or conducted virtually via video conference.

Task 2—Technical Working Group (TWG)

2.1 Establish TWG. The role of the TWG will be to advise the contractor on an individual task order or collection of task orders that address a similar topic. The TWG advises by providing feedback and suggestions to the contractor, typically as part of meetings where the contractor presents evaluation plans or results and seeks TWG input in real time, or by providing written feedback after a review of materials sent by the contractor. Each task order awarded under this IDIQ will specify whether a new TWG will be established, for example, if new types of expertise are needed, or a previously established TWG can be used to provide ongoing advice on the new work.

The contractor(s) shall propose to the COR a set of primary and alternate members for each TWG panel. Each panel shall be comprised of eight to ten individuals that, together, have expertise as specified in a task order. The contractor shall evaluate the potential for conflicts of interest by assessing whether any individual considered for inclusion on the TWG has a bias or financial interest in the outcome of the study or study tasks. These conflicts of interest can be perceived or actual, and are described in a form that IES provides to the contractor. The contractor shall have TWG members complete conflict of interest forms before appointment to the TWG as consultants.

2.2 Convene full TWG Meeting. Full-day TWG meetings shall be held either in person in Washington, DC, or virtually via video conference, as specified in a task order. The first meeting of a TWG typically involves discussion of issues related to the design of the study or study tasks. Subsequent TWG meetings may cover different topics, such as methodological and policy reactions to preliminary study findings, as specified in a task order.

2.3 Convene mini TWG meeting. The contractor shall convene half-day, video conference meetings of a TWG, including all or a subset of TWG members. These shorter meetings are typically to discuss a narrow issue related to a task order that requires more immediate and targeted input, but may also be an opportunity to update the TWG on new activities. The date for the meeting and TWG members to be invited shall be set in consultation with the COR or specified in a task order.

Task 3— Feasibility and Design

The contractor shall undertake activities to assist IES in determining the key parameters of an evaluation of policies, programs, and strategies to accelerate the learning of children with disabilities, that might be carried out under one or more task orders.

3.1 Feasibility Assessment for Effectiveness or Outcomes Studies. As specified in a particular task order, the contractor, or the contractor in consultation with the COR, shall identify one or more policy-relevant programs and strategies, and assess the feasibility of examining the strategies through a random assignment, regression discontinuity design, other quasi-experimental, or descriptive outcomes evaluation. A program and strategy may be as broad as a federal grant program or as narrow as a specific intervention that can be supported by IDEA funds and is commonly used in schools and/or has some suggestive evidence of effectiveness.

To conduct the feasibility assessment, the contractor shall gather information, as appropriate, such as through interviews with federal, state, and local educators and administrators, with federal grantees, and with program and strategy providers. The purpose of the interviews, for example, may be to obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites. This type of exploration may inform, for example, the feasibility of obtaining sufficient data to conduct a regression discontinuity design or the likely validity of such a design. Following collection and analysis of this information, the contractor shall develop options regarding the study design methods, outcomes, sample size (through statistical power calculations), and data collection approach.

3.2 Identify and Define the Treatment for Effectiveness or Outcomes Studies. As specified in a particular task order, the contractor, or the contractor in consultation with the COR, shall identify and define one or more policy-relevant programs and strategies for possible inclusion in an impact or outcome study. A program and strategy may be as broad as a federal grant program or as narrow as a specific intervention that can be supported by IDEA funds and is commonly used in schools and/or has some suggestive evidence of effectiveness.

To identify and define the treatment for an effectiveness or outcome study, the contractor shall gather information, as appropriate. For example, the contractor may conduct interviews with federal, state, and local educators and administrators, with federal grantees, with program and strategy providers, and relevant content experts. The purpose of the interviews, for example, may be to obtain information about the prevalence of a particular strategy of interest, how it is implemented, its costs, training requirements, and vendor capacity to support implementation. Information may also be gathered and analyzed, as appropriate, from existing administrative data sources, such as from federal, state, or local databases and websites. The contractor may also conduct a broad or targeted review of relevant literature to collect information on the evidence of effective or promise of various programs and strategies. Information may be gathered and analyzed, as appropriate, from existing repositories.

This type of exploration may inform, for example, the selection and definition of one or more programs and strategies for inclusion in an effectiveness or outcome study. Following collection and analysis of this information, the contractor shall develop a description of relevant programs and strategies, their policy relevance, intended implementation, training requirements, and evidence of effective or promise.

3.3 Survey Sample Design. As needed, the contractor shall assist ED in determining the design options for one or more nationally-representative surveys. In consultation with the COR, the contractor shall identify feasible and policy relevant stratification approaches and the number of states, service providers, local school districts, schools, teachers, parents, students, or other type of respondent needed to fulfill the objective of the survey, as specified by the COR.

3.4 Study Data Collection and Analysis Plan. The contractor shall develop a plan for addressing the research questions for one or more studies, as specified by the COR through a task order. A plan shall first describe the information topics, domains, and constructs that are necessary to achieve the objectives of the study and, at a high level, how that information might be obtained and analyzed. Such a plan must be written in a way that it could be included in an Office of Management and Budget (OMB) Forms Clearance package and, in the case of effectiveness studies, a pre-registered research plan. A plan shall also then provide an outline for the intended report(s), which clearly specify how each section of the report will be structured to ensure that each research question is answered in turn. The contractor shall use this outline to create a detailed data collection instrument crosswalk, which lays out what specific data are needed to support each part of the outlined report.

3.5 Data Collection Instruments. The contractor shall develop study intake forms, survey questionnaires, data extraction forms or instructions, outreach or recruitment letters, or other instruments, as specified by the COR, to be used in conducting a study under one or more task orders in this contract. These instruments shall be developed in accordance with the data collection plan, and specifically the “crosswalk” developed in Task 3.3. The instruments shall minimize burden on respondents while ensuring that key data items are collected.

3.6 Select Intervention Provider(s). For effectiveness studies of interventions, programs, or strategies to accelerate the improvement of children with disabilities, one or more external vendors may be needed to provide support for implementation or technical assistance to study sites. In these cases, the contractor, in consultation with the COR and as specified in a task order, shall establish an expert panel to advise on selection of providers, announce the provider opportunity widely including a request for proposals (RFP), collect the submitted proposals, and support the expert panel in reviewing and recommending providers for the specified role in an evaluation study.

3.7 Study Critique(s). The contractor shall review a study or set of studies, as specified by the COR, to develop information on relevant policies and practices and methodology, including study strengths and areas that could be improved, inform the design and conduct of descriptive or impact studies.

Task 4—Systematic Evidence Review

The contractor shall conduct an evidence review following the procedures and standards used by the WWC, including the use of WWC-certified study reviewers. IES will validate certification by cross-checking proposed study reviewers with WWC’s database of certified reviewers. The work shall include determining policy-relevant categories of strategies to examine, developing a review protocol, searching the literature to identify relevant studies, screening in eligible studies, coding eligible studies, assessing the methodological quality of each eligible study, analyzing the results of eligible studies that meet WWC standards or meet standards with reservations, and synthesizing the results across studies.

The task’s evidence report shall address the question of what is known about the effectiveness of one or more strategies that aim to improve the outcomes of children with disabilities – either in one topic area or many, as specified by the COR. Topic areas may include improving key outcomes such as math and literacy achievement, inclusion in the general education classroom, functional and social-emotional skills, behavior/attendance, educational attainment, employment, and earnings. The report shall include a description of the methodology of the review and the state of the literature on programs and strategies that aim to improve the outcomes of children with disabilities (number of studies identified in the scan for particular outcome measures, the number that meet WWC evidence standards). Most importantly, the report shall summarize the evidence from the studies meeting WWC standards and meeting WWC standards with reservations and identify gaps in the literature.

4.1 Create an Evidence Review Plan. Prior to conducting the review, the contractor shall submit a proposed plan for conducting the evidence review. The evidence review plan shall include parameters for the systematic evidence review, such as databases and other sources that the contractor will consult, search terms the contractor will use, and any limitations that will be imposed (e.g. in terms of publication years). The evidence review plan shall also include the contractor’s plans for systematically compiling the evidence base in an evidence report (e.g., report outline, proposed tables shells, intervention summary sheets, appendices).

4.2 Conduct the Review. Using the final evidence review plan, the contractor shall conduct the review and submit a draft evidence review report. The report length will be specified in the particular task order, but is expected to be no more than 15 pages, with information on the technical aspects of the review and full documentation of the interventions, data, analytic methods, and results of studies included in appendices. The contractor shall also include a one-page summary of the evidence report that contains enough information to form the basis for a press release.

The evidence review report will undergo review from IES’ Standards and Review Office (SRO). The contractor shall be responsive to any SRO comments and make revisions to the report until it clears SRO. The contractor shall provide a web-ready (i.e. 508-compliant), final report to the COR after clearing SRO.

Task 5 –Data Collection Approvals and Documentation

5.1. Prepare OMB Forms Clearance Packages, Document Institutional Review Board Approval, and Pre-Register Research Plan. The contractor shall prepare a forms-clearance package in order to receive approval from OMB for data collection. The contractor shall ensure that the clearance package follows the required OMB format, justifying the necessity for collecting data and comprehensively responding to each required item in the instructions, including: (1) the objectives of the study or collection of studies, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, (7) limitations of the study(ies) and their potential use for policy decisions, and (8) table shells indicating how data will be tabulated and analyzed.

All forms clearance packages shall be brief, clear, and well written. Each package shall include the forms and instructions that will be used to obtain data or recruit respondents. The COR will submit the final package to begin the OMB approval process. During the initial 60-day public comment period, the contractor shall pretest the data collection forms so that burden information can be updated in the final package. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process.

The contractor shall obtain approval from its Institutional Review Board (IRB) for the evaluations conducted under any task order. The effort to obtain initial approval and to update approval as new task orders are issued should be conducted in conjunction with OMB clearance package efforts.

The contractor shall pre-register the research plan for any effectiveness evaluations conducted under a task order, once the recruited sample sizes are known. The registry shall be chosen by the contractor with approval from the COR. An example of an acceptable registry is the Registry of Efficacy and Effectiveness Studies (REES).

5. Prepare OMB Forms Clearance Packages for an Expedited Review. The contractor shall prepare a forms-clearance package in order to receive expedited approval from OMB for data collection. The contractor shall ensure that the clearance package follows the required OMB format for expedited reviews, justifying the necessity for collecting data and comprehensively and concisely responding to each required item in the instructions, including: (1) the objectives of the study or collection of studies, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, and (7) limitations of the study(ies) and their potential use for policy decisions.

The package shall be developed to obtain OMB approval using IES’ Generic (expedited) Clearance vehicle, which requires less documentation and results in approval within roughly 15 days in comparison to a typical ICR submitted through the regular OMB clearance process. All forms clearance packages shall be brief, clear, and well written. Each package shall include the forms and instructions that will be used to obtain data for ED internal decision-making. The COR will submit the final package to begin the OMB approval process.

The contractor shall obtain approval from its Institutional Review Board (IRB) for the evaluations conducted under any task order. The effort to obtain initial approval and to update approval as new task orders are issued should be conducted in conjunction with OMB clearance package efforts.

Task 6 – Outreach and Site Recruitment

The contractor shall develop and implement a strategy for identifying and recruiting districts, schools, grantees, or other type of participant to meet the sample size requirements specified in a task order. As part of the recruitment process, the contractor shall prepare notification letters and information packets for the prospective study sites. The notification packet shall include a non-technical brochure describing the study that is suitable for distribution to a broad audience of policy makers and educators. The contractor shall include in the letters and information packets general information on the study as well as specific information on the importance of the study, its purposes, products, timelines, and the expected benefits to be derived from participation in the study. Recruitment may occur through mailings, telephone calls, in-person visits, or discussion with potential sites at key conferences or meetings, as specified in the task order.

The contractor shall submit an outreach and/or recruitment plan, draft letters, and all other notification materials to the COR for review. Once approved by the COR, the contractor shall send the notification materials to potential sites and execute the recruitment plan. The contractor shall consult with the COR to finalize the list of sites for the study, and subsequently submit documentation of memoranda of understanding for those sites to the COR.

As part of the recruitment process, the contractor shall comply with: The Privacy Act of 1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants.

Task 7— Intake and Random Assignment for Effectiveness Studies

The contractor may need to set up a process for enrolling participants into a study (“intake,” including obtaining consent) and for random assignment. Intake may occur at one or more specific points in time – e.g., at the beginning of each semester or of a school year -- or on a rolling basis over an extended period. Random assignment may be of students, teachers/classroom, schools, or other units depending on the evaluation design and will be specified in the task order. Random assignment may be conducted by the contractor or in the field (at schools or other sites) to accommodate rolling intake. The contractor shall develop protocols and procedures to conduct intake and random assignment to be minimally disruptive to participants and program staff who may be involved in intake.

Task 8—Implement and Monitor the Programs and Strategies in Effectiveness Studies

8.1 Training and Implementation Support. For task orders that involve conducting an effectiveness study where the contractor or an external vendor provides technical assistance to grantees or training/implementation support to study sites, the contractor shall ensure that these supports are provided with fidelity, according to the study plans. These supports may include the provision of in-person or virtual training during the summer or throughout the implementing school year, the provision of materials (such as electronic equipment, curriculum, textbooks, technological hardware or software, etc.).

8.2 Monitoring Implementation. The contractor shall monitor implementation of the intervention. The contractor shall ensure there is an appropriate firewall, such that there is a clear separation between those who are providing and supporting the implementation and those who are monitoring the implementation.

Task 9—Data Collection

IES expects that studies conducted under these task orders will rely on a variety of data sources. These may include surveys, interviews, or focus groups of agency staff at the state or local level, parent, student, teacher, or school surveys, federal records, state records, district records, school records, or records from other administrative systems, including ones that might be maintained by other agencies. Other data sources could include those collected through vendor systems, classroom observations, and site visits. For example, the U.S. Department of Health and Human Services has the National Directory of New Hires (NDNH), which is a useful source of data on employment and earnings. In addition, some task orders may call for analyzing data collected and maintained by IES’ National Center for Education Statistics (NCES). The specific data to be collected or extracted shall be specified by task order.

9.1 Administer state special education administrator/early intervention coordinator surveys. The contractor shall administer the relevant survey to administrators in each state. Either the ED program office or the contractor may compile lists of these individuals. The contractor shall follow up with the universe of state administrators through mail, email, and telephone calls in order to achieve a response rate of 100 percent.

9.2 Administer nationally-representative survey of local educational agencies/service providers. Following outreach under task 6, as approved by the COR, the contractor shall administer the relevant survey to a sample of school district/service provider respondents in charge of overseeing special education/early intervention at the local level. The contractor shall follow up with sampled respondents through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

9.3 Administer surveys of Technical Assistance and Dissemination grantees funded by the Office of Special Education Programs. The contractor shall administer the relevant survey to a sample or population of grantees that provide support to states and local education agencies. The contractor shall follow up with respondents through mail, email, and telephone calls in order to achieve a response rate of 100 percent.

9.4 Administer student surveys. The contractor shall administer approved notification and consent, as specified in the task order, and the relevant survey to students participating in an effectiveness study in the most efficient manner possible. The contractor shall follow up with respondents through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

9.5 Administer school surveys. The contractor shall administer the relevant survey to schools participating in an effectiveness study in the most efficient manner possible. The contractor shall follow up through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

9.6 Administer teacher surveys. The contractor shall administer the relevant survey to teachers participating in an effectiveness study in the most efficient manner possible. The contractor shall follow up through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

9.7 Administer parent surveys. The contractor shall administer approved notification and consent, as specified in the task order, and the relevant survey to parents participating in an effectiveness study in the most efficient manner possible. The contractor shall follow up with respondents through mail, email, and telephone calls in order to achieve a response rate of 85 percent.

9.8 Conduct classroom observations. The contractor shall conduct live or video observations of classroom teaching, student learning/behavior, or their interaction as specified in the task order. The contractor shall coordinate scheduling and follow up to achieve a response rate of 85 percent.

9.9 Collect student records data. The contractor shall obtain records for students in an effectiveness study using the most efficient process possible. These records may be obtained from state data systems (e.g., states’ longitudinal data systems) or from districts or schools directly and include some combination of students’ achievement scores on state assessments in core subjects such as math and ELA, demographic characteristics, special education status including disability category, individual education programs (IEPs), course taking, grades, attendance, or attainment, as specified in the task order. The contractor shall follow up to achieve a response rate of 85 percent.

9.9.1 Obtain Birth Records data. If specified in a task order, the contractor shall arrange with states to obtain birth records, in order to understand neighborhood and school mobility characteristics of children.

9.10 Collect teacher records data. The contractor shall obtain records for teachers in an effectiveness study using the most efficient process possible. These records may be obtained from state data systems (e.g., states’ longitudinal data systems) or from districts or schools directly, and include some combination of teachers’/school staff’s evaluation scores, certification, experience, education, grade/subject assignments, retention, or demographic characteristics, as specified in the task order. The contractor shall follow up to achieve a response rate of 85 percent.

9.11 Administer student assessment. As specified in the task order, the contractor shall administer a student assessment in participating schools in an effectiveness study. Such an assessment may be necessary if existing state assessments do not provide the appropriate sensitivity or focus for the intervention tested. These study-administered assessments could focus on outcomes of interest such as math or ELA conceptual knowledge, or social-emotional skills. The contractor shall follow up to achieve a response rate of 85 percent.

9.12 Administer teacher assessment. As specified in the task order, the contractor shall administer a teacher assessment in participating schools in an effectiveness study. Such an assessment may be necessary if teacher knowledge on particular issues is a core outcome of the intervention being tested. These study-administered assessments could focus on outcomes such as subject-matter content knowledge, pedagogy, or instruction with children with disabilities. The contractor shall follow up to achieve a response rate of 85 percent.

9.13 Conduct interviews or focus groups with state/district/school administrators, teachers, parents, or students. As specified in the task order, the contractor shall conduct interviews or focus groups with one or more respondent groups. The contractor shall strategically recruit, schedule, and follow up with respondents to achieve a response rate of 85 percent.

9.14 Collect data from intervention providers. As specified in the task order, the contractor shall obtain relevant data from providers who support implementation of programs and strategies to improve the outcomes of children with disabilities. Such data could come from meta-data in the provider’s system, other existing documents, or interviews or surveys of the vendors – for the purposes of providing information about the programs and strategies, including implementation and costs. These existing documents should be collected using the most efficient process possible.

9.15 Obtain NSC data. The contractor shall set up a Memo of Understanding with the National Student Clearinghouse (NSC) and obtain outcome data on postsecondary educational enrollment and attainment for a school improvement study, as specified in a task order.

9.16 Access other administrative records. The contractor shall assist IES in setting up a Memo of Understanding with another agency or organization in order to obtain data on services or outcomes for students in an effectiveness study, as specified in a task order. The contractor shall specify the data to be extracted and, if the data will be transferred to the contractor, arrange for secure transfer, or use the data on site at the other agency/organization.

9.17 Obtain NCES Restricted Use File data. The contractor shall obtain a NCES restricted use license and the data in order to conduct extant data analysis.

Task 10—Data Analysis and Reporting

10.1 Analysis Plan. The contractor shall develop a plan that describes the study design, if relevant, and identifies the data sources, key measures, and intended data analyses that will be included in the evaluation’s written products, as specified in the task order. The plan shall include table shells and, if relevant, the statistical power for measuring impacts on the planned outcomes with the achieved sample size. The plan shall specify any sensitivity tests the contractor plans to conduct.

10.2 Evaluation Report. Once the analysis plan developed under Task 10.1 is approved, the contractor shall submit an outline for the report where the findings from analyses will be presented. This outline should be an update (as needed) of the outline developed in Task 3.3 (Study Data Collection Plan). The contractor shall conduct data analyses and prepare a report describing what is being studied and the results of the analysis. The contractor shall borrow the introduction, study design description, and data sources from the final analysis plan, rather than developing this from scratch, to the extent that it meets the COR’s approval.

The contractor shall make sure that the report is written and organized so that it is accessible to policy makers and research-savvy practitioners. The report text length is expected to be no more than 15 pages, with information on the technical aspects of the study included in appendices. To the extent feasible, the report shall include visually intuitive figures and graphics rather than tables. The contractor shall also include a one-page, stand-alone summary of the report that contains enough information to form the basis for a press release.

Both the draft report and one-page summary will be submitted by the COR for approval through IES’ Standards and Review Office (SRO). The SRO process includes review by two external researchers with expertise in the relevant topic. The contractor shall be responsive to SRO’s comments and make revisions to the report until it clears SRO. After the report and 1-pager are cleared by SRO, the contractor shall provide web-ready (i.e. 508-compliant) versions of the report and one-pager.

10.3 Internal Study Brief. The contractor shall conduct data analyses as outlined in an approved plan developed under Task 10.1. The contractor shall prepare a brief that includes mostly exhibits (figures, graphs, tables) and bullet points of key findings. The text of the brief shall be no more than 15 pages long but the set of exhibits the text draws upon may be significantly longer.

10.4 Study Snapshot. The contractor shall conduct data analyses as outlined in an approved plan developed under Task 9.1. The contractor shall prepare a snapshot that includes 3-5 main headlines with a set of supporting bullets. Each headline should be accompanied by one or two exhibits (figures, graphs, tables). The snapshot is meant to provide a quick first look at data, with minimal text and details, as compared to an evaluation report. The main body of the snapshots are typically less than 15 pages.

The snapshot will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the snapshot until it clears SRO. After the snapshot is cleared, the contractor shall provide web-ready (i.e. 508-compliant) versions of the final snapshot.

10.5 How-To Guide. The contractor shall conduct analyses of implementation data as outlined in an approved plan developed under Task 9.1. The contractor shall prepare a How-To guide that is geared toward practitioners. The purpose is to provide guidance on how best to implement a particular school improvement strategy or implementation that whose effectiveness was studied and found to be promising. The snapshot should include considerations of vendor selection, capacity requirements to implement, flexibilities for implementation, costs, potential pitfalls and ways to overcome them, as well as other necessary technical details. There is not a page limit for the guide, but the contractor shall balance the need for sufficient detail with the need to make the guide approachable and accessible to a practitioner audience.

The guide will undergo SRO review. The contractor shall be responsive to SRO’s comments and make revisions to the guide until it clears SRO. After the guide is cleared, the contractor shall provide web-ready (i.e. 508-compliant) versions of the final guide.

10.6 Data File. The contractor shall prepare data files that contain all data collected for an evaluation report, as specified by the COR, with all direct personal identifiers removed. The contractor shall provide documentation for the files including variable labels, category labels, the unit of observation, weights, and how missing data were addressed.

The contractor shall also provide IES with a risk disclosure analysis for submission to NCES’ Disclosure Review Board (DRB). The contractor shall provide the risk disclosure analysis. After DRB approval of the codebooks and risk disclosure analysis, the contractor shall submit to the COR the data files consistent with NCES standards4. ASCII or SAS file formats are acceptable.

Task 11—Logistical Support for National Assessment of IDEA

ED may be required to develop and submit reports to Congress providing an independent, national assessment of IDEA. Those reports will draw on the findings from the task orders under this contract or related studies under other contracts, as well as on studies supported by others centers within IES or being conducted outside of IES.

The contractor shall provide logistical support to ED in developing these reports. Such support may include developing tables and figures, editing and formatting the report, and ensuring that the final version of the report is 508-compliant. The contractor shall produce up to four drafts of a report, as well as a final version once it has been approved by SRO.

Task 12—Monthly Communication with ED and Reporting

12.1 Ongoing Communication. The contractor shall provide updates on task order activity and identify issues or problems primarily through email. The contractor shall participate in monthly conference calls with the COR as needed during periods of task order activity. These calls will be used to update the COR on the progress of task order activities and to discuss any issues or anticipated problems and proposed solutions to those problems. Although the contractor may be required to perform this subtask for different task orders, ED expects that the emails and monthly calls will pertain to all current task orders in the aggregate and not that there will be separate monthly calls for each task order.

12.2 Monthly Reporting. The contractor shall submit one copy of the following reports each month to the COR and CO:

· Monthly Progress Report/Exception Reports. The contractor shall submit monthly progress reports. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

· Monthly Manpower/Expenditure Reports. The contractor shall submit monthly expenditure reports. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs for each task order and for each task within a task order, and specifying for all travel the locations, duration, and personnel for each trip.

· Security Clearance List Maintenance. The contractor shall maintain lists of each staff member assigned to work on the contract, the required security clearance level of their position (see table below), and their progress toward attaining clearance.

Act, or the Privacy Act.

MR

Researcher (Analyst, Programmer, Statistician, Research Assistant) No risk

Act, or the Privacy Act.

N/A

Act, or the Privacy Act.

N/A

IV. INFORMATION TECHNOLOGY AND INFORMATION SECURITY REQUIREMENTS

The contractor shall ensure that all data in a computerized form shall be suitable for data editing and corrective actions consistent with the U.S. Department of Education’s Information Quality Guidelines in order to provide for quality control of data entry and yielding verified and accurate records.

The contractor shall ensure that all reports comply with the appropriate technical standards for compliance with Section 508 of the Rehabilitation Act of 1973 as amended, at 36 CFR 1994, and be subject to conformance testing by the Department’s Assistive Technology Program. All Section 508 violations uncovered during the testing process shall be remediated by the vendor at no charge to the Department.

The Department has determined that the work under this contract does not involve a federal information system and therefore does not require a Security Authorization (SA) or…