Task Order 0001 IDEA PWS 12.2.21.docx

U.S. Department of Education Performance Work Statement Institute of Education Sciences National Center for Education Evaluation

Task Order 0001 - Performance Work Statement

EVALUATING PROGRAMS AND STRATEGIES TO IMPROVE THE OUTCOMES OF STUDENTS WITH DISABILITIES:

DESIGN AND CONDUCT OF AN IMPACT EVALUATION OF PROFESSIONAL DEVELOPMENT FOR GENERAL AND SPECIAL EDUCATION TEACHERS TO IMPROVE INSTRUCTION AND ACADEMIC OUTCOMES FOR STUDENTS WITH DISABILITIES

In awarding the basic ordering agreement (IDIQ) on “Evaluating Programs and Strategies to Improve the Outcomes of Students with Disabilities,” the U.S. Department of Education (ED), Institute of Education Sciences (IES) will also award this task order (0001) to perform key activities for an evaluation of teacher professional development designed to improve instruction for and outcomes of students with disabilities in general education classrooms. The tasks and subtasks referred to are those outlined in the IDIQ Performance Work Statement. The Task Order will have a 24-month period of performance and will set the stage for future task orders to complete the impact evaluation.

Background

There are growing indications that pandemic related education disruptions have resulted in loss of learning for all students, with perhaps the greatest consequences for students such as those with an individualized education plan.[footnoteRef:1],[footnoteRef:2] This new context creates two challenges. First, the extent of the learning loss requires educators to identify and implement strategies that significantly accelerate student learning. Second, since all students have been affected but by differing amounts, longstanding inequities are likely exaggerated. Together these challenges call for approaches that can be used broadly but tailored to address individual student needs to accelerate their learning and reduce achievement gaps. [1: Most Students With Disabilities Still Attend Remotely. Teachers Say They're Falling Behind (edweek.org) ] [2: The impact of COVID-19 on student achievement and what it may mean for educators (brookings.edu) ]

Students with disabilities are increasingly taught in general education classrooms rather than in facilities or spaces that isolate them from their peers[footnoteRef:3]. Research links instruction in inclusive environments to stronger development of cognitive, social-emotional, and behavioral skills and post-secondary outcomes for students with disabilities.[footnoteRef:4],[footnoteRef:5] Perhaps as a result, the Individuals with Disabilities Education Act (IDEA) called for these children to be educated alongside others without disabilities in regular classrooms and following the general education curriculum, to the maximum extent possible. By 2018, data suggest that most students with disabilities are spending more than three-quarters of their school day in these inclusive, general education classrooms.[footnoteRef:6] [3: U.S. Department of Education, Office of Special Education and Rehabilitative Services, Office of Special Education Programs, 42nd Annual Report to Congress on the Implementation of the Individuals with Disabilities Education Act, 2020, Washington, D.C. 2021] [4: Odom, S. L., et al. (2004). Preschool inclusion in the United States: A review of research from an ecological systems perspective. Journal of Research in Special Educational Needs, 4(1), 17–49. ] [5: Rojewski, J. W., Lee, I. H., & Gregg, N. (2015). Causal effects of inclusion on postsecondary education outcomes of individuals with high-incidence disabilities. Journal of Disability Policy Studies, 25(4), 210 – 219.] [6: U.S. Department of Education, Office of Special Education and Rehabilitative Services, Office of Special Education Programs, 42nd Annual Report to Congress on the Implementation of the Individuals with Disabilities Education Act, 2020, Washington, D.C. 2021]

This progress towards more inclusion has placed greater demands on general education teachers and raised the stakes for how effectively they work with all students. However, teachers often report that they feel unprepared to meet the needs of students with disabilities[footnoteRef:7]. Most receive little training on this topic despite the increased expectations and responsibilities. Identifying strategies that improve teachers’ preparation in this way would enhance their professional growth and equity in students’ access to instruction, both key priorities for ED. [7: Most Classroom Teachers Feel Unprepared to Support Students With Disabilities (edweek.org)]

This task order addresses this critical need by evaluating one or more professional development approaches designed to improve academic achievement of students with disabilities, specifically those with an individualized education program, in general education classrooms. By “approach” IES means a class or category of educational strategies that policy makers might encourage state or local leaders to adopt if found to be effective – through a reauthorization of IDEA, government rules or regulations, or technical assistance. Congress and ED officials do not typically promote particular developers, vendors, commercial products, or minor aspects of instruction – and in some cases are prohibited from doing so. Professional development (PD) approaches that are policy relevant might include coaching or joint training of general education and special educator teachers.

Key Considerations for the Professional Development to be Studied

To improve classroom instruction for students with an individualized education plan, this task order will study a professional development approach that focuses on differentiated instruction and its application to special education students in the general education classroom. Evidence indicates that differentiated instruction can be an effective strategy to improve learning for students generally[footnoteRef:8]. However, there is limited evidence of how to tailor differentiated instruction practices to serve students with disabilities or in so doing whether it is effective. [8: https://www.ascd.org/el/articles/differentiated-learning]

Differentiated instruction for the purposes of this task order references a set of teaching practices that facilitate tailoring instruction and instructional opportunities to meet individual student needs. Because of this study’s focus, it will be important to feature those practices that are most likely to benefit students with disabilities who receive the bulk of their instruction within the general education classroom. Consistent with evidence about effective professional development, ED anticipates that the delivery of the professional development will follow best practices highlighted in research.

Key considerations for defining the treatment for the evaluation:

There are a number of considerations that will affect the professional development that will be studied. The offeror must describe their understanding of these considerations and their implications for choosing a particular strategy to evaluate.

· Instructional practices for focus in the professional development. ED expects the professional development to focus on high-leverage instructional practices that are based on prior evidence of effectiveness and that have the greatest likelihood of improving the academic achievement of students with disabilities who receive instruction in the general education classroom. To promote teacher take up and feasibility of full implementation of the practices, it’s also important for the identified the practices to be targeted (a total of 3-4 practices), relatively easy for teachers to implement, and tailored to the needs of their students with an IEP. It’s important that the practices meet the needs of all or most of the students with disabilities typically included in the general education classroom, versus a subset of students, and are applicable across various content areas. It’s also important that the practices have the potential to benefit all students in the classroom so that teachers see the value of the practices more generally to facilitate take up. The professional development might emphasize, for example, practices such as ways to appropriately group students to enhance their learning or teacher actions to increase student opportunities for practice with feedback. Finally, it’s critical that the selected practices are feasible for teachers to implement well with modest professional development. Particularly in the current circumstances, teachers have limited time for professional development.

· Extent and format of professional development. ED expects the delivery of the professional development to be consistent with best practice. This suggests that the professional development (PD) involve initial training intensive enough to help classroom teachers understand specific instructional practices to be implemented. However, to change practice in the classroom, research indicates that more active application with feedback is effective, such as occurs with coaching. Implementation of the professional development shall balance the need to provide enough training and ongoing support to enable a fair and reasonable test of the approach while ensuring the extent of the professional development is feasible to implement and policy relevant.

· Participation of district coaching staff, general education classroom teachers, and special education staff. ED expects that to be policy relevant the approach will be coordinated with or designed to enhance existing resources and capacities. Thus, the strategy shall include training and involvement of relevant staff (such as district coaches and special educators) who serve general educators in the targeted grades to coordinate and support the evaluation’s strategy to be implemented. District coaches, with training, can provide ongoing support to implement the desired practices; special education staff are best positioned to support general education teachers with any tailoring of the practices needed for their students with disabilities.

The purpose of this study is to inform whether this type of professional development, described, can be effective in supporting the learning of students with disabilities (SWDs) when taken to scale and to explore its effectiveness in various contexts. This “proof of concept” evaluation is appropriate given the limited existing evidence but great need to improve academic achievement of students with disabilities in general education classrooms.

Key Considerations for the Design of the Study

There are a number of factors that will affect the evaluation design or successful execution of the study. The offeror must fully explain and justify their proposed approach based on the offeror’s understanding of the evidence and relevant content and methodological knowledge. Other design issues may arise as the evaluation is carried out that will need to be resolved.

· Sample, Random Assignment, and Statistical Power. ED has some preliminary expectations for the design and sample size for the evaluation, but the contractor shall carefully consider the tradeoffs of the options and any alternatives that would achieve the goals of this task order. The expectations or early estimates include that the evaluation will be conducted in up to 15 districts and 150 disadvantaged and/or low-performing elementary schools, focusing on third through fifth grade. ED expects that the evaluation will use a random assignment design that includes a “treatment” arm and a “business-as-usual” group. However, there are various possible ways to conduct the random assignment, for example at the school level, grade level, or classroom level. The contractor shall justify the minimum detectable effects that are policy relevant for each key contrast described below.

· Generalizability. ED strongly prefers the evaluation results to be applicable to disadvantaged and/or low-performing elementary schools nationwide. The contractor shall carefully consider the best strategy for accomplishing this goal, including use of state-of-the-art techniques both as part of the recruitment plan and during the analysis phase.[footnoteRef:9],[footnoteRef:10] [9: Stuart, E. A., Bradshaw, C. P., & Leaf, P. J. Assessing the generalizability of randomized trial results to target populations. Prevention Science, 16, 475-485. https://doi.org/10.1007/s11121-014-0513-z] [10: Tipton, E. & Olsen, R. B. (2018). A review of statistical methods for generalizing from evaluations of educational interventions. Educational Researcher, 47, 516-524. https://doi.org/10.3102/0013189x18781522]

· Counterfactual. Schools increasingly provide professional development to support general education teachers’ instruction of special education students. Additionally, special educators support general educators in various ways (e.g., through consultation, coaching, and paired teaching). The contractor shall consider what is the most policy-relevant service contrast for the evaluation and what “business-as-usual” contexts are most appropriate.

· Student Outcomes. ED expects that at a minimum, student achievement scores on state assessments in math and reading will be a key outcome of interest. However, the contractor shall also consider the policy relevance of additional outcomes, such as study-administered assessments and/or non-assessment outcomes particularly relevant for students with disabilities. The contractor shall also consider valid and reliable measures that facilitate learning from this study such as teacher classroom practice.

The goal of the evaluation design is to ultimately address the following overarching research questions:

1. After one year of implementation, does professional development for teachers and staff supporting students with disabilities in the general education classroom improve the math and reading achievement of students with disabilities relative to the business-as-usual teaching practices? Does this strategy work better for particular students with disabilities, such as students with less or more significant needs?

2. What are the effects of the professional development on students without disabilities? Positive or negative?

3. Are any changes in the achievement of students with disabilities related to teacher characteristics, such as their teaching experience? Or, related to features of implementation such as extent of coordination with special educators?

4. Does the professional development improve teachers’ instructional practices? If so, is the impact on teacher practice related to n student achievement impacts?

5. If the professional development demonstrates evidence of effectiveness, what is its cost and cost effectiveness?

Scope of Work

The specific work to be conducted under Task Order 0001 includes the following tasks from the basic ordering agreement:

· Task 1: The contractor shall hold an initial meeting (Subtask 1.1 Initial Meeting) with ED within 10 business days after the effective date of the Task Order to discuss details regarding the tasks, scheduling activities, and other issues related to the task order. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed as part of the contract.

· Task 2: The contractor shall establish a TWG (Subtask 2.1 Establish TWG) comprised of 10 individuals that together have expertise relevant to this evaluation, including the needs of students with disabilities, professional development including coaching, instruction and supports for students with disabilities, and causal evaluation methods. The contractor shall submit the proposed list of primary and alternate TWG members to the COR within four weeks of task order award. The contractor shall contact each approved member and submit letters of commitment and documentation of consulting rates to the COR four weeks after COR approval of the TWG list. The contractor shall then hold an initial meeting (Subtask 2.2 Convene full TWG Meeting) of the TWG that will solicit the TWG’s thoughts on the study design, including random assignment, implementation, data collection, analysis, and reporting plans. The meeting shall be held by the end of November 2022 in person in Washington, DC in ED or contractor facilities, if public health conditions allow. Otherwise, the meeting shall be held virtually via video conference. The contractor shall cover the honorarium for TWG members, and if applicable, travel and per diem expenses. Eight weeks prior to the meeting, the contractor shall provide to the COR a draft agenda, a draft slide deck, and a plan for a guided discussion of key topics and issues. Within one week after the meeting, the contractor shall submit a brief but detailed summary of the key observations and recommendations provided by TWG members, along with the contractor’s proposed actions. The contractor shall not make any significant changes in the task order activities based solely upon the recommendation of TWG members without the written approval of the COR and CO.

· Task 3: The contractor shall perform work as described under Subtask 3.4 Study Data Collection and Analysis Plan, Subtask 3.5 Data Collection Instruments, and Subtask 3.6 Select Intervention Provider(s).

Under subtask 3.4, the contractor shall develop a plan to address the evaluation’s research questions. The plan shall build on the contractor’s proposal and be refined in consultation with the COR. Within 10 weeks of award, the contractor shall submit a draft version of the plan. The COR will review and provide comments on the draft plan within 2 weeks. Within 4 weeks after receiving COR comments, the contractor shall make revisions and submit a final version of the plan.

Under subtask 3.5, the contractor shall develop the actual instruments according to the approved plan. These may include:

· Spring 2025 survey of study principals. ED initially anticipates surveying all principals from schools participating in the study (150).

· Spring 2025 survey of study general and special education teachers. ED initially anticipates surveying all study teachers, up to 1350 general education teachers and 225 special education teachers.

· Spring 2025 teacher instruction observation. ED initially anticipates observing all study classrooms, once, and a random half of the study classrooms in each school twice.

· Spring 2025 interview with the provider in the evaluation. ED initially anticipates up to two vendors depending on the vendors’ capacity to provide professional development on the scale required in the study.

· District administrative data. ED initially anticipates collecting in up to 15 districts student-level data on special education status including disability category, accommodations and modifications, math and reading achievement, demographics.

· The contractor also shall collect professional development participation rates, fidelity of provider implementation, cost of implementation information.

Within 12 months of the contract award, the contractor shall submit a draft version of the study instruments. The COR will review and provide comments on the draft plan within 3 weeks. Within 4 weeks after receiving COR comments, the contractor shall make revisions and submit a final version of the instruments.

Under subtask 3.6, the contractor shall run a competition to select the PD provider(s) to train those staff participating in the evaluation. The contractor shall first establish an expert panel of three individuals to advise on the selection of the providers. These experts may also serve on the technical working group described in Task 2 and shall collectively have expertise in the needs of students with disabilities, the provision of teacher professional development for students with disabilities. The contractor shall submit a list of proposed expert panel members to the COR no later than 4 weeks after award. The contractor shall contact each proposed member and obtain and submit letters of commitment and documentation of consulting rates to the COR within 4 weeks of COR approval of the proposed expert panel list.

The contractor shall prepare a draft announcement specifying that proposals will be requested and including the expected timelines for the proposal process, as well as a draft request for proposals (RFP) including the timelines for proposals, provider requirements, and the criteria by which the proposals shall be evaluated. The RFP shall also specify the maximum amount of funding that will be available to providers. The contractor shall submit the draft announcement and RFP to the COR for review within 6 months after award. The contractor shall make revisions and submit a revised announcement and RFP within 2 weeks of COR comment and send the RFP to the expert panelists for review. The contractor shall request that expert panelists provide recommended changes within 2 weeks of receiving the draft RFP. The contractor shall make any necessary revisions within two weeks of receiving expert panel comments. After COR approval, the contractor shall widely disseminate the announcement and, subsequently, the RFP.

Within 3 business days of closing the request for proposals, the contractor shall send proposals to expert panel members for review. The contractor shall request that each member of the expert panel review the provider proposals and write a brief review of each proposal, including ratings for each provider across a set of pre-established criteria as well as each provider’s strengths and weaknesses, along with recommendations of up to 4 vendors/providers for inclusion in a panel meeting, where there will be an opportunity for experts to find out more information on the proposals and for potential negotiation. The contractor shall compile expert panel members’ reviews of proposals and recommendations of proposals for inclusion in the expert panel meetings within one week of receiving reviews. The contractor shall arrange a meeting with expert panel members and developers of each of up to 4 providers recommended by the expert panel. The meeting shall be one or two days and shall include time for each provider to present their plan and answer any questions from the panel, contractor, and/or ED. The contractor shall plan to arrange and pay for up to two staff for each provider to travel to Washington, DC, present on their training plans, and answer questions that the expert panel, the contractor, or ED may have. If public health conditions do not allow for an in-person meeting, the meeting shall be conducted virtually via video conference.

Following the panel meetings, the contractor shall request that each panel member write a brief review of each proposed plan (including strengths and weaknesses). The contractor shall compile and provide to the COR the expert panel members’ reviews of the presentations/proposals within one week of receiving expert panel reviews. In consultation with ED, the contractor shall negotiate with and secure the full cooperation of the selected provider(s). The contractor shall submit subcontract(s) with the selected provider no later than four weeks after the expert panel meeting with providers.

· Task 5: The contractor shall perform work as described under Subtask 5.1 Prepare OMB Forms Clearance Packages, Document Institutional Review Board Approval, and Pre-Register Research Plan. The contractor shall prepare an OMB Forms Clearance package describing plans to collect data that are consistent with subtasks 3.4 and 3.5. By the end of August 2023, the contractor shall submit a draft OMB package. The COR shall review and provide comments within two weeks and the contractor shall have two weeks after receiving comments from the COR to submit the final OMB clearance package. The COR will then submit the package to begin the OMB approval process. During the initial 60-day public comment period, the contractor shall pretest the data collection forms so that burden information can be updated in the final package. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process. The contractor shall also ensure that Institutional Review Board approval is obtained no later than when the OMB package is cleared. Finally, the contractor shall pre-register the research plan no later than two weeks after recruitment (Task 6).

· Task 6 Outreach and Site Recruitment: The contractor shall prepare the site recruitment plan, including related outreach materials, and submit to the COR within 12 months of award. As part of the recruitment planning process, the contractor shall consider the appropriateness and optimal level of incentives for study site participation. Recruitment shall be completed no later than March 2024. During recruitment, the contractor shall submit to the COR progress reports on a biweekly basis.

· Task 7 Intake and Random Assignment for Effectiveness Studies: Following recruitment of study sites, the contractor shall conduct random assignment, as indicated in the contractor’s proposal and refined and finalized in Task 3. Random assignment shall be completed no later than April 2023. During study intake and random assignment periods, the contractor shall provide biweekly progress updates to the COR.

· Task 12: Within one week of each monthly call (Subtask 12.1 Ongoing Communication), the contractor shall send a short email summarizing decisions made in collaboration with the COR. The contractor shall submit progress reports/exception reports and manpower/expenditure reports (Subtask 12.2 Monthly Reporting) for the given month no later than the end of the following month.

Schedule of Deliverables

All deliverables shall be submitted electronically to the COR. The contractor shall also electronically submit one copy of the monthly report and final deliverables to the CO.

Task Order Contract

Task 1 – Meetings

Task 2 – Technical Working Group (TWG)

Task 3 – Feasibility and Design

Task 5 – Prepare OMB Forms Clearance Packages, Document Institutional Review Board Approval, and Pre-Register Research Plan

Task 6 – Outreach and Site Recruitment

Task 7 – Intake and Random Assignment for Effectiveness Studies

Task 12 – Monthly Communication with ED and Reporting