Transition_Support_PWS_03-14-19.pdf

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

EVALUATION OF TRANSITION SUPPORTS FOR YOUTH WITH DISABILITIES

Performance Work Statement

I. INTRODUCTION

The National Center for Education Evaluation and Regional Assistance (NCEE) of the U.S.

Department of Education’s (the “Department”) Institute of Education Sciences (IES) requires a contract to design and evaluate the effectiveness of transition supports for youth with disabilities served under the Individuals with Disabilities Education Act (IDEA). The Department will award a cost-reimbursement with fixed fee contract with a base period to conduct design/feasibility assessment work and options to conduct an impact study. The purpose of the 32-month base period design work is to assess the feasibility of conducting a large-scale random assignment study through activities including a systematic evidence review, development of study design options, and a transition strategy/program competition.

Based on the study design options and the outcome of the transition strategy/program competition, ED may decide to evaluate two transition strategies/programs rather than one (i.e., conduct a 3-arm study instead of a 2-arm study). Optional Periods are structured to support an evaluation of either one transition strategy/program (2-arm study) or two transition strategies/programs (3-arm study).

• Optional Period 1 includes the recruitment necessary for an impact study of one transition strategy/program and the conduct of the initial training in the summer prior to the first implementation year (2022-2023 school year). Optional Period 1A includes recruitment of additional sample and training for a second transition strategy/program (in a 3-arm study).

• Optional Period 2 covers the first implementation year of one transition strategy/program, including the ongoing support for implementation and data collection for the 2022-2023 school year. Optional Period 2A includes the ongoing support for implementation of a second transition strategy/program and data collection (for additional sample in a 3-arm study).

• Optional Period 3 is for a second year of implementation, including any necessary training, ongoing support for implementation, and data collection for the 2023-2024 school year. Optional Period 3A includes the training and ongoing support for implementation of a second transition strategy/program as well as data collection (for additional sample in a 3-arm study) during the 2023-2024 school year.

• Optional Period 4 includes data analysis, convening a Technical Working Group meeting to discuss preliminary findings, report preparations, and development of a restricted-use data file. Optional Period 4A includes data analyses, report preparations, and development of restricted-use files for a second transition strategy/program.

Exhibit 1. Contract Periods of Performance

Dates Contract Month Overlap Duration Base Period (Design/Feasibility)

9/15/19 to 5/15/22

Months 1-32 32 months

Optional Period 1 (and Optional Period 1A, if exercised)

8/15/2021 to 11/15/2022

Months 23-38 10 month overlap with Base

16 months

Optional Period 2 (and Optional Period 2A, if exercised)

6/15/2022 to 10/15/2023

Months 33-49 6 month overlap with Optional Period 1

16 months

Optional Period 3 (and Optional Period 3A, if exercised)

5/15/2023 to 9/15/2026

Months 44-84 6 month overlap with Optional Period 2

41 months

Optional Period 4 (and Optional Period 4A, if exercised)

8/15/2023 to 9/15/2027

Months 47-96 38 month overlap with Optional Period 3

50 months

A. LEGISLATIVE AUTHORITY

This evaluation of transition supports for youth with disabilities is authorized in Section 664 of the Individuals with Disabilities Education Act (IDEA, P.L. 108-446), as amended, which assigns to IES the responsibility to conduct studies and evaluations of the implementation and impact of programs supported under IDEA.

B. BACKGROUND

A central goal of the Individuals with Disabilities Education Act 2004 (IDEA) is to help students with disabilities prepare for education, work, and independent living after secondary school.

However, more than a decade later, students with disabilities continue to lag their peers in rates of high school graduation (66% vs. to 84%), 1 enrollment in postsecondary education (61% vs.

84%),2 and employment (35% vs. 78%, among youth aged 25-34)3.

1 Public high school 4-year adjusted cohort graduation rate (ACGR), by selected student characteristics and state: 2010-11 through 2015-16 (Table 2018-46) retrieved December 7, 2018 from https://nces.ed.gov/programs/digest/d17/tables/dt17_219.46.asp?current=yes.

2 Web Tables: Characteristics and Outcomes of Undergraduates with Disabilities (NCES 2018-432).

https://nces.ed.gov/pubs2018/2018432.pdf

3 McFarland, J., Hussar, B., de Brey, C., Snyder, T., Wang, X., Wilkinson-Flicker, S., Gebrekristos, S., Zhang, J., Rathbun, A., Barmer, A., Bullock Mann, F., and Hinz, S. (2017). The Condition of Education 2017 (NCES 2017-144). U.S. Department of Education.

Washington, DC: National Center for Education Statistics. Retrieved December 7, 2018 from https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2017144.

https://nces.ed.gov/programs/digest/d17/tables/dt17_219.46.asp?current=yes https://nces.ed.gov/pubs2018/2018432.pdf

IDEA emphasizes the role that transition services can play, including transition planning4, in improving postsecondary outcomes among students with disabilities. Recent research conducted by IES suggests that, while transition planning is happening, students with disabilities are not consistently engaging in the types of transition services that may improve their outcomes after high school. Specifically, the National Longitudinal Transition Study 2012 found that most students who had participated in a transition planning meeting had discussed their interests, strengths, preferences, and post-school options with school staff.5 However, youth with disabilities were found to be less likely than their peers to have taken a college entrance test or a high school course for college credit, or to have participated in a paid work experience.6 To the extent that these or other services can facilitate their transition to additional education and training after high school—either individually or in combination as part of a comprehensive set of supports—students with disabilities who do not access them may be missing out.

There is some evidence suggesting the importance of certain types of preparations for students with disabilities’ transition to postsecondary education and employment. These include individualized transition planning, participation in inclusive high school settings, career and technical education, work experience while in high school, and functional life skills instruction.7,8,9 However, few of these strategies have been rigorously evaluated.10

ED seeks to advance the evidence base in the area of transition services for youth with disabilities through the conduct of this new study. To achieve this objective, there are two major components of this requirement: (1) assessing the feasibility of conducting a large-scale random assignment, and (2) an option to conduct an impact study to determine effectiveness.

C. DESIGN AND FEASIBILITY ASSESSMENT

The goal of the contract’s design and feasibility work (base contract) is to determine if it is possible to conduct a large-scale impact evaluation of promising transition services for secondary students with an individualized education program (IEP). Three key deliverables shall be prepared through components of this work: (1) a report synthesizing evidence from reviewed

4 IDEA requires the development of a transition plan as part of his or her IEP by the time the student reaches age 16. That plan must address his or her postsecondary goals and the transition services required to reach those goals.

5 Preparing for Life after High School: The Characteristics and Experiences of Youth in Special Education. Findings from NLTS

2012. Volume 2: Comparisons among Disability Groups. Available at https://ies.ed.gov/ncee/pubs/20184007/ 6 Preparing for Life after High School: The Characteristics and Experiences of Youth in Special Education. Findings from NLTS

2012. Volume 1: Comparisons with Other Youth. Available at https://ies.ed.gov/ncee/pubs/20184007/ 7 Rowe, D.A., Alverson, C.Y., Unruh, D.K., Fowler, C.H., Kellems, R., & Test. D.W. (2014). A Delphi study to operationalize evidence-based predictors in secondary transition. Career Development and Transition for Exceptional Individuals, 38(2) 113-126.

https://journals.sagepub.com/doi/10.1177/2165143414526429

8 Test, D.W., Mazzotti, V.L., Mustian, A.L., Fowler, C.H., Kortering, L., & Kohler, P. (2009). Evidence-based secondary transition predictors for improving post-school outcomes for students with disabilities. Career Development for Exceptional Individuals, 32(3), 160-191. https://sites.bu.edu/miccr/files/2015/03/Evidence-based-secondary-transition-predictors-for-improving-post-school-outcomes-for-students-with-disabilities.pdf

9 Tomasello, J. & Brand, B. (2018). How ESSA and IDEA Can Support College and Career Readiness for Students with Disabilities:

Considerations for States. College & Career Readiness & Success Center. Available at https://ccrscenter.org/sites/default/files/ESSA-IDEA_CollegeCareerReadiness.pdf

10 Improving Post-High School Outcomes for Transition-Age Students with Disabilities: An Evidence Review. Available at https://ies.ed.gov/ncee/pubs/20134011/index.asp https://ies.ed.gov/ncee/pubs/20184007/ https://ies.ed.gov/ncee/pubs/20184007/ https://journals.sagepub.com/doi/10.1177/2165143414526429 https://sites.bu.edu/miccr/files/2015/03/Evidence-based-secondary-transition-predictors-for-improving-post-school-outcomes-for-students-with-disabilities.pdf https://sites.bu.edu/miccr/files/2015/03/Evidence-based-secondary-transition-predictors-for-improving-post-school-outcomes-for-students-with-disabilities.pdf https://ccrscenter.org/sites/default/files/ESSA-IDEA_CollegeCareerReadiness.pdf https://ies.ed.gov/ncee/pubs/20134011/index.asp studies and summarizing what is known about the effects of transition strategies/programs, (2) a report on the design options for a large-scale impact study that includes a description of the promising transition strategies/programs to prioritize and approaches to addressing anticipated challenges, and (3) a Request for Proposal inviting applications from researchers, publishers, and other organizations involved in the development and, if applicable, provision of training on the prioritized transition strategies/programs.

For the purposes of this work, the focus of the effectiveness study can be a combination of transition strategies or a transition programs offered to youth with an individualized education program (IEP) enrolled in secondary school and that includes individualized transition planning, college and career advising, academic, vocational, social skills training, career and technical education, work experience, and/or other supports (e.g., tutoring, mentoring, and follow-up after completing the program).

The Department is prioritizing programs implemented and tested with students with disabilities but will consider transition strategies/programs that have been implemented with all high school students and have some preliminary evidence of efficacy/effectiveness in improving students’ high school completion, enrollment in postsecondary education/training, employment, and/or independent living. The Department expects some adaptation or integration of existing materials may be necessary as part of preparations for the impact study.

Through the design and feasibility work, the contractor shall address the following research questions, keeping in mind the unique set of challenges that special education requirements may present for effectiveness research, including challenges integrating transition strategies/programs with the supports required based on student’s individualized education program (IEP).

• What is known about the effectiveness of transition strategies/programs? And for whom? ED is interested in examining the evidence on the efficacy or effectiveness of transition strategies/programs in improving students’ achievement of the outcomes envisioned by IDEA, including high school completion, enrollment in postsecondary education/training, employment, and independent living. This includes understanding the groups/subgroups of students that the strategies/programs are targeted to support and for whom there is evidence of efficacy/effectiveness.

• What are the transition strategies/programs feasible to evaluate in an impact study?

Based on the evidence review, ED is interested in identifying promising transition strategies/programs most suitable for a large-scale evaluation study. This includes identifying the groups/subgroups of students that the strategies/programs are specifically geared for and would, therefore, be appropriate to include in the evaluation.

• What are design options for the impact study? ED is interested in design specifications taking into account responses to the following questions:

o What is the transition strategy/program to be evaluated? Is there a single strategy/program or are there a number of promising strategies/programs to consider for a multi-arm study?

o Are there important variations of the transition strategy/program that are policy relevant to study and could be varied within the study?

o What is the appropriate, policy-relevant counterfactual?

o At what level should random assignment be conducted?

o For what minimum detectable effect size should the study be powered? How large would the study sample need to be (number of sites, number of students) to detect an impact of this magnitude?

o What recruitment activities would be required to achieve such a sample size?

What are anticipated challenges to recruitment? Are there any activities that could be conducted as part of base period’s feasibility assessment to help facilitate recruitment and mitigate some of the challenges?

o What will be required to obtain outcome data through administrative sources (e.g., MOU/ agreements, data used for matching procedure)? What are anticipated challenges with relying solely on administrative data to examine outcomes? How can these challenges be remedied or mitigated?

o What are the training and supports necessary to implement the transition strategy/programs with fidelity (and sufficient intensity)?

o Are there vendors/providers that can provide the necessary training in the transition strateg(ies)/program(s)? Can these providers adequately train the requisite number of districts/schools/staff in the transition strategies/programs for a large-scale impact evaluation?

o Is a pilot of this training necessary prior to large-scale implementation in an impact study?

Key Components of Design and Feasibility Assessment

Systematic Evidence Review

To identify possible transition strategies/programs to consider in the design of the impact study, the contractor shall conduct a systematic evidence review of studies examining the efficacy or effectiveness of transition services/programs in improving high school completion, enrollment in postsecondary education/training, employment, and independent living of youth with an IEP.

These transition strategies/programs will include, but not be limited to, those described in IES What Works Clearinghouse (WWC) intervention reports and reviews of individual studies11.

Widely-used transition strategies/programs will also be considered for inclusion in the evidence review.

The contractor shall use WWC-certified reviewers to review the identified studies using WWC standards and procedures, unless already reviewed and in WWC intervention reports. The

11 WWC reports and reviews available at https://ies.ed.gov/ncee/wwc/FWW https://ies.ed.gov/ncee/wwc/FWW contractor shall synthesize evidence from studies determined to meet WWC standards or meet WWC standards with reservations in a report. The report shall summarize what is known about the effects of the transition strategies/programs and identify where there are gaps in the literature.

The report will be reviewed by IES and publicly released as a product of the study.

Design Options

The study design options will prioritize randomized controlled trials (RCT), to support strong causal inferences about the impacts of the transition strategy(ies)/program(s) on students’ high school and post-school outcomes, including educational attainment, employment, and earnings.

When designing options, there are some issues to consider. They include, but are not limited to, the following:

• Intervention: a consideration in designing the study will be the components of the transition strategy/program, the complexity of the strategy/program model (e.g., whole school, classroom-based, individual), the key participants for successful implementation (e.g., teachers, other district/ school staff, vocational rehabilitation service providers, work-based learning community partners), and the implementation intensity (or dosage) necessary.

• Counterfactual: a key consideration is the nature of the transition practices implemented by the control group and the underlying programs of services available to students in the control group.

• Recruitment: consideration of challenges with recruitment for random assignment study given IDEA requirement for transition services as well as challenges recruiting and retaining high school student participants in study.

• Training and implementation: a consideration is the training and professional development required to ensure the transition strategy/program is implemented with fidelity (and sufficient intensity) and the supports necessary to assist with implementation.

• Crossover/contamination: depending on the design and level of random assignment, there is potential for staff in the control group to learn about and possibly implement some aspects of the transition strateg(ies)/program(s) being evaluated.

• Data collection: consideration about the feasibility of obtaining outcomes of interest through administrative records, the data required to obtain administrative records (i.e., through matching procedures), the number of years of data collection (i.e., whether two years post high school will be adequate for looking at preliminary post-school outcomes), the frequency and timing of data collection, and the targeted sample for data collection.

The design options report shall weigh different approaches to various challenges and provide recommendations.

Transition Strategy/Program Competition

If feasible based on the evidence review and design work, the contractor shall develop an open solicitation, in consultation with an expert panel and ED, inviting applications from researchers, publishers, and other organizations involved in the development and, if applicable, provision of training and support for the prioritized transition strategies/programs. They are referred to in the performance work statement as “training providers”. The solicitation will focus on the transition strategies/programs identified through the evidence review as both promising and suitable for the impact study. In response to the solicitation, interested parties will submit a description of their proposed transition strategy/program, any evidence of effectiveness, requirements for training and implementation, staff experience in delivering relevant training and/or support, and their capacity to deliver this training/support at the necessary scale. The contractor shall oversee the expert panelists’ review of proposals in order to identify up to four transition strategies/programs for inclusion in panel meetings in Washington, DC. After consideration of the COR and expert panel’s recommendations, the contractor shall select up to two transition strategies/programs for inclusion in the study.

D. IMPACT STUDY

ED may decide to conduct the impact study based on the feasibility and design work conducted during the base period of this contract. For the purpose of responding to this performance work statement, offerors shall assume that the optional study is a two-arm study with an option for a third-arm if there are two promising transition strategies/programs selected to test or if there are variations of the transition strategies/programs that may be of interest.

The two-arm study will address the following questions:

• What is the impact of the transition strategy/program on students’ educational attainment and employment outcomes? The study will examine the effectiveness of the transition strategy/program on students’ completion of high school completion, enrollment in postsecondary education/training, employment, and earnings.

• What are the effects of the transition strategy/program for policy-relevant subgroups of students?

• Was the transition strategy/program implemented with fidelity (and sufficient intensity)?

• How do the services/supports that students in the transition strategy/program receive differ from those received by students not in the program?

• Are there particular services/supports, as components of the transition strategy/program, that were more challenging to implement than others?

• What is the cost effectiveness of the transition strategy/program?

The three-arm option will address the following questions:

• What is the impact of the second transition strategy/program on student’s educational attainment and employment outcomes? How do the impacts of the two transition strategies/programs compare?

• What are the services/supports that students are receiving in each transition strategy/program?

• What is cost effectiveness of the second strategy/program? How does the cost effectiveness of the two transition strategies/programs compare??

E. ASSUMPTIONS CONCERNING DESIGN OPTIONS AND IMPACT STUDY

ED’s assumptions for the purpose of responding to the performance work statement are noted below:

Intervention: The contractor shall hold an open competition to select up to two transition strategies/programs to evaluate in this study.

Random assignment: ED assumes randomly assigning schools to the transition strategy/program or business-as-usual groups. Transition planning is required under IDEA by the time student reaches sixteen, and students may begin to receive transition services as early as 14. ED assumes implementation of the transition program would involve the whole school and its staff, and therefore school-level assignment would be most appropriate.

Anticipated sample size: ED assumes that 200 schools/sites will be recruited to participate in a two-arm study, of which half (100 schools) will receive training to implement the transition strategy/program. (For a three-arm study, ED assumes that an additional 100 schools/sites will need to be recruited and trained to implement a second transition strategy/program.)

Implementation support: ED assumes the contractor will coordinate and monitor the training for, and implementation of, the transition strategies/program in at least 100 schools per treatment, across at least two years. Critical to the impact study will be careful monitoring of the providers to ensure that the training and ongoing supports are carried out as planned and of the school staff’s implementation to ensure the transition program is implemented with fidelity (and sufficient intensity).

Data Collection: The study will include data collection on implementation and student experiences/outcomes for at least two years while the students attend high school and on student outcomes for at least two years after students’ transition out of high school. The study prioritizes the collection of administrative data to examine outcomes. This includes: school district data on student achievement, attendance, inclusion in the general education classroom, course taking and school completion; Rehabilitation Services Administration (RSA) data on participation in vocational rehabilitation (VR) services and benefits received; postsecondary enrollment data (from the National Student Clearinghouse and Federal Student Aid records); and receipt of supplemental security income and other federal benefits, and/or employment and earnings (Census; RSA VR individual case records), depending on the focus of the selected transition program(s).

I. SCOPE OF WORK

This section specifies the tasks and subtasks that the contractor shall perform.

BASE CONTRACT – Design and Feasibility

Task 1—Meeting with ED

1.1 Initial Meeting. The contractor’s project director and up to two key project staff members shall meet with the CO/CS, COR and other appropriate ED staff within 10 business days after the effective date of the contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. The meeting will be held in ED’s office in Washington, DC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to them. Within two weeks following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed as part of the contract.

1.2 Ongoing Communication. The contractor shall participate in regular meetings with the COR to update the COR on the progress of contract activities and to discuss any issues or anticipated problems and proposed solutions to those problems. ED assumes the meetings will occur by conference call every other week during the base period.

Deliverable: Kick-off meeting agenda

Memo summarizing initial meeting

Task 2—Design and Feasibility Technical Working Group (TWG)

2.1 Establish the Design and Feasibility TWG. The role of the Design and Feasibility TWG will be to advise the contractor on the work conducted as part of the base tasks. This includes the systematic evidence review approach and findings as well as the contractors’ determinations of the feasibility of conducting an effectiveness study and any design options.

The panel shall be comprised of up to ten individuals and should include highly qualified individuals with experience related to special education, transition supports for youth with disabilities, and evaluation methodology.

The contractor shall submit a list of potential TWG members to be reviewed and approved by the COR within two weeks of contract award. The contractor shall contact each proposed member and submit letters of commitment and documentation of consulting rates to the COR four weeks after COR approval of the TWG list.

The contractor shall evaluate the potential for conflicts of interest by assessing whether any individual considered for inclusion on the TWG has a bias or financial interest in the outcome of the study. The contractor shall have TWG members complete conflict of interest forms before appointment to the TWG.

2.2 Support TWG Meeting: Two in-person meetings of the TWG will be held during the base contract period (see Tasks 2.4 and 2.5). Three weeks prior to this meeting, the contractor shall provide the COR a draft agenda, a draft presentation slides, and a plan for a guided discussion of key topics and issues. Within two weeks after the meeting, the contractor shall submit a detailed summary of the meeting, including key observations and recommendations provided by TWG members, along with the contractor’s proposed actions.

2.3 Consult with TWG Members as Needed: Before the TWG meeting, and throughout the completion of the base period tasks, members of the TWG shall be available for ongoing consultation with the contractor and relevant ED staff. This may include having TWG members review products such as the draft evidence review plan (Task 3.1). The contractor shall provide copies of all reviews of products by the TWG to the COR within five business days of receipt by the contractor. If the TWG recommends any significant deviation from the proposed design/feasibility activities, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR and CO.

2.4 Convene TWG meeting. The contractor shall convene a one-day meeting of the TWG in order to discuss the contractor’s plans for and progress on the systematic evidence review (Task 3.1) and discuss the types of transition strategies/programs of policy interest to the field. The date for the meeting shall be set in consultation with the COR, but is expected to take place by January 2020. The meeting will take place in ED facilities in Washington, DC.

The contractor shall cover the travel, per diem, and honorarium expenses of the TWG members.

2.5 Convene Second TWG meeting. The contractor shall convene a second one-day meeting of the TWG for the purpose of reviewing preliminary findings from the systematic evidence review (Task 3.2) and the contractors’ recommendations regarding feasible design options for an effectiveness study (Task 5). The date for the meeting shall be set in consultation with the COR, but is expected to take place by February 2021. The meeting will take place in ED facilities in Washington, DC. The contractor shall cover the travel, per diem, and honorarium expenses of the TWG members.

Deliverable: List of potential TWG members

Letters of commitment and consulting rates

Draft TWG meeting materials

Summaries of TWG meetings

Task 3—Systematic Evidence Review

The contractor shall conduct the evidence review following the procedures and standards used by the WWC, including the use of WWC-certified study reviewers. This shall include determining policy-relevant categories transition strategies/programs to examine, developing a review protocol, searching the literature to identify relevant studies, screening in eligible studies, coding eligible studies, assessing the methodological quality of each eligible study, analyzing the results of eligible studies that meet WWC standards or meet standards with reservations, and synthesizing the results across studies.

The evidence report that is the culmination of this task shall address the question of what is known about the effects of transition programs/ strategies in improving key outcomes for youth with disabilities, such as educational attainment, employment, and earnings. The report shall include a description of the methodology of the review including its parameters and the state of the literature (number of studies identified in the scan for particular outcome measures, the number that meet WWC evidence standards). Most importantly, the report shall summarize the evidence from the studies meeting WWC standards and meeting WWC standards with reservations and identify gaps in the literature.

3.1 Create an Evidence Review Plan. Prior to conducting the review, the contractor shall submit a proposed plan for conducting the evidence review. The evidence review plan shall include parameters for the systematic evidence review, such as databases and other sources that the contractor will consult, search terms the contractor will use, and any limitations that will be imposed (e.g. in terms of publication years). The evidence review plan shall also include the contractor’s plans for systematically compiling the evidence base in an evidence report (e.g., report outline, proposed tables shells, summary sheets for each transition strategy/program, appendices).

The contractor shall submit the draft evidence review plan to the COR by December 6, 2019.

The COR will review the draft evidence review plan within two weeks. The contractor shall make revisions and submit a final evidence review plan to the COR by February 7, 2020. The final evidence review plan shall be fully responsive to COR comments on the draft plan.

3.2 Conduct the Review. Using the final evidence review plan, the contractor shall conduct the review and submit a draft evidence review report by December 18, 2020. The report length is expected to be no more than 15 pages, with information on the technical aspects of the review and full documentation of the transition strategies/programs, data, analytic methods, and results of studies included in appendices. The contractor shall also include two shorter summaries of the report:

• A one-pager. This shall be a stand-alone introduction to the report and contain enough information to form the basis for a press release.

• A four-pager. This shall be written to be suitable for ED program staff, reporters and policy makers (Congress and Hill staffers). It shall include key graphics and highlights.

The COR will review the draft report within three weeks. The contractor shall incorporate revisions to address COR comments and submit a revised report fully responsive to COR comments by March 15, 2021. The COR will submit the revised evidence review report to IES’ Standards and Review Office (SRO) for review. The COR will provide comments from peer review to the contractor within approximately two months. The contractor shall be responsive to SRO’s comments and make revisions to the report until it clears SRO. The contractor shall provide a web-ready (i.e. 508-compliant) version of the report to the COR (the final report) within two weeks after the report is cleared by SRO.

ED cannot estimate the level of effort required for this work because the number of relevant studies and portion that will need to be coded is unknown. For the purposes of scoping this work effort, offerors shall assume that a search of the literature may result in 12,000 studies which would need to be screened and 50 studies that would need to be fully coded. The actual amount may be more or less.

Deliverables: Draft evidence review plan

Final evidence review plan

Draft evidence review report

Revised evidence review report

Final evidence review report, web-ready

Task 4 - Feasibility of Obtaining Administrative Data

The contractor shall identify outcome data of policy interest for the study, in consultation with TWG members, ED, and other key stakeholders and explore the feasibility of obtaining these data from administrative sources.

4.1 Finalizing plans for obtaining outcome data. The contractor shall expand on and refine the plans submitted in the proposal in response to further information obtained through the design/feasibility work. The contractor shall submit a memo by January 17, 2020 listing the outcomes of policy interest, prioritized in terms of importance to the field. The memo shall include the administrative source for each outcome, describing the types of variables available from each source, and identifying challenges the contractor would face obtaining data from each source. The COR will review the list and select the administrative sources for the contractor to further explore for feasibility.

4.2 Assessing feasibility. The contractor shall discuss plans to assess feasibility for obtaining the prioritized outcome data identified under Subtask 4.1. The contractor, in consultation with the COR, shall determine the necessary steps to obtain data from each source and, as necessary, proceed with establishing agreements (MOUs, research application). The contractor shall update the outcome data memo (subtask 4.1) with the feasibility determination for each source by December 18, 2020.

Deliverable: Memo on possible outcome data

Revised memo addressing feasibility

Task 5 – Develop Design Report

The contractor shall develop a design report that identifies possible transition strategies/programs (and characteristics of those strategies/programs) to prioritize, provides specifications for the impact study, and options for consideration. The report shall include key study details such as a description of the transition strategy/program (and supports needed for implementation, if applicable), the likely service contrast, required sample size, the relevant outcome measures that will be feasible to obtain (Task 4), estimated costs, and other issues identified in Part I Section C of this PWS. The report shall also include proposed approaches to addressing the various issues anticipated. The contractor shall submit the draft design report to the COR by January 15, 2021.

The COR will review within two weeks. The contractor will submit a final design report by March 5, 2021.

Deliverables: Draft design report

Final design report

Optional Task 6 – Transition Strategy/Program Competition

Decision to exercise this optional task will be made by January 29, 2021, based on the evidence review (Task 3.2) and draft study design report (Task 5). To confirm feasibility, the contractor shall determine whether there are any potential training providers interested in participating in a randomized trial of their proposed transition strategy/program(s). The contractor shall establish a group of three experts in the area of transition services/programs promoting the educational attainment and employment of youth with disabilities. In consultation with the expert panel and ED, the contractor shall develop an open solicitation and selection criteria, based on the evidence review (Task 3) and design report (Task 5). The contractor shall publish the solicitation and contact training providers to ascertain their interest and ability to participate in a randomized trial of their transition strategy/program. In response to the solicitation, interested developers will submit a description of their proposed transition strategy/program, any evidence of effectiveness, requirements for training and implementation, staff experience in delivering relevant training and/or support, and their capacity to deliver this training/support at the necessary scale. The contractor shall oversee the expert panelists’ review of proposals in order to identify up to 4 developers for inclusion in panel meetings in Washington, DC. After consideration of the COR and expert panel’s recommendations, the contractor shall select up to two transition strategy(ies)/program(s) for inclusion in the impact study.

6.1 Expert Panel for the Competition. In consultation with the COR, the contractor shall establish an expert panel of three persons to advise on the selection of the transition strategies/programs. The panel shall include at least one expert serving on the design/feasibility technical working group (Task 2). The contractor shall submit a list of proposed expert panel members to the COR no later than two weeks after optional task is exercised. The COR will review and approve the list of proposed expert panel members within two weeks of receiving the list. The contractor shall contact each proposed member and obtain and submit letters of commitment and documentation of consulting rates to the COR within 2 weeks of COR/IES approval of the proposed expert panel list.

The contractor shall evaluate the potential for conflicts of interest by assessing whether any individual considered for inclusion on the TWG has a bias or financial interest in the outcome of the study. The contractor shall have TWG members complete conflict of interest forms before appointment to the TWG.

6.2 Advance Announcement. The contractor shall prepare a draft advance announcement for the upcoming request for proposals and the expected timelines for the proposal review process.

The contractor shall submit the draft announcement to the COR for review within 6 weeks after optional task is exercised. The COR will review within two weeks after receipt. The contractor shall release the revised advance announcement and request that relevant associations share the announcement.

6.3 Request for Proposal (RFP). Based on the contractor’s study design and in consultation with the COR, the contractor shall prepare a draft of the request for proposals inviting training providers to submit a description of their proposed transition strategy/program, any evidence of effectiveness, requirements for training and implementation, staff experience in delivering relevant training and/or support, and their capacity to deliver this training/support at the necessary scale. The contractor shall submit the draft request for proposals to the COR for review by 6 weeks after optional task is exercised. The COR will review within one week of receiving the draft request for proposals. The contractor shall make revisions and send the request for proposals, including the criteria by which the proposals shall be evaluated, to the expert panelists for review within one week of COR comment. The contractor shall make any necessary revisions within one week of receiving expert panel comments.

6.4 Solicitation. By June 2021, the contractor shall release the RFP and widely disseminate information about the competition to training providers for transition strategies/programs suitable for the impact study effort. The contractor shall invite training providers to submit a letter indicating their intent to bid. The contractor shall submit weekly reports updating the COR on the status of outreach efforts and the solicitation. The contractor shall set a deadline for submission of proposals by 6 weeks after RFP posting.

6.5 Review of Proposals. Within one day after closing the request for proposals, the contractor shall send proposals to expert panel members for review. The contractor shall request that, within one week, each member of the expert panel review the proposals and write a brief review of each proposal, including each provider’s strengths and weaknesses, along with recommendations of up to four transition strategies/programs for inclusion in the panel meetings. The contractor shall compile expert panel members’ reviews of proposals and conduct teleconferences with expert panelists as necessary in order to develop a ranking of proposals for inclusion in the panel meetings in DC. The contractor shall provide the compiled reviews and recommendations of these experts to the COR within two weeks after proposals are submitted.

The contractor shall arrange a meeting with expert panel members and training providers of up to four transition strategies/programs recommended by the expert panel. The meeting shall be one or two days and shall include time for each provider to present their plan and answer any questions from the panel, contractor, and/or ED. The contractor shall plan to arrange and pay for up to two staff for each provider to travel to DC, present on their plans, and answer questions that the expert panel, the contractor, or ED may have. Following the panel meeting, the contractor shall request that each panel member write a brief review of each proposed plan (including each program’s strengths and weaknesses) along with a recommendation of which transition strategies/programs to select for inclusion in the evaluation. The contractor shall compile expert panel members’ reviews of the presentations/proposals, conduct teleconferences with expert panelists as necessary, and consult with ED in order to select a transition strategy/program(s) for the evaluation within one week of receiving expert panel reviews. The contractor shall provide the reviews (and recommendation) of these experts to the COR within two weeks after the panel meeting.

The contractor shall negotiate with and secure the full cooperation of the recommended training provider(s). The contractor shall submit subcontracts with the training providers for the selected transition strateg(ies)/program(s) under Option Period 1 Task 12, if ED decides to move forward with impact study.

Deliverables: List of proposed expert panel members

Letters of commitment from proposed expert panel members

Draft advance announcement

Revised advance announcement

Draft RFP

Revised RFP

Final RFP

Expert panel review of solicitation and criteria

Expert panel review of proposals

Compilation of expert panel reviews for inclusion in panels

Expert panel recommendation of provider(s) after panel

Compilation of expert panel reviews for selection of provider(s)

Task 7—Monthly Reporting, Months 1 - 32

Throughout the base period (months 1 – 32), the contractor shall submit one copy of the following reports each month to the COR and CO:

• Monthly Progress Report/Exception Report. The contractor shall submit monthly progress reports due within 10 business days after the end of each month. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

• Monthly Manpower/Expenditure Reports. The contractor shall submit monthly expenditure reports due within 10 business days after the end of each month. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

• Security Clearance Spreadsheet. Along with the monthly progress and expenditure reports, the contractor shall submit a spreadsheet that lists each staff member assigned to work on the contract, the required security clearance level of their position (see Appendix A), and their progress toward attaining clearance. After contract award, the COR will provide a spreadsheet template for the selected contractor to use for this reporting purpose.

Deliverables: Monthly progress reports and security clearance spreadsheets

OPTION PERIOD 1 – Impact Study Recruitment and Initial Training

ED will make a decision on whether to conduct the impact study and exercise Optional Period 1 no later than June 15, 2021. If ED chooses to exercise Optional Period 1, the contractor shall complete tasks 8-13 to recruit sites and conduct the initial training.

Task 8 - Meeting with ED

8.1 Initial Meeting. The contractor’s project director and up to two key project staff members shall meet with the COR and other appropriate ED staff within 10 business days of ED exercising Optional Period 1. The purpose of the meeting will be to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to Option Period 1. The meeting shall be held in ED’s office in Washington, DC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to them.

Within two weeks following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed as part of the contract.

8.2 Ongoing Communication. The contractor shall participate in regular meetings with the COR to update the COR on the progress of contract activities and to discuss any issues or anticipated problems and proposed solutions to those problems. ED assumes the meetings will occur by conference call every other week during Optional Period 1.

Deliverable: Memo summarizing initial meeting

Task 9 – Prepare OMB Forms Clearance Packages, Develop Data Collection Forms, and Document Institutional Review Board Approval

9.1 Develop OMB package. The contractor shall prepare a forms-clearance package that describes the evaluation’s design and includes the necessary plans and instruments required for OMB approval of the impact study. The contractor shall ensure that the clearance package justifies the necessity for collecting the data and comprehensively responds to each required item in the instructions. In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions,

(3) a crosswalk between the research questions and the information to be collected, (4) description of sampling and recruiting procedures, (5) instructions to study participants, (6) description of and justification for incentives, (7) limitations of the study and its potential use for policy decisions, and (8) table shells indicating how data will be tabulated and analyzed

The contractor shall submit the OMB package to the COR by November 1, 2021. The COR will review the package within two weeks. The contractor shall submit the revised OMB clearance package to the COR by December 15, 2021. The COR will then submit the package to ICCD for submission to OMB.

The contractor shall submit the OMB package mindful that clearance requires a minimum of 120 days and how this impacts the agreed schedule of deliverables. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process.

9.2 IRB approval. The contractor shall coordinate its own Institutional Review Board (IRB) approval for the piloting, field-testing, and full-scale data collection with the IRB approval required by districts/sites. Specifically, the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Deliverables: Draft forms clearance package

Task 10 -- Impact Study TWG

10.1 Establish Impact Study TWG. The role of the Impact Study TWG will be to advise the contactor on the refinement of the impact study design , as well as its preliminary findings (under Task 29).