PWS_6.5.18_FINAL_Presolicitation_Amend.pdf

Performance Work Statement

ASSESSING EVIDENCE OF EFFECTIVENESS

IN ADULT EDUCATION

I. INTRODUCTION

The U.S. Department of Education (ED) is conducting a Congressionally-mandated

National Assessment of Adult Education. As part of this assessment, ED will award a cost-reimbursement with fixed fee contract with a base period to identify promising adult education strategies and design approaches to evaluate them rigorously and options to conduct up to two impact evaluations. The specifics of these optional studies will be based on the feasibility and design work conducted during the contract’s base period, and thus these options are provided as illustrative examples. The contract will also include an optional task to provide logistical support for a National Assessment of Adult Education report to Congress.

The base contract period will be for 24 months. If the option tasks are executed, the contract will be extended by up to 75 months, for a total period of performance of up to

99 months.

The program office responsible for the evaluation will be the National Center for

Education Evaluation and Regional Assistance within the Institute of Education Sciences

(IES). IES will work in collaboration with the Office of Career, Technical, and Adult

Education (OCTAE), the office within ED that administers the Adult Education program.

A. Legislative Authority

Title II of the Workforce Innovation and Opportunity Act (WIOA) of 2014

, known as the Adult Education and Family Literacy Act (AEFLA), is the legislation currently governing the federal investment in adult education. Section 242(b) of WIOA mandates that ED carry out rigorous research and evaluation on effective adult education and literacy activities, one component of which is an independent evaluation of Title II programs and activities. Section 242(c) of WIOA also authorizes ED to examine various issues related to the effectiveness of adult education programs and services.

B. Background https://www.gpo.gov/fdsys/pkg/PLAW-113publ128/pdf/PLAW-113publ128.pdf https://www.gpo.gov/fdsys/pkg/PLAW-113publ128/pdf/PLAW-113publ128.pdf

Higher-level skills are increasingly required to succeed in the American workforce, yet many adults in the United States lack them. Over 25 million adults have not earned a high school diploma or its recognized equivalent.

Even among those with at least a secondary credential, a lack of proficiency with the English language can be a significant barrier to a family-sustaining income and to full integration as citizens. According to the American

Community Survey, over 20% of U.S. residents speak a language other than English at home.

Congress has sought to help individuals address these challenges – and the nation’s workforce development needs -- by providing funds for adult education, most recently through WIOA.

WIOA authorizes formula funding to State agencies that award competitive grants to local providers of adult education. Federal funds are for adult education activities designed to assist adults in: becoming literate and obtaining the skills necessary for employment; attaining a secondary school diploma and transitioning to postsecondary education and training; improving English skills and acquiring an understanding of

American civics, for English language learners. The core types of adult education programs are often referred to as adult basic education (ABE) (for those who lack basic skills in reading, writing, and mathematics), adult secondary education (ASE) (including training to earn a high school diploma or its recognized equivalency credential), and

English as a Second Language (ESL).

A key goal of the National Assessment of Adult Education is to respond to Congress’ interest in information on the effectiveness of adult education programs, and to inform the field about programs, activities and services that are supported by evidence. One way to fulfill this goal is to conduct a review of already completed studies of adult education approaches. This review will include impact studies of programs and strategies that are allowable uses of WIOA funds and will be conducted according to IES’ What Works

Clearinghouse (WWC) review procedures and standards.

In addition to reviewing existing evidence of effectiveness, ED seeks to expand the evidence base in adult education through the conduct of new studies examining the effectiveness of adult education programs, activities, or services. Conducting adult education impact studies is particularly challenging given a myriad of factors, such as learners’ persistence rates and limitations on the availability of outcome data. The selected contractor will therefore draw upon its knowledge of the field—coupled with findings from the evidence review—to identify promising programs, activities, or

Ryan, C., & Bauman, K. (2016). Educational Attainment in the United States: 2015. Current Population

Reports. https://www.census.gov/content/dam/Census/library/publications/2016/demo/p20-578.pdf.

https://factfinder.census.gov/faces/tableservices/jsf/pages/productview.xhtml?src=bkmk https://ies.ed.gov/ncee/wwc/Protocols#procedures https://www.census.gov/content/dam/Census/library/publications/2016/demo/p20-578.pdf https://ies.ed.gov/ncee/wwc/Protocols#procedures services to examine rigorously, and develop design options for feasible studies. ED may exercise contract options to conduct one or two studies that the contractor designs.

Findings from the systematic evidence review, as well as from any effectiveness studies conducted under this contract, will be part of the National Assessment of Adult

Education. The assessment also includes an implementation study of adult education that

ED is currently conducting under a separate contract.

Findings from the implementation study and from work conducted as part of this contract will be included in a report to

Congress that summarizes what has been in learned. The selected contractor for this contract shall provide logistical support to ED in developing that report.

C. Feasibility and Design Work

The selected contractor shall explore the feasibility of conducting effectiveness studies, and provide design options. The contractor shall identify policy-relevant adult education strategies and assess the feasibility of examining those strategies through a random assignment or regression discontinuity design. For the purpose of this work, the focus of an effectiveness study (i.e., the intervention) can be an adult education program or a particular service or activity that is provided within an adult education program. ED will consider any intervention that is an allowable use of WIOA funds, but is particularly interested in those that address the priorities policymakers introduced in passing WIOA.

ED will consider interventions that are innovative and already being implemented in a number of adult education programs, and hence an opportunistic rigorous examination of effectiveness could be conducted. Ideally, such interventions should at least have suggestive evidence of effectiveness (e.g. through correlational studies). ED will also consider interventions that may not be in widespread use across sites but for which there is reasonable evidence of effectiveness (e.g. through small-scale effectiveness studies) and that could be expanded to multiple sites. ED is not interested in evaluating interventions that will require development work.

Systematic Evidence Review

As one step in identifying possible interventions to study and assessing feasibility, the selected contractor shall conduct a systematic evidence review to identify studies of adult education interventions that address questions of effectiveness in improving key adult learner outcomes. These include the extent to which adult education interventions:

increase literacy skills, result in attainment of a high school diploma or its recognized equivalent, lead to transition to postsecondary education and training, and enhance employment and earnings. The review shall also include studies that examine the extent https://ies.ed.gov/ncee/projects/evaluation/pathways_wioa.asp https://ies.ed.gov/ncee/projects/evaluation/pathways_wioa.asp to which the provision of support services to those enrolled in adult education and literacy activities increases the rate of enrollment in, and successful completion of, such programs and the extent to which different types of providers (e.g. community college-based) improve the skills of adult learners.

The contractor shall use WWC-certified reviewers to review the identified studies using

WWC standards and procedures. The contractor shall synthesize the evidence from studies determined to meet WWC standards or meet WWC standards with reservations in a report that summarizes what is known about the effectiveness of adult education interventions. The report shall also identify where there are gaps in the effectiveness literature. The report will be reviewed by IES and publicly released as a product of the

National Assessment of Adult Education.

Research Questions

The contractor shall address the following research questions, bearing in mind that the adult education context presents a unique set of challenges for effectiveness research, including challenges with recruiting sites, recruiting adult learners, and obtaining reliable and valid outcome data that cover most of the study participants:

1. What is known about the effectiveness of adult education programs, activities and services? ED is interested in examining the evidence of effectiveness of adult education programs, activities, and services in improving key outcomes identified in WIOA.

2. What adult education programs, activities and/or services are feasible to examine with an impact study? ED is interested in exploring the feasibility of conducting policy-relevant impact studies that examine the extent to which adult education interventions affect key outcomes.

3. What are design options for impact studies? ED is interested in design specifications for up to five impact studies, including answers to the following:

What is the intervention? Are there important variations of the intervention that are policy relevant to study and could be varied within the study design?

What are the characteristics of adult learners expected to most benefit from the intervention or any of its variations in particular?

What is the most feasible and policy-relevant counterfactual?

If a random assignment design is possible, what is the unit of assignment? If the intervention is to be studied through a regression discontinuity design, what is the assignment measure upon which an individual’s group status will be based?

What are the expected outcomes, and what are the most credible measures to capture the outcomes? What national or state administrative databases could be used to reduce the burden and costs of data collection and what are their strengths and weaknesses over study-administered measures?

For what minimum detectable effect size should the study be powered?

How many sites and adult learners would be needed to detect an impact of this magnitude?

What recruitment activities would be required to achieve such a sample size?

What, if any, implementation supports would be needed in order for the study to be a fair test?

D. Effectiveness Studies

Based on the feasibility and design work conducted by the selected contractor as part of the base period of this contract, ED may elect to conduct up to two effectiveness studies.

These studies are identified as Option 1 and Option 2 within the scope of work (see

Section II, Parts A and B). Strictly for the purpose of responding to this performance work statement, offerors shall assume that the Option 1 study will focus on an evaluation of integrated education and training (IET) programs, as defined in Title II of WIOA. Also strictly for the purpose of responding to this performance work statement, offerors shall assume that the Option 2 study will focus on computer-based tutoring in literacy and language skills for adult ESL learners. The tutoring intervention will be an off-the-shelf product that study sites are willing to pilot.

Option 1 Study

In this example study, the contractor would examine the effectiveness of IET programs, through a study that addresses the following research questions:

What is the impact of integrated education and training (IET) programs on adult learners’ educational attainment and employment outcomes? The study will examine the effectiveness of IET in increasing adults’ educational attainment, such as attainment of a high school diploma or its recognized equivalent and/or a professional credential. It will also examine the extent to which IET increases the likelihood of employment and its impact on wages.

Are there particular subgroups of adult learners for whom IET programs are effective? For instance, if feasible, the study will examine the effectiveness of IET programs for learners in particular educational functioning levels in ABE, ASE, and ESL programs.

Are there particular features of IET programs that appear promising? For instance, the study will examine the effectiveness of IET programs that vary in duration and intensity.

ED’s assumptions regarding the design of the study are presented below:

Site Recruitment: The selected contractor will recruit adult education sites that already offer an IET program and that have a managed enrollment policy, rather than open enrollment. To facilitate random assignment (described below), selected sites will be those for which there is more interest in the program than can be accommodated

(oversubscription) or there could be oversubscription with some minimal outreach to recruit participants from the contractor or from the program staff with contractor help.

Programs will be selected to include some natural variation in terms of program duration and intensity; ED assumes that recruited programs will range from 3 to 18 months in length.

Random assignment of adult learners: Recruited sites must agree to allow the evaluation contractor to randomly assign about half of the adults who seek to enroll in the site’s IET program to be given an opportunity to participate in the program (treatment adults). The remaining adults who seek to enroll will not be given an opportunity to participate in the program or will be given a delayed opportunity to participate (control adults). Depending on the main policy question ED decides to pursue with this study and the feasibility of doing so, the control adults may be given an opportunity to participate in only the adult education component and not the occupational training component of the IET program.

Service Contrast: Although control adults will not be provided an immediate opportunity to participate in the full IET program, they may access other services in the community, including other services provided by the recruited site.

Anticipated sample size: ED assumes that 30 sites will be recruited to participate in the study. Each site will contribute approximately 300 learners to the sample, for a total of

9,000 learners across the sites. ED anticipates that two cohorts of learners will be required to meet this sample size.

Data Collection: The study will require interviews of program directors and review of program materials in order for the study’s report to describe key features of each site’s

IET program. (Some—though not all—of this information may be available from the contractor’s site recruitment work). The study will also require the collection of individual-level data. This will include background information on adult learners (e.g.

demographics and other baseline data typically captured by sites during intake and available through program records): social security numbers; information on program attendance and exit for those assigned to the treatment group (also likely obtained from program records). Outcome data on employment and earnings will be obtained from the

National Directory of New Hires (NDNH).

Data will be obtained for 6 months and 12 months following the end of the IET program in which the learner attempted to enroll.

Outcome data on educational attainment will ideally be obtained through administrative records, such as those of the high school equivalency test vendors, the National Student

Clearinghouse, and ED’s Office of Federal Student Aid. The study will also include the option of conducting a survey of adult learners in the treatment and control groups in order to obtain data on the service contrast, educational attainment, employment status, and earnings. ED will exercise the option if it is feasible, including that the survey can be conducted with a response rate of at least 85 percent. If the option is exercised, the survey will be administered one time, approximately 6 months after the end of the IET program for which the participant attempted enrollment.

Option 2 Study

In this example study, the contractor would examine the effectiveness of a computer-based tutoring intervention. The study addresses the following research questions:

To what extent do adult English language learners utilize the tutoring program?

What is the impact of the tutoring program on learners’ persistence in their adult education program?

What is the impact on learners’ literacy and language skills?

ED’s assumptions regarding the design of the study are presented below:

The NDNH provides quarterly earnings from state Unemployment Insurance (UI) records, including data from some employers not included in the UI program (e.g., the federal government). The NDNH is maintained by the Office of Health and Human Services’ Office of Child Support Enforcement. ED is an authorized user of the database and will coordinate with HHS to obtain NDNH data for study participants.

Intervention: The contractor shall hold an open competition to select one or more tutoring interventions that will be the focus of this demonstration study. Priority will be given to those that have features shown to be associated with key outcomes (including through studies conducted with K-12 populations) or features that are believed by experts to be important. The intervention will be intended to supplement instructors’ classroom-based instruction; learners will be able to use the intervention outside of the classroom, at a time and place that is convenient to them.

Site Recruitment: The selected contractor will recruit adult education sites to participate in the study. Selected sites will be those who have a managed enrollment policy, are willing to implement the intervention and whose business as usual services provide a policy relevant service contrast.

Random assignment of sites: Recruited sites must agree to allow the evaluation contractor to randomly assign them to a study group. One group will consist of sites in which all learners and instructors in study classrooms will be given user credentials to access the tutoring software. Instructors at these sites will also receive training on how to use the software and how to introduce and encourage its use among learners. The second group will be sites in which learners and instructors in study classrooms will not be introduced to the tutoring intervention nor given user credentials.

Anticipated sample size: ED assumes that sixty sites will be recruited to participate, with each site contributing two cohorts of two classrooms to the study. Each class is estimated to have fifteen learners, for a total of approximately 3,600 study participants.

Data Collection: The study will require the collection of individual-level data. This includes: background information on adult learners (e.g. demographics and other baseline data typically captured by sites during intake); social security numbers; information on program attendance and exit (also likely obtained from program records), information on the extent to which participants at sites assigned to the treatment group utilized the intervention (ED presumes this information will be available through reports generated from the tutoring program); outcome data on study participants’ language and literacy skills (from program records as well as from a study-administered assessment). The study will also include the option to collect employment, wage, and educational attainment outcomes from administrative records approximately twelve months after the end of the intervention.

E. Period of Performance

Recruiting sites with a managed enrollment policy will increase the feasibility of the study meeting learner sample size requirements, with those assigned to the treatment group having the opportunity to experience the intervention for a sufficient period of time before outcome data is collected. Sites with an open enrollment policy may also be considered, if feasible.

The contract shall consist of a set of base period tasks along with three sets of optional tasks. The base tasks include a systematic evidence review along with additional feasibility and design work for the effectiveness studies. Option I tasks are for the conduct of an effectiveness study, should one prove feasible. Option 2 tasks are for the conduct of a second effectiveness study, if feasible. Option 3 tasks are for logistical support to ED with preparing a written report that will summarize and integrate the findings from this contract (systematic evidence review and effectiveness studies) with the findings from the implementation study contract. The table below shows how the optional tasks, if executed, are anticipated to effect the total period of performance.

Base Tasks Option 1

Tasks

Option 2 Tasks Option 3 Task

Sept. 2018 –

Sept. 2020

(months 1- 24)

March 2020 –

Oct. 2025

( 18-85)

Sept. 2021 –

Dec. 2026

(Months 35-99)

Sept. 2025 – Dec.

(Months 84 - 99)

II. SCOPE OF WORK

BASE CONTRACT – Feasibility and Design

Task 1—Meeting with ED

1.1 Initial Meeting. The contractor’s project director and up to two key project staff members shall meet with the CO/CS, COR and other appropriate ED staff within 10 business days after the effective date of the contract award to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to the contract. The meeting will be held in ED’s office in Washington, DC.

The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to them. Within two weeks following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed as part of the contract.

1.2 Ongoing Communication. The contractor shall participate in regular meetings with the COR to update the COR on the progress of contract activities and to discuss any issues or anticipated problems and proposed solutions to those problems. ED assumes the meetings will occur by conference call every other week during the first 18 months of the contract and one time per month for the remaining six months of the base period.

Deliverable: Memo summarizing initial meeting

Task 2—Design and Feasibility Technical Working Group (TWG)

2.1 Establish the Design and Feasibility TWG. The role of the Design and Feasibility

TWG will be to advise the contractor on the work conducted as part of the base tasks.

This includes the systematic evidence review approach and findings as well as the contractors’ determinations of the feasibility of conducting effectiveness studies and any design options.

The panel shall be comprised of up to eight individuals and should include highly qualified individuals with experience related to adult education and evaluation methodology. The panel shall include a subset of individuals who are participating in the TWG for IES’ National Assessment of the Implementation of Adult Education under the Workforce Innovation and Opportunity Act.

The contractor shall evaluate the potential for conflicts of interest by assessing whether any individual considered for inclusion on the TWG has a bias or financial interest in the outcome of the study. The contractor shall have TWG members complete conflict of interest forms before appointment to the TWG.

The contractor shall submit a list of potential TWG members to be reviewed and approved by the COR within two weeks of contract award. The contractor shall contact each proposed member and submit letters of commitment and documentation of consulting rates to the COR four weeks after COR approval of the TWG list.

2.2 Support TWG Meeting and Consult with TWG Members as Needed: Two in-person meetings of the TWG will be held during the base contract period (see Tasks 2.3 and

2.4). Three weeks prior to this meeting, the contractor shall provide the COR a draft agenda, a draft slide deck, and a plan for a guided discussion of key topics and issues.

Within two weeks of the meeting, the contractor shall submit a detailed summary of the meeting, including key observations and recommendations provided by TWG members, along with the contractor’s proposed actions.

Before the TWG meeting, and throughout the completion of the base period tasks, members of the TWG shall be available for ongoing consultation with the contractor and relevant ED staff. This may include having TWG members review products such as the draft evidence review plan (Task 3.1). The contractor shall provide copies of all

The COR will provide the list of TWG members on the implementation study to the selected contractor upon contract award.

reviews of products by the TWG to the COR within five business days of receipt by the contractor. If the TWG recommends any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the recommendation of TWG members without the written approval of the COR and CO.

2.3 Convene First TWG meeting. The contractor shall convene a one-day meeting of the

TWG in order to discuss the contractor’s plans for and progress on the systematic evidence review (Task 3.1) and potential effectiveness studies for which the contractor may further explore feasibility (see Task 4). The date for the meeting shall be set in consultation with the COR, but is expected to take place by January 2019.

The meeting will take place in ED facilities in Washington, DC. The contractor shall cover the travel, per diem, and honorarium expenses of the TWG members.

2.4 Convene Second TWG meeting. The contractor shall convene a second one-day meeting of the TWG for the purpose of reviewing preliminary findings from the systematic evidence review (Task 3.2) and the contractors’ recommendations regarding feasible effectiveness studies (Task 6). The date for the meeting shall be set in consultation with the COR, but is expected to take place by December 2019. The meeting will take place in ED facilities in Washington, DC. The contractor shall cover the travel, per diem, and honorarium expenses of the TWG members.

Deliverable: List of potential TWG members

Letters of commitment and consulting rates

Draft meeting materials

Summaries of TWG meetings

Task 3 – Prepare OMB Forms Clearance Packages

The contractor will need to collect data from adult education programs in order to determine the feasibility of each of the up to seven studies selected by the COR (see Task

6). This will include interview data to determine whether sufficient sample exists for impact studies and to identify design options. The contractor shall prepare a forms-clearance package in order to receive approval from OMB for this effort. The contractor shall ensure that the clearance package justifies the necessity for collecting the data and comprehensively responds to each required item in the instructions, including: (1) the study mandate and objectives, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, (7) limitations of the study and its potential use for policy decisions, and (8) table shells indicating how data will be tabulated and analyzed

The contractor shall submit the OMB package to the COR by October 15, 2018. The

COR will review the package within two weeks. The contractor shall submit the final

OMB clearance package to the COR by November 5, 2018. The COR will then submit the package to RIMS for submission to OMB.

The contractor shall submit the OMB package mindful that clearance requires a minimum of 120 days and how this impacts the agreed schedule of deliverables. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process.

Deliverables: Draft forms clearance package

Final forms clearance package

Task 4—Systematic Evidence Review

The contractor shall conduct the evidence review following the procedures and standards used by the WWC, including the use of WWC-certified study reviewers. This shall include determining policy-relevant categories of adult education interventions to examine, developing a review protocol, searching the literature to identify relevant studies, screening in eligible studies, coding eligible studies, assessing the methodological quality of each eligible study, analyzing the results of eligible studies that meet WWC standards or meet standards with reservations, and synthesizing the results across studies.

The evidence report that is the culmination of this task shall address the question of what is known about the effectiveness of adult education programs and strategies in improving key outcomes such as literacy skills, educational attainment, enhanced employment and earnings, as well as ancillary outcomes such as persistence and program completion. The report shall include a description of the methodology of the review including its parameters and the state of the adult education effectiveness literature (number of studies identified in the scan for particular outcome measures, the number that meet WWC evidence standards). Most importantly, the report shall summarize the evidence from the studies meeting WWC standards and meeting WWC standards with reservations and identify gaps in the literature.

4.1 Create an Evidence Review Plan. Prior to conducting the review, the contractor shall submit a proposed plan for conducting the evidence review. The evidence review plan shall include parameters for the systematic evidence review, such as databases and other sources that the contractor will consult, search terms the contractor will use, and any limitations that will be imposed (e.g. in terms of publication years). The evidence review plan shall also include the contractor’s plans for systematically compiling the evidence base in an evidence report (e.g., report outline, proposed tables shells, intervention summary sheets, appendices).

The contractor shall submit the draft evidence review plan to the COR by December

15, 2018. The COR will review the draft evidence review plan within two weeks.

The contractor shall make revisions and submit a final evidence review plan to the

COR by January 15, 2019. The final evidence review plan shall be fully responsive to

COR comments on the draft plan.

4.2 Conduct the Review. Using the final evidence review plan, the contractor shall conduct the review and submit a draft evidence review report by January 15, 2020.

The COR will review the draft memo within three weeks. The contractor shall incorporate revisions to address COR comments and submit a revised report fully responsive to COR comments by March 1, 2019. The COR will submit the revised evidence review report to IES’ Standards and Review Office (SRO) for review. The

COR will provide comments from peer review to the contractor within approximately two months. The contractor shall be responsive to SRO’s comments and make revisions to the report until it clears SRO. The contractor shall provide a web-ready

(i.e. 508-compliant) version of the report to the COR (the final report) within two weeks after the report is cleared by SRO.

ED cannot estimate the level of effort required for this work because the number of relevant studies and portion that will need to be coded is unknown. For the purposes of scoping this work effort, offerors shall assume that a search of the literature may result in 800 studies which would need to be screened and 20 studies that would need to be fully coded. The actual amount may be more or less.

Deliverables: Draft evidence review plan

Final evidence review plan

Draft evidence review report

Revised evidence review report

Final evidence review report, web-ready

Task 5—Identify Possible Adult Education Interventions

The contractor shall identify types of interventions of policy interest to the field. The contractor shall coordinate with state and local providers to conduct interviews to identify and prioritize types of interventions for study with a random assignment or regression discontinuity design. The contractor shall conduct the interviews via phone or in-person when efficient to do so (e.g. instances when adult education state directors and/or local providers are gathered at national or regional conferences). Offerors shall assume that study staff will travel to two adult education conferences for this purpose and that individual phone call interviews will be conducted with up to nine state or local directors.

The contractor shall use the information gathered from these interviews, along with preliminary knowledge from the systematic evidence review (Task 4) and the contractor’s general knowledge of the field, to identify possible interventions to study that are policy relevant, would make a significant contribution to the adult education effectiveness literature, and that have some indication of feasibility. The contractor shall submit a memo that includes up to fifteen possible interventions to the COR by

December 1, 2018. The memo shall prioritize the interventions in terms of importance to the field and include short descriptions of the interventions and identified challenges the contractor would face in conducting impact studies of the interventions. The COR will review the list and select up to ten possible interventions for the contractor to further explore for feasibility (see Task 6).

Deliverable: Memo on possible interventions

Task 6 –Explore Feasibility

6.1 Collect data. The contractor shall conduct interviews with up to fifty adult education programs in order to explore the feasibility of conducting the studies selected by the

COR (see Task 4). The interviews shall be designed to collect information to:

determine sites’ suitability as potential sites for impact studies; inform study design options; identify potential challenges to rigorous evaluation. ED anticipates that the interviews will occur beginning in Spring 2019.

6.2 Summarize findings. By September 15, 2019, the contractor shall submit a memo to the COR that summarizes the information learned under Task 6.1 and makes recommendations regarding impact studies that appear feasible. The COR will review the memo within two weeks and provide feedback to the contractor regarding which studies should be pursued within the design report (Task 7).

Deliverables: Feasibility memo

Task 7 – Develop Design Report

The contractor shall develop a design report that provides specifications for up to five impact studies. For each study, the report shall include key study details such as a description of the intervention (and supports needed for implementation, if applicable), the likely service contrast, required sample size, the relevant outcome measures, estimated costs, and other issues identified in Part I Section C of this PWS. The contractor shall submit the draft design report to the COR by December 15, 2019. The

COR will review within two weeks. The contractor will submit a final design report by

January 15, 2020.

Deliverables: Draft design report

Final design report

Task 8—Monthly Reporting, Months 1 - 24

Throughout the base period (months 1 – 24), the contractor shall submit one copy of the following reports each month to the COR and CO:

Monthly Progress Report/Exception Report. The contractor shall submit monthly progress reports due within 10 business days after the end of each month.

The reports shall summarize the major activities and accomplishments for the reporting period. In addition, they shall provide information for each project task regarding significant findings and events, problems encountered, and staff use.

The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions which may be needed from ED. If there are no exceptions, the reports shall state that there are no exceptions. If there are exceptions to the management plan, the contractor shall describe the plan for resolving the problems.

Monthly Manpower/Expenditure Reports. The contractor shall submit monthly expenditure reports due within 10 business days after the end of each month. These reports, prepared and signed by the project director, shall summarize the actual personnel assignments for the month just completed, showing for each staff member the hours charged by task. The report shall project similar assignment information for the upcoming month. The reports shall also exhibit expenditures, segregating project costs by individual and by task, and specifying for all travel the locations, duration, and personnel for each trip.

Security Clearance Spreadsheet. Along with the monthly progress and expenditure reports, the contractor shall submit a spreadsheet that lists each staff member assigned to work on the contract, the required security clearance level of their position, and their progress toward attaining clearance. After contract award, the COR will provide a spreadsheet template for the selected contractor to use for this reporting purpose.

Deliverables: Monthly progress reports and security clearance spreadsheets

A. OPTION 1 TASKS

ED will make a decision on whether to exercise Option 1 no later than March 31, 2020.

If ED chooses to exercise Option 1, then the contractor shall complete optional tasks 9-

17.

Task 9 – Study 1 Meeting with ED

9.1 Initial Meeting for Study 1. The contractor’s project director and up to two key project staff members shall meet with the COR and other appropriate ED staff within

10 business days of ED exercising Option 1. The purpose of the meeting will be to discuss details regarding the tasks outlined in the proposal, scheduling activities, and other issues related to Study 1. The meeting shall be held in ED’s office in

Washington, DC. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to them. Within two weeks following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed as part of the contract.

9.2 Ongoing Communication. The contractor shall participate in regular meetings with the COR to update the COR on the progress of contract activities related to Study 1 and to discuss any issues or anticipated problems and proposed solutions to those problems. ED assumes the meetings will occur by conference call every other week throughout the period of performance for Study 1.

Task 10—Study 1 Technical Working Group

10.1 Establish the Study I TWG. The role of the Study 1 TWG will be to advise the contactor on the design of Study 1, as well as its preliminary findings. The panel shall be comprised of eight members who, as a group, have expertise in adult education issues and in research methodology that is relevant to the focus of Study 1. ED anticipates that, at a minimum, there will be some overlap in membership between the

Study 1 TWG and the Design and Feasibility TWG (Task 2). However, participation on the Design and Feasibility TWG is not a prerequisite for participation on the Study

1 TWG.

whether any individual considered for inclusion on the TWG has a bias or financial approved by the COR within two weeks of award of Option 1. The contractor shall of consulting rates to the COR within four weeks of approval.

10.2 Support TWG Meetings and Consult with TWG Members as Needed: Two in-person meetings of the TWG will be held (see Tasks 10.3 and 10.4) as well as one meeting conducted via webinar (see Task 10.5). The two in-person meetings will focus on the design of Study 1 and early preliminary findings from Study 1

(respectively). The webinar meeting will focus on preliminary follow-up findings from Study 1. Three weeks prior to each meeting, the contractor shall provide the

COR a draft agenda, a draft slide deck, and a plan for a guided discussion of key topics and issues. Within two weeks of each meeting, the contractor shall submit a detailed summary of the meeting, including key observations and recommendations provided by TWG members and the contractor’s proposed actions. Proposed actions cannot go forward unless approved by the COR.

Before the TWG meetings, and throughout the completion of Study 1, members of the TWG shall be available for ongoing consultation with the contractor and relevant

ED staff. This may include having TWG members review study products such as data collection forms. The contractor shall provide copies of all reviews of products by the TWG to the COR within five work days of receipt by the contractor. If the TWG recommends any significant deviation from the proposal, study design, instrumentation, data collection, or data analysis, the contractor shall submit the recommendation to the COR for approval prior to implementation. The contractor shall not make any significant changes in the study based solely upon the

10.3 Convene First Study 1 TWG Meeting. The contractor shall convene one one-day meeting of the TWG. The meeting shall focus on from the design of Study 1. The date for the meeting shall be set in consultation with the COR, but is expected to be in

Fall 2020. The meeting will take place in ED facilities in Washington, DC. The contractor shall cover the travel, per diem, and honorarium expenses of the TWG members.

10.4 Convene Second Study 1 TWG Meeting. The contractor shall convene a one-day meeting of the TWG focused on early preliminary findings from Study 1. These are the findings that will be included in the Study Report (see Task 15.2). The date for the meeting shall be set in consultation with the COR, but is expected to be Spring

2024. The meeting will take place in ED facilities in Washington, DC. The contractor shall cover the travel, per diem, and honorarium expenses of the TWG members.

10.5 Convene Study 1 TWG Webinar. The contractor shall convene one half-day meeting of the TWG. The meeting shall focus on the follow-up findings from Study

1. These are the findings that will be included in the Study Brief (see Task 15.3). The date for the meeting shall be set in consultation with the COR, but is expected to be in

Fall 2024.

Deliverable: List of potential TWG members

Task 11 – Site Recruitment

The contractor shall develop and implement a strategy for identifying and recruiting adult education providers that is appropriate for effectively and efficiently addressing the study’s research questions. The contractor shall recruit a sufficient number of sites to meet the Study 1 sample size assumptions provided in Part I Section D of this PWS.

As part of the recruitment process, the contractor shall prepare notification letters and information packets for the prospective study sites. The notification packet shall include a non-technical brochure describing the study that is suitable for distribution to a broad audience of policy makers and educators. The contractor shall include in the letters and information packets general information on the study as well as specific information on the importance of the study, its purposes, products, timelines, and the expected benefits to be derived from participation in the study. The contractor shall submit draft letters and all other notification materials to the COR for review by July 1, 2020. Once approved by the COR, the contractor shall send the notification materials to providers and follow-up with the necessary telephone calls to secure the cooperation of grantees in the evaluation.

The contractor shall also host up to four webinars to provide information regarding the study to interested sites. ED anticipates that the contractor will advertise the webinars through state adult education agencies or regional consortiums of adult education providers. The webinars, which ED anticipates will be identical in content, shall be scheduled to maximize provider participation and shall highlight the requirements of the study as well as the benefits to grantees. The contractor shall submit draft webinar slides to the COR two weeks prior to the first scheduled webinar and revised webinar slides at least one week prior to the first scheduled webinar date.

The contractor shall conduct site visits to adult education programs as needed, in order to further explore site suitability and secure the cooperation of eligible sites. ED assumes that the contractor will conduct up to 20 of such visits.

The contractor shall consult with the COR to finalize the list of sites for the study, and subsequently submit memoranda of understanding for those sites to the COR by July 15, 2021. ED’s assumption is that study implementation will begin in Fall 2021.

As part of the recruitment process, the contractor shall comply with: The Privacy Act of

1974, P.L. 93-579, 5 USC 552 a; the “Buckley Amendment,” Family Educational and

Privacy Act of 1974, 20 USC 1232 g; The Freedom of Information Act, 5 USC 522; and related regulations, including but not limited to: 41 CFR Part 1-1 and 45 CFR Part 5b and, as appropriate, the Federal common rule or ED’s final regulations on the protection of human research participants.

During recruitment, the contractor shall submit to the COR progress reports on a biweekly basis.

Deliverables: Site recruitment materials

Draft webinar slides

Final webinar slides

Memoranda of understanding from sites

Progress reports on site recruitment

Task 12 – Prepare OMB Forms Clearance Packages and Document Institutional

Review Board Approval

12.1 Develop OMB package. The contractor shall prepare a forms-clearance package that describes the evaluation’s design and includes the necessary plans and instruments required for OMB approval of Study 1. The contractor shall ensure that the clearance package justifies the necessity for collecting the data and comprehensively responds to each required item in the instructions. In addition, the contractor shall ensure that the forms clearance package includes: (1) the study mandate and objectives, (2) the research questions, (3) a crosswalk between the research questions and the information to be collected, (4) instructions to study participants, (5) description of and justification for incentives, (6) discussion of questions likely to be deemed “sensitive” by OMB, (7) limitations of the study and its potential use for policy decisions, and (8) table shells indicating how data will be tabulated and analyzed

The contractor shall submit the OMB package to the COR by November 1, 2020.

The COR will review the package within two weeks. The contractor shall submit final OMB clearance package to the COR by December 15, 2020. The COR will then submit the package to RIMS for submission to OMB.

The contractor shall submit the OMB package mindful that clearance requires a minimum of 120 days and how this impacts the agreed schedule of deliverables. The contractor shall be prepared to make up to two revisions to forms and justification as required during the approval process.

12.2 IRB approval. The contractor shall coordinate its own Institutional Review Board

(IRB) approval for the piloting, field-testing, and full-scale data collection with the

IRB approval obtained by districts from their own institutions. Specifically, the contractor is responsible for ensuring that all aspects of the program have been approved by the appropriate IRBs, including its own IRB.

Task 13— Intake and Random Assignment of Learners

The contractor shall provide support to sites as needed with generating interest in IET program enrollment among learners, in order that sites will have a sufficient number of learners for the contractor to randomly assign to condition. ED assumes that a total of five study sites will require such support.

The contractor shall design and implement a protocol for study intake and random assignment of learners within sites. ED assumes that recruited sites will have a managed enrollment process for their IET programs, in which case enrollment will happen over a short window of time that occurs prior to the start of the program. Learners who are eligible to receive adult education services under WIOA and seek to enroll in sites’ IET programs shall be randomly assigned by the contractor to the treatment group or the control group. The contractor shall obtain information on all study participants’ background characteristics from programs’ intake records. Such information shall include, but not necessarily be limited to, learner demographics, educational background, and functioning levels. The contractor shall also collect learners’ social security numbers in order to match with administrative databases such as the NDNH.

During study intake and random assignment periods, the contractor shall provide weekly progress updates to the COR. The updates shall provide information on the number of learners who consent to participate in the study, by site and by treatment group.

During implementation (which ED assumes will take place between Fall 2021 and Spring

2…