Draft Task Order 0001 PWS.docx

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation

DRAFT

Task Order 0001 Performance Work Statement (PWS)

EVALUATING CAREER PATHWAYS PROGRAMS AND STRATEGIES:

Understanding the Implications of a Key Shift in Perkins V Accountability:

Defining Who is a “Concentrator”

In awarding the basic ordering agreement (IDIQ) on “Evaluating Career Pathways Programs and Strategies,” the U.S. Department of Education (Department), Institute of Education Sciences (IES) will also award Task Order 0001 as described below. The tasks referred to are those outlined in the IDIQ Performance Work Statement. Task Order 0001 will have a 20-month period of performance. This task order will help the Department generate policy-relevant findings for the Congressionally-mandated National Evaluation of Career and Technical Education under Perkins V (NECTEP).

Background

One change to federal CTE policy in Perkins V affects which students states are held accountable for in the Perkins performance measurement system. Before Perkins V, state agencies had the flexibility to define a CTE “concentrator” as they preferred. Most, but not all, states chose to count students who completed 3 or more credits of CTE in a single CTE program area at the secondary level. However, to increase consistency and improve comparability so that annual state reporting could provide a national picture of CTE participation and outcomes, Perkins V required all states to define a concentrator as a student who completed at least 2 courses in the same CTE program or program of study.

This seemingly modest shift in the law could have significant implications for which students are included in reporting, and students who choose to more fully invest in CTE may be different from those who participate less or in less focused ways. Policy makers need this context to interpret Perkins V reporting and to judge the potential consequences of the change before Perkins reauthorization discussions begin again. This quick-turnaround study will use the 2019 NAEP transcripts, academic assessments, and background survey to simulate some of the possible effects of the change just as states were beginning to phase it in. Below are examples of the policy-relevant questions IES believes are worth addressing, although offerors are welcome to suggest revisions and additions:

1. In what ways does use of the new definition of “concentrator” change what would have been the depiction of secondary CTE students? Does the potentially more standardized but lower benchmark for a concentrator suggest that CTE was serving more or fewer students who are from historically disadvantaged backgrounds – e.g., students of color, from low-income households, who have a disability or are learning English? What types of locales have the greatest share of concentrators under the different concentrator definitions? What types of students are most likely to be dropped from or added to the pool of CTE concentrators under the new versus old definitions?

2. If students with a lesser investment in CTE are included in the accountability system, how might outcomes differ? For example, if concentrators must complete only 2 rather than 3 courses (or credits) in the same CTE program area or program of study, does average achievement in English/language arts, math, and science appear higher or lower? Would the gap between CTE concentrators and other students be narrower or more pronounced? Would proficiency or use of technology among concentrators look different?

Key Analytic Considerations to Be Addressed through This Task Order

An important aspect of this task order will be for the contractor, in consultation with the COR, to refine and finalize key aspects of the study, including its analytic focus, the methods to be used in carrying it out, and how results will be presented.

Examples of analytic considerations include:

· Descriptive versus predictive analyses. Study analyses might describe the characteristics of students included under differing definitions of a concentrator and the outcomes of those students or they might use predictive modeling to simulate differences under different possible definitions.

· Defining a course. While the Perkins V legislation provides a definition of CTE concentrator, the way in which states operationalize that definition can vary. For example, states can decide how many credit hours equate to a single course. The study will need to decide how it will define a course. Each state provided the Department with its definition of a CTE concentrator through the Perkins V state plan.[footnoteRef:2] [2: The Department’s Office of Career, Technical, and Adult Education (OCTAE) sponsors a website on Perkins accountability that includes a state’s definition of CTE concentrator in the Perkins state plan and the state’s accountability data using a data explorer to generate relevant tables of information: https://cte.ed.gov/dataexplorer/build ]

· How to report the analyses. There are potentially multiple analyses or ways of approaching the analyses from which summary conclusions will need to be drawn and presented. In addition, the report should have a particular focus, organization, and presentation style that can optimally communicate the key findings to policymakers.

Scope of Work

The specific work to be conducted under Task Order 0001 includes:

Task 1—Meetings

The contractor shall hold an initial meeting with the Department to discuss details regarding the tasks, scheduling activities, and other issues related to the task order. The contractor shall send the COR a detailed meeting agenda a week in advance of the meeting, identifying key topics for discussion. The meeting shall be held virtually by video conference. Following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each shall be addressed as part of the contract.

The contractor shall hold a study briefing for IES leadership and staff to solicit feedback on preliminary study findings after the key findings memo (Task 10) but before the TWG meeting (Task 2). The contractor shall prepare briefing materials including briefing slides and discussion questions to facilitate the presentation and solicit IES feedback.

Task 2—Technical Working Group (TWG)

The contractor shall establish a TWG comprised of 5-6 individuals that together have expertise relevant to evaluating CTE concentrator definitions, including those with expertise in CTE research and evaluation, CTE administration at the state and local levels, federal accountability systems, and education/workforce data quality. The contractor shall propose a list of primary and alternate TWG members to the COR, and upon COR approval, contact each prospective member and submit letters of commitment and documentation of consulting rates to the COR.

The contractor shall seek the TWG’s written feedback after a review of materials sent by the contractor consisting of the study design, including data analysis and reporting plans.

The contractor shall convene a half-day TWG meeting held virtually by videoconference to solicit the TWG’s feedback on methodological and policy reactions to preliminary study findings. The contractor shall prepare meeting materials including TWG meeting slides and discussion questions to facilitate the presentation and solicit TWG feedback.

Task 9—Data Collection

The contractor shall obtain an NCES restricted use license to obtain data from the 2019 administration of the National Assessment of Educational Progress (NAEP) to 12th grade students, including the 2019 NAEP High School Transcript Study (HSTS). Together, the data will contain a background survey, math, reading, and science assessments, and transcripts for the nationally-representative sample. Based on the approved analysis plan (under Task 10), the contractor shall extract data, as specified in the analysis plan.

Task 10—Data Analysis and Reporting

To guide the analysis and subsequent reporting the contractor shall add to and refine the report planning template submitted as part of their technical proposal, updating to reflect the initial kickoff meeting discussions and subsequent technical direction from the COR. Once the COR has approved the template, the contractor shall then develop an analysis plan, that details how each of the elements in the report planning template will be carried out, including the data sources that will be used, the key measures that will be constructed, and the specific analyses that will be conducted. After the COR approves the plan, the contractor shall conduct data analyses.

The contractor shall update the previously completed report planning template into a key findings memo once the analyses are completed. The memo will refine the motivation for the study and for the key research questions and answer the research questions and sub-questions using the approved analysis plan. The key findings memo, combined with the analysis plan, will provide the basis for the presentation at the TWG meeting described in Task 2. Based on the key findings memo and incorporating TWG and COR feedback, the contractor shall prepare a draft evaluation report. The contractor shall ensure that the report is written and organized so that it is accessible to policymakers and practitioners rather than academic researchers. The report length is expected to be no more than 15 pages, with information on the technical aspects of the study and any additional and relevant data tables included in appendices. The contractor shall also prepare a one-page stand-alone “study highlights” companion to each report (“1-pager”) to provide and contextualize the main findings.

The report and its appendices will be submitted by the COR for approval through IES’ Standards and Review Office (SRO). The contractor shall be responsive to SRO’s comments and make revisions until the report and appendices are cleared by SRO. After SRO clearance, the contractor shall provide web-ready (i.e., 508-compliant) versions.

Task 12—Monthly Communication with ED and Reporting

The contractor shall provide updates on task order activity and identify issues or problems primarily through email. The contractor shall also participate in biweekly conference calls with the COR.

DRAFT – Task Order 0001 Performance Work Statement The contractor shall submit monthly progress reports, expenditure reports, and security clearance lists. The contractor shall submit these to the COR for the given month no later than the end of the following month.

Schedule of Deliverables

All deliverables shall be submitted electronically to the COR. The contractor shall also electronically submit one copy of the monthly report and final deliverables to the CO.

Task Order 0001

Task 1 – Meetings

Task 2 – Technical Working Group (TWG)

Task 9 – Data Collection

Task 10— Data Analysis and Reporting

Task 12 – Monthly Communication with the Department and Reporting