Peer Review - SOW.doc

Increasing the Effectiveness of Employment and Training Administration Activities Through Peer Review of Research Products and Relevant Staff Training

STATEMENT OF WORK

Background

Increasingly, the Employment and Training Administration (ETA) recognizes the growing need to raise the effectiveness of its programs by incorporating evidence-based practices in its policy development, program designs, provision of technical assistance and other program-related activity. Incorporating evidence-based practices in each stage of program development will require new ways of thinking and conceptualizing and will require guidance for ETA staff on how to proceed given the limited staff experience with evidence-based programming.

Coincident to this evolving recognition, ETA desires that its sponsored research, demonstrations, and evaluations be peer reviewed to help ensure that they produce scientifically valid evidence. In addition, ETA wishes to obtain a peer review of its evaluation program. This review will be similar in nature to the review process for individual research, demonstration and evaluation projects.

Under this contract, the contractor will assist ETA by performing and organizing peer reviews of selected projects, conducting a review of ETA’s evaluation projects and annual spending plans and creating and delivering a program staff briefing that will impart an understanding of evidence-based practices and how they apply to the implementation of ETA’s strategic workforce development priorities.

Tasks The evaluation will encompass a total of one base and four option years, provided the government exercises the complete set of options detailed below. (Projected timelines are inclusive of all contractual options.)

The contractor shall perform the following tasks:

Base year:

1. Peer review of the specific study designs. Following development of draft design plans for specific projects, contractor will conduct a peer review to assess whether the proposed study design contains any serious flaws. Each review will be conducted by two contractor staff and one external, respected social science researcher selected in consultation with ETA. The review will not seek to provide feedback on all aspects of the proposed study design but rather will concentrate on identifying any serious methodological shortcomings.

2. Peer review of specific project’s interim and/or final reports. Once each study is underway, contractor will use a peer review process to assess whether the study is producing (or for a final report, has produced) valid evidence. Each interim report will be reviewed by two contractor staff. Each final report will be reviewed by two contractor staff and one external, respected social science researcher selected in consultation with ETA. A checklist of key items to guide the review will be created by contractor in consultation with ETA.

3. Peer review of ETA evaluation program. This review will be conducted by one contractor staff and two external, respected social science researchers selected in consultation with ETA. This peer review of will assess the policy/program importance of evaluations commissioned by ETA over the past five years (2006-2010), their quality, and the extent to which they draw on earlier ETA/outside research findings, so as to build cumulative evidence.

4. Training for program office staff. This briefing will impart an understanding of the term “evidence-based practices” and the context for how OMB is incorporating evidence-based practices across the Federal government. It will include examples of how evidence-based practices have influenced the development of policy and program designs by Federal, state and local governments and how agencies are incorporating evidence-based practices into their design and provision of technical assistance, including dissemination of best practices. It will also include sharing strategies for finding and identifying workforce development and related evidence-based practices and an understanding of the type of information and/or resources that practitioners will need to increase their replication of evidence-based research.

In accordance with FAR Clause 52.217-7, Option for Increased Quantity – Separately Priced Line Item, the government reserves the right to exercise the following options for increased quantity of services:

First Option year:

At the Government’s discretion and pending funds availability, DOL may exercise an option for increased quantity of services in the form of:

1. Additional peer review of the specific study designs. Following development of draft design plans for specific projects, contractor will conduct a peer review to assess whether the proposed study design contains any serious flaws. Each review will be conducted by two contractor staff and one external, respected social science researcher selected in consultation with ETA. The review will not seek to provide feedback on all aspects of the proposed study design but rather will concentrate on identifying any serious methodological shortcomings.

2. Additional peer review of specific project’s interim and/or final reports. Once each study is underway, contractor will use a peer review process to assess whether the study is producing (or for a final report, has produced) valid evidence. Each interim report will be reviewed by two contractor staff. Each final report will be reviewed by two contractor staff and one external, respected social science researcher selected in consultation with ETA. A checklist of key items to guide the review will be created by contractor in consultation with ETA.

3. Additional peer review of ETA evaluation program. This review will be conducted by one contractor staff and two external, respected social science researchers selected in consultation with ETA. This peer review of will assess the policy/program importance of evaluations commissioned by ETA over the past five years (2007-2011), their quality, and the extent to which they draw on earlier ETA/outside research findings so as to build cumulative evidence.

4. Annual training for program office staff. This briefing will impart an understanding of the term “evidence-based practices” and the context for how OMB is incorporating evidence-based practices across the Federal government. It will include examples of how evidence-based practices have influenced the development of policy and program designs by Federal, state and local governments and how agencies are incorporating evidence-based practices into their design and provision of technical assistance, including dissemination of best practices. It will also include sharing strategies for finding and identifying workforce development and related evidence-based practices and an understanding of the type of information and/or resources that practitioners will need to increase their replication of evidence-based research.

Second Option year:

At the Government’s discretion and pending funds availability, DOL may exercise an option for increased quantity of services in the form of:

1. Additional peer review of the specific study designs. Following development of draft design plans for specific projects, contractor will conduct a peer review to assess whether the proposed study design contains any serious flaws. Each review will be conducted by two contractor staff and one external, respected social science researcher selected in consultation with ETA. The review will not seek to provide feedback on all aspects of the proposed study design but rather will concentrate on identifying any serious methodological shortcomings.

2. Additional peer review of specific project’s interim and/or final reports. Once each study is underway, contractor will use a peer review process to assess whether the study is producing (or for a final report, has produced) valid evidence. Each interim report will be reviewed by two contractor staff. Each final report will be reviewed by two contractor staff and one external, respected social science researcher selected in consultation with ETA. A checklist of key items to guide the review will be created by contractor in consultation with ETA.

3. Additional peer review of ETA evaluation program. This review will be conducted by one contractor staff and two external, respected social science researchers selected in consultation with ETA. This peer review of will assess the policy/program importance of evaluations commissioned by ETA over the past five years (2008-2012), their quality, and the extent to which they draw on earlier ETA/outside research findings so as to build cumulative evidence.

4. Annual training for program office staff. This briefing will impart an understanding of the term “evidence-based practices” and the context for how OMB is incorporating evidence-based practices across the Federal government. It will include examples of how evidence-based practices have influenced the development of policy and program designs by Federal, state and local governments and how agencies are incorporating evidence-based practices into their design and provision of technical assistance, including dissemination of best practices. It will also include sharing strategies for finding and identifying workforce development and related evidence-based practices and an understanding of the type of information and/or resources that practitioners will need to increase their replication of evidence-based research.

Third Option year:

At the Government’s discretion and pending funds availability, DOL may exercise an option for increased quantity of services in the form of:

1. Additional peer review of the specific study designs. Following development of draft design plans for specific projects, contractor will conduct a peer review to assess whether the proposed study design contains any serious flaws. Each review will be conducted by two contractor staff and one external, respected social science researcher selected in consultation with ETA. The review will not seek to provide feedback on all aspects of the proposed study design but rather will concentrate on identifying any serious methodological shortcomings.

2. Additional peer review of specific project’s interim and/or final reports. Once each study is underway, contractor will use a peer review process to assess whether the study is producing (or for a final report, has produced) valid evidence. Each interim report will be reviewed by two contractor staff. Each final report will be reviewed by two contractor staff and one external, respected social science researcher selected in consultation with ETA. A checklist of key items to guide the review will be created by contractor in consultation with ETA.

3. Additional peer review of ETA evaluation program. This review will be conducted by one contractor staff and two external, respected social science researchers selected in consultation with ETA. This peer review of will assess the policy/program importance of evaluations commissioned by ETA over the past five years (20079-2013), their quality, and the extent to which they draw on earlier ETA/outside research findings so as to build cumulative evidence.

4. Annual training for program office staff. This briefing will impart an understanding of the term “evidence-based practices” and the context for how OMB is incorporating evidence-based practices across the Federal government. It will include examples of how evidence-based practices have influenced the development of policy and program designs by Federal, state and local governments and how agencies are incorporating evidence-based practices into their design and provision of technical assistance, including dissemination of best practices. It will also include sharing strategies for finding and identifying workforce development and related evidence-based practices and an understanding of the type of information and/or resources that practitioners will need to increase their replication of evidence-based research.

Fourth Option year:

At the Government’s discretion and pending funds availability, DOL may exercise an option for increased quantity of services in the form of:

1. Additional peer review of the specific study designs. Following development of draft design plans for specific projects, contractor will conduct a peer review to assess whether the proposed study design contains any serious flaws. Each review will be conducted by two contractor staff and one external, respected social science researcher selected in consultation with ETA. The review will not seek to provide feedback on all aspects of the proposed study design but rather will concentrate on identifying any serious methodological shortcomings.

2. Additional peer review of specific project’s interim and/or final reports. Once each study is underway, contractor will use a peer review process to assess whether the study is producing (or for a final report, has produced) valid evidence. Each interim report will be reviewed by two contractor staff. Each final report will be reviewed by two contractor staff and one external, respected social science researcher selected in consultation with ETA. A checklist of key items to guide the review will be created by contractor in consultation with ETA.

3. Additional peer review of ETA evaluation program. This review will be conducted by one contractor staff and two external, respected social science researchers selected in consultation with ETA. This peer review of will assess the policy/program importance of evaluations commissioned by ETA over the past five years (2010-2014), their quality, and the extent to which they draw on earlier ETA/outside research findings so as to build cumulative evidence.

4. Annual training for program office staff. This briefing will impart an understanding of the term “evidence-based practices” and the context for how OMB is incorporating evidence-based practices across the Federal government. It will include examples of how evidence-based practices have influenced the development of policy and program designs by Federal, state and local governments and how agencies are incorporating evidence-based practices into their design and provision of technical assistance, including dissemination of best practices. It will also include sharing strategies for finding and identifying workforce development and related evidence-based practices and an understanding of the type of information and/or resources that practitioners will need to increase their replication of evidence-based research.

Period of Performance The period of performance of this evaluation will be one base year plus four one year options to be exercised at the discretion of the Government.

Deliverables

The contractor shall submit the following reports and other documents to the Contracting Officer’s Technical Representative designated for the contract. All documents shall be submitted electronically.

1. Monthly Progress Reports. The first such report will be due 30 calendar days after the contract beginning date and subsequent reports will be due monthly thereafter. Reports should address monthly progress of tasks specified in the statement of work, identification of problems and plans for the resolution of those problems, and plans for the upcoming month.

2. Reports on peer reviews of study designs. Following a peer review of each study design, contractor shall prepare and submit a brief report on the findings of the peer review process, including suggestions to remedy any design flaws that are identified. There may be up to four-to-six such reports during the base period of performance and each option period.

3. Reports on peer reviews of interim and final reports. Following a peer review of an interim report, contractor shall prepare and submit a brief report on the review’s findings, including whether the study is producing valid evidence. Following a peer review of a final report, contractor shall prepare and submit a brief report on the review’s findings, including whether the study has produced valid evidence. There may be up to four-to-six such reports during the base period of performance and each option period.

4. Annual reports on the peer review of ETA’s evaluation program. Following each peer review, contractor shall prepare and submit a brief report on the review’s findings, including a rating of ETA’s overall evaluation efforts.

5. Training materials for ETA staff engaged in guiding and advising the public workforce system. These materials and documents shall be delivered in draft two weeks prior to each scheduled briefing.

PAGE