WWC_R2D2_DRAFT_Task_Order_2_i3_2.25.15.pdf

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

WHAT WORKS CLEARINGHOUSE:

REVIEWS, REPORTING, DISSEMINATION, & DEVELOPMENT (R2D2)

Performance Work Statement – Task Order #2 CONDUCT REVIEWS FOR THE i3 COMPETITION

VERSION DATE: 2.25.15

Task Order #2: Conduct Reviews for the i3 Competition

Performance Work Statement

WHAT WORKS CLEARINGHOUSE:

REVIEWS, REPORTING, DISSEMINATION, & DEVELOPMENT (R2D2)

I. Scope of Work: Conduct Reviews for the i3 Competition

Section 1: Purpose

The purpose of this task order is to conduct reviews of studies that are submitted to the U.S.

Department of Education (the Department) as evidence for the Fiscal Year 2015 Investing in Innovation (i3) grant competition. The contractor shall review studies for three i3 competitions: Scale-Up, Validation, and Development. The contractor shall plan to conduct reviews for up to 105 individual studies for the entire i3 competition.

Section 2: Contract Type

This task order will be cost-reimbursement.

Section 3: Period of Performance

The period of performance for this task order is anticipated to be from August 2015 through December 31, 2015.

Section 4: Work Requirements and Task Descriptions

Task 1.1: Locate PDFs for studies.

Within one week of task order award, the COR will email study references to the contractor and depending on the timeline, may also email PDFs of identified studies. The COR will also email the contractor an Excel spreadsheet that includes an applicant number and the citations of studies that need to be reviewed by the WWC. Using the Excel spreadsheet supplied by the COR, the contractor shall locate PDFs for the studies that applicants have submitted for review.

The study review timeline will be determined within one week of award of this task order. After the review timeline is finalized by the Contracting Officer’s Representative (COR) and the contractor, the contractor shall have approximately three weeks to complete reviews for the Scale-Up and Validation studies, and approximately 10 business days to complete reviews for the Development studies.

Task 1.2: Review studies and produce a Master Review Guide.

The contractor shall search the WWC database of reviewed studies (http://ies.ed.gov/ncee/wwc/ReviewedStudies.aspx) to determine whether a given study has already been reviewed by the WWC. If a study has not been reviewed, the contractor shall conduct a review for the study, providing a final Master Review Guide (MRG) for each study. An MRG is a final version of a Study Review Guide (SRG) that has been quality checked for errors. For each study, two WWC reviewers that have been certified in version 3.0 standards will complete an SRG that assesses the quality of the research according to established WWC standards (access the current SRG template here:

http://ies.ed.gov/ncee/wwc/StudyReviewGuide.aspx).

The contractor shall ensure that a senior level team member performs a quality check of all work to ensure accuracy and consistency prior to finalizing the MRG.

The contractor shall follow all standard WWC review procedures, including having reviewers sign a conflict of interest form to indicate that the reviewer is eligible to review the study. In accordance with WWC processes, an author query shall be conducted for any information that is missing from the study. For studies that require author queries, the contractor shall submit initial MRGs indicating the possible ratings. Once the author query process is complete and final ratings are determined, the contractor shall submit revised MRGs. Given the high stakes and quick turnaround nature of the grant competitions, the contractor shall ensure that they are conducting WWC reviews that are accurate and will stand up to inquiry, if the need arises.

Task 1.3: Develop a summary table for each competition.

The contractor shall develop a summary table in MSWord to display the results of each grant competition. This summary table shall include a column to identify the applicant number, study citation, final study rating, a rationale for the rating, whether there was an author query, description of both the author reported and the WWC computed findings, and a paragraph that provides a quick synopsis of study features. This paragraph shall include a description of the intervention or practice, the design, population, and sample size. If the given study has already been reviewed by the WWC, the contractor shall obtain the MRG from the WWC Pre- K to12 Contractor (Contract Number ED-IES-13-C-0010) to complete the summary table. The table should be modeled after the table provided in Appendix A. The contractor shall send this summary electronically to the COR at the close of the review period, at the same time that MRGs are submitted.

Task 1.4 Add studies into the WWC database of studies reviewed.

All studies using an eligible design will be added by the contractor to the online database. The contractor shall follow the protocol established by the website contractor, which requires completing a template for each study that needs to be http://ies.ed.gov/ncee/wwc/ReviewedStudies.aspx http://ies.ed.gov/ncee/wwc/StudyReviewGuide.aspx entered into the database and submitting that document to the WWC Website Contractor. The R2D2 contractor shall submit final MRGs (electronically) to the Pre-K-12 Content and website contractors for archiving in the reviewed studies database no later than December 15, 2015.

Section 5: Government Furnished Information (GFI)

The COR shall provide the following documents to the contractor in the mediums and formats stated below:

• Study references and PDFs of identified studies - Medium: via email.

• An Excel spreadsheet that includes an applicant number and the citations of studies that need to be reviewed by the WWC. - Medium: via email

Section 6: Schedule of Deliverables

Task Activity Deliverables Estimated Due Date

1.1 Locate PDFs for studies Copies of PDFs, when

possible 3 weeks after the effective date of the task order

1.2 Review studies and produce a Master

Review Guide

MRGs 10 days to 3 weeks after the effective date of the task order, depending on the exact competition timeframe

1.3 Develop a summary table for each

competition

Summary Table 3 weeks after the effective date of the task order

1.4 Add studies into the WWC database

of studies reviewed

Verification that studies have been added to the database

By December 15, Appendix A

Example Summary Table for Grant Review Competitions

Applicant Citation Final Rating Rationale for

Rating Author Query Status

Author’s Findings WWC Findings Description Statistically Significant?

If not Significant, Substantively Important?

Statistically Significant?

If not Significant, Substantively Important?

01234567 Bulgren, J. A., Deshler, D. D., Schumaker, J. B., & Lenz, B. K. (2000).

The use and effectiveness of analogical instruction in diverse secondary content classrooms.

Journal of Educational Psychology, 92(3), 426-441

Meets WWC Evidence Standards without Reservations

This study uses a cluster randomized design to examine the effects of analogical instruction on knowledge outcomes. The study has low cluster and subcluster attrition, and therefore meets WWC group design standards without reservations

N/A Both achievement measures are reported as statistically significant

N/A Both measures needed cluster correction, and one was no longer significant after correction

N/A In an experimental study of high school students across three high schools in a Midwestern suburban district, Bulgren et al.

examine the effect of analogic instruction using an instructional device called the Concept Anchoring Table (CAT) and an instructional routine called the Concept Anchoring Routine. The anchoring tables are used as a visual tool for displaying an analogy that connects known information with new information.

The Concept Anchoring Routine then uses three instructional phases to indicate the importance of the new concept, co-construct the CAT with students, and ask reflective content and process oriented questions about the exercise. Eight science classes taught by three teachers in three high schools were randomly assigned to either Condition 1 (n=39) or Condition 2 (n=44). All classes received two 35-minute content lessons to teach four concepts: pyramid of numbers, commensalism, food web, and heterotroph. In Condition 1, the concept of commensalism was taught using the analogical instruction technique (using the Concept Anchoring Table and Concept Anchoring Routine), and the other three concepts were taught without using analogical instruction. In Condition 2, the concept of pyramid of numbers was taught using the analogical instruction technique and the other three concepts were taught without using analogical instruction. The study was an RCT with low attrition and so meets WWC standards without reservations.

The WWC confirmed the author reported finding of a positive statistically significant impact on student knowledge of commensalism.