Attachment_B_WWC_R2D2_IDIQ_SOW.docx

ATTACHMENT B – WWC R2D2 IDIQ Statement of Work

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance

WHAT WORKS CLEARINGHOUSE:

REVIEWS, REPORTING, DISSEMINATION, & DEVELOPMENT (R2D2)

IDIQ STATEMENT OF WORK

Table of Contents

I. Introduction

The U.S. Department of Education (Department), via the National Center for Education Evaluation and Regional Assistance (NCEE) of the Institute of Education Sciences (IES), has the need for a contractor to provide a broad spectrum of support for the What Works Clearinghouse (WWC). This support will include conducting reviews of studies submitted as part of the Department’s grant competitions, reviewing studies that receive Department funding, developing WWC reports, and creating new products to disseminate the findings of the WWC, as the need arises.

Authorizing Legislation Section 171(b) of the Education Sciences Reform Act of 2002 states that the mission of the National Center for Education Evaluation in the Institute of Education Sciences shall be to provide technical assistance, and to encourage the use of scientifically valid education research and evaluation throughout the United States. The WWC encourages the use of scientifically valid education research by serving as the public’s central and trusted source on what works in education, providing accessible information on study quality.

II. Background

The WWC was established in 2002 by the Department/IES to provide educators, policymakers, researchers, and the public with a transparent way to determine whether there is evidence of effectiveness behind a program or practice and the extent to which those findings can be trusted. The WWC was charged to develop and establish WWC Study Evidence Standards and a systematic review system. The goal was to promote better informed education decision making through a web-based dissemination system that features comprehensive, systematic, high-quality reviews[footnoteRef:1] of studies on the effectiveness of educational interventions (i.e., programs, products, practices, and policies). The website, www.whatworks.ed.gov is the primary “face” of the WWC, displaying the results of the systematic review process. [1: A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review. For further information about the WWC Systematic Review Process, refer to www.whatworks.ed.gov/reviewprocess/review.html.]

Current WWC Contract Mechanisms The WWC investment currently operates through three distinct contract vehicles. The “WWC Pre-K to 12 Content Contract” (ED-IES-13-C-0010) focuses on reviews covering pre-kindergarten through 12th grade. The “WWC Postsecondary Contract” (ED-IES-12-C-0084) focuses on reviews in the areas of college readiness and postsecondary success. These two contracts maintain a focus on generating reviews based on prioritized topics within each area. These two contracts maintain a WWC Statistical and Technical Analysis Team, produce key products (listed below), and manage inquiries that arise in response to reviews. The “WWC Website Contract” (ED-IES-13-C0013) supports the production and implementation of all WWC website materials.

The Reviews, Reporting, Dissemination, and Development Contract (referred to here as R2D2) adds a fourth WWC vehicle that allows for the timely review and reporting of studies submitted as part of the Department’s evidence-based grant competitions. In addition, this contract will create or update reports, particularly Intervention Reports, for a subset of these studies and identify innovative ways to further develop the WWC as a recognizable brand to the public. All contracts must continue to work in a cohesive, coordinated manner to represent the entire WWC investment.

Current WWC Products Each of the WWC contracts is responsible for creating or featuring products that translate study reviews into findings that are accessible to the public. To develop these products, the WWC uses objective and transparent standards and procedures to assess the scientific merit of studies of the effectiveness of education interventions. Underlying each of these reviews is a Study Review Guide (SRG), which is completed by WWC certified reviewers to assess the quality of the research according to established WWC standards (access the current SRG template here: http://ies.ed.gov/ncee/wwc/StudyReviewGuide.aspx). SRGs are used to develop a set of products that summarize the results of the systematic review. Currently, reviews are conducted for the following types of products:

· Intervention Reports. Intervention reports summarize all studies published during a specific time period that examine the effectiveness of an intervention. Using its objective and transparent standards, the WWC rates the quality of the evidence in each study. For studies that meet WWC evidence standards (with or without reservations), the WWC combines the findings to generate overall estimates of the size of effects for the intervention. The WWC also rates the level of evidence on the intervention’s effectiveness, taking into consideration the number of studies, the sample sizes, and the magnitude and statistical significance of the estimates of effectiveness.

· Practice Guides. These guides contain practical recommendations that educators can use to address specific challenges in their classrooms and schools. The recommendations are based on reviews of research as well as the expertise and professional judgments of a panel of nationally recognized experts that includes both researchers and educators.

· Reports on Individual Studies. The WWC conducts reviews of individual studies to produce quick reviews and single study reviews. Quick Reviews summarize evidence presented in a study that has garnered notable mention in the press. Single Study Reviews summarize evidence from other notable studies of interest that otherwise might not appear in an Intervention Report for some time. These reports describe the program, policy, or practice studied, indicate whether the study meets WWC evidence standards (with or without reservations), and summarize the study findings on effectiveness. In addition to quick reviews and single study reviews, the WWC has also begun conducting reviews of individual studies in response to the increasing federal emphasis on evidence-based grant making. This contract will substantially help to serve that emerging need, expanding the ability of the WWC to review more research.

III. Purpose

The WWC reviews studies that applicants submit to the Department’s grant competitions that use research evidence as a priority. In the past, the specific grant competitions have included Investing in Innovation (i3), Supporting Effective Educator Development, Arts in Education, the Strengthening Institutions Program, and First in the World. To facilitate use of evidence in grant competitions and to make results of its reviews widely available, the WWC has developed a streamlined process to review these studies, produce Study Review Guides, enter studies in the WWC database of reviewed studies (http://ies.ed.gov/ncee/wwc/ReviewedStudies.aspx), and produce reports as needed. Over the past two years, in addition to continuing its primary work of producing products described in Section II above, the WWC has reviewed over 200 studies as part of these competitions.

The purpose of the R2D2 contract is twofold. First, this contractor shall conduct the reviews of individual studies, using reviewers who have been certified to conduct WWC reviews. These reviews of individual studies are conducted to support grant competitions and other initiatives of a variety of entities within the Department, including but not limited to the Office of Innovation and Improvement (OII), the Office of Postsecondary Education (OPE), the Office of Elementary and Secondary Education (OESE), and IES. These reviews occur on an as-needed basis and involve quick turnaround times for a significant volume of work, which will be estimated at the time each Task Order is awarded. The contractor may be asked to manage a series of reviews for competitions that may have overlapping and brief timelines for review, often occurring during the last few months of a given fiscal year. In short, the contractor must be able to conduct reviews “at scale”; it must marshal sufficient WWC-certified reviewers and have sufficient quality assurance procedures to review many studies in a short time frame to a high standard of quality.

The second purpose of this contract is to support the development of reports and expand dissemination efforts for the WWC. As the need has risen for the WWC to conduct more study reviews, there is also a need to create reports for a subset of those studies and to develop a strategy around the studies that will be prioritized for products. The contractor shall plan to develop a set of products based on study reviews that is complementary to existing WWC products. In addition, dissemination and branding continue to be critical factors to the broader success of the WWC. The contractor shall identify ways to enhance the reach and effectiveness of the WWC, including but not limited to developing infographics, videos, and other materials for the website or for distribution through other means, such as social media. The Department expects the methods and mediums of communication to continue to evolve rapidly, and the contractor must be prepared for communication strategies to change over the course of the contract.

IV. Period of Performance

The period of performance for this IDIQ contract is 60 months from the date of contract award.

V. Types of Task Requirements

Individual task orders will not be issued past the last day of the IDIQ period of performance. Task orders may be issued on a firm-fixed-price or cost-reimbursement basis. The work performance by the contractor will be defined in specific work assignments (task orders) which will specify task objectives, list all detailed requirements, and describe the work to be performed, deliverables, and schedules. The nature of the work will vary from task order to task order.

VI. Anticipated Scope of Work

R2D2 is designed to cover a variety of the WWC’s new needs for reviewing studies and disseminating their findings.

The amount of work anticipated under this contract will be affected by the schedule of work that is undertaken by the Department. This section provides a general overview of the requirements to be performed over the master IDIQ contract’s period of performance.

Task 1: Management & Cross-WWC Contract Coordination

The contractor shall manage task orders in an efficient manner that fosters communication with the COR and grant programs supported. The contractor shall have sound project management processes and personnel who can manage multiple task orders simultaneously, as it is anticipated that several task orders can have overlapping deadlines. Typical task order management activities are outlined below.

Activity 1.1 Kickoff Meeting. The contractor’s Project Director and Key Project Staff (up to three) shall meet with the WWC Project Manager, Contracting Officer’s Representative (COR), and other appropriate Department staff within two weeks following the award date of the initial IDIQ contract. During this Kickoff meeting, Department staff and the contractor shall discuss details regarding the anticipated schedule of activities for the coming year and other issues related to the contract. The meeting shall be at IES’s offices in Washington, D.C. The contractor shall provide an agenda for the meeting within five business days after the effective date of the initial contract award. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within two weeks following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting and how each will be addressed. This task will be firm-fixed-price.

Activity 1.2: Biweekly Conference Calls. The contractor’s project director shall submit bi-weekly abbreviated notes to the COR on the status of each of the tasks under the contract, including details on the status of each review underway. The contractor shall submit the notes at least one hour prior to regularly scheduled bi-weekly telephone updates. Outstanding issues or actions requiring follow up by either the COR or the contractor’s project director will be discussed in weekly conference calls between the contractor’s project director and the COR. The contractor shall update the abbreviated status report based on discussions and submit the report to the COR within one day following the call.

Activity 1.3: Periodic in-person Meetings. The project director and up to two additional key staff shall meet with the COR and other appropriate NCEE staff in Washington, D.C. on an as needed basis, in person or by conference call, not to exceed four times per year. The purpose of the meeting is to brief NCEE on the progress of the contract and to discuss issues as they arise. Within two weeks following each meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed.

Activity 1.4: Communication and Coordination with the COR and NCEE Leadership. The Project Director for this contract shall plan for at least monthly communication with the COR and other key NCEE Leadership. The content of these phone calls shall be content related to the particular task order that is exercised.

Activity 1.5: Monthly Reporting. In addition to the reports that are required for each task as described, for cost reimbursable tasks, the contractor shall include discussion of progress, issues, and problems in a monthly progress report. The monthly reports shall include a budget summary of costs per task and per subtask, including expenditures in relation to progress on content development and other work streams, so the COR can easily see how much has been spent on work in a given task/subtask. The reports shall also specify the extent to which the project is on schedule, briefly describe the activities planned for the next month, identify and discuss significant deviations from the substantive and time factors in the management plan, and identify and discuss any decisions that may be needed from NCEE.

Activity 1.6 Quality Control. The contractor shall create procedures for collecting and analyzing data on the quality and accuracy of task order activities delivered and the products developed under this contract. These quality control measures may include tools such as internal quality checks on WWC study reviews to ensure that all reviewers are accurate and up to date on WWC procedures.

Activity 1.7 Collaboration and Coordination with the Other WWC Contractors. As the WWC investment continues to grow, it is critical that all contractors operate in a coordinated, collaborative fashion. Key personnel from the R2D2 Contract shall have at least bimonthly communication with the project director of the other WWC contracts (via conference call), as requested by the COR. The content of these phone calls shall include discussion about new developments in the WWC standards, protocol requirements, methodological issues, expected volume of work, issues and challenges, any upcoming changes that may be occurring at the WWC, or other relevant topics. The COR will participate in these phone calls, as needed. The contractor shall include a memo summarizing key issues and concerns raised at the meeting and how each will be addressed as part of the monthly report. This memo shall be provided two days after the meeting.

Task 2: Review Individual Studies

Activity 2.1 Conduct Reviews for the Department’s Grant Competitions. The contractor shall plan to conduct reviews of studies submitted as part of the Department’s various grant competitions that require evidence. The review includes completing a Study Review Guide (SRG) for each study. The contractor shall determine whether studies meet WWC Evidence Standards and whether findings indicate intervention effectiveness. The contractor shall manage a series of reviews of studies that are submitted for grant competitions that may have overlapping and brief timelines for review. These reviews tend to occur at the end of the fiscal year, but can happen at any point in the year.

Activity 2.2 Conduct Reviews of Studies funded by IES. The contractor shall conduct annual WWC reviews of a select number of IES-funded research projects, including draft reports, monographs, and presentations, as needed to assess whether preliminary final study results meet WWC Evidence Standards and whether findings indicate intervention effectiveness. The contractor shall produce an assessment of the studies that indicates the number of studies that met standards and the overall areas that studies needed improvement. This IES-projects that need to be reviewed will be selected by the NCEE and send to the R2D2 contractor.

Activity 2.3 Conduct Reviews of Studies funded by the Department. The contractor shall review a select number of studies funded across the Department to determine whether studies meet WWC Evidence Standards and whether findings indicate intervention effectiveness. The contractor shall also consider strategies for whether findings should inform developments or updates for existing WWC products.

Task 3: Develop a Series of Products to Disseminate Information about the WWC

IES intends to introduce new products that enhance and expand the content and reach of the WWC while also reflecting a commitment to the rigorous research standards that are at the heart of the WWC. A primary goal of the new WWC contract is to develop new products that are clearly applicable and useful to different audiences.

Activity 3.1 Identify a Plan for Creating New Products. The contractor shall propose new ideas that improve and expand the content, reach, and usefulness of the WWC while also maintaining or enhancing the WWC commitment to rigorous standards and exceptionally high quality.

Activity 3.2 Create New Products. The contractor shall implement the aforementioned plan and develop new ideas that improve and expand the content, reach, and usefulness of the WWC while also maintaining or enhancing the WWC commitment to rigorous standards and exceptionally high quality. New products can build on existing WWC topic areas or can target new areas. These products may include but are not limited to, new reports or summaries, infographics, videos, webinars, text for website landing pages, podcasts, guides, or activities, including those carried out on social media. The R2D2 contractor shall propose an overarching vision for how to best communicate what works in education to a variety of audiences. All of these items must be created in a format the can be easily disseminated via the WWC website.

Activity 3.3 Develop and Update Intervention Reports. The contractor shall propose a plan for both developing and updating WWC Intervention Reports. The contractor shall propose to create reports based on individual studies reviewed and/or based on expansions of existing WWC topic areas (http://ies.ed.gov/ncee/wwc/topics.aspx). The contractor may also propose a new area for consideration to NCEE. All suggested approaches for reports must be based in WWC standards, follow established protocols, and be consistent with the procedures used across all WWC contracts.

Task 4: Manage a “Satellite” WWC Help Desk

The Contractor shall manage a “satellite” WWC Help Desk that operates in coordination with the main Help Desk that is managed by the WWC Pre-K to 12 Contractor. The “satellite” help desk shall field requests for information or suggestions from the public related specifically to work completed by the R2D2 contractor. Typical requests include those for more information on WWC reports or questions about study findings. The “satellite” Help Desk shall identify requests that may require routing to a senior team member, such as the Project Director, for further analysis. These requests will generally come in through the public contact portal on the IES website but sometimes are emailed directly from the requestor. The contractor shall maintain a record of all Help Desk communications and be prepared to provide responses to all requests.

VII. Types of Deliverables

The contractor shall submit the following major types of final deliverables for each task order.

Monthly Progress Reports. On the 15th of each month during which the task order is in effect, the contractor shall submit a monthly progress report to the COR that describes each task order. The progress report shall include, but not be limited to, information such as a brief summary of major activities and accomplishments by task, problems resolved or in need of resolution, and work to be completed in the next month.

Quality Control Plan. The contractor shall provide a plan that describes the types of methods and instruments to be used to measure the quality of the various R2D2 contractor activities and products for each task order.

Study Review Guides/Master Review Guides. Currently, the WWC SRG is an Excel workbook that guides WWC-certified reviewers through documenting the characteristics of studies, including features that pertain to a study’s eligibility for inclusion in a review being conducted under a particular WWC protocol. The WWC SRG assists the reviewer in assessing the study design and implementation against the WWC standards, and in coding the study findings in a systematic manner that is consistent with WWC reporting guidelines. A Master Review Guide (MRG) is the internal WWC name for an SRG that is final, has been double coded, and verified by a senior reviewer. The contractor shall provide all MRGs to the COR for all grant competitions and reviews of studies funded by the Department. As advised by the COR, the contractor may submit lists of MRGs that are completed for each task order.

Summary Tables of Study Reviews. The contractor shall submit summaries for each individual grant competition or review of IES studies as indicated by the COR. This summary table shall include a column to identify the applicant number, study citation, final study rating, a rationale for the rating, whether there was an author query, and a description of both the author reported and the WWC computed findings.

WWC Reports. The contractor shall develop WWC reports as appropriate. These reports may include, but are not limited to, Intervention Reports (New or Updated), Single Study Reviews, Quick Reviews, or other products as decided with the COR. The contractor shall establish a plan for deciding whether individual studies that have an existing WWC Intervention report should be integrated into an updated Intervention Report, or whether a Single Study Review is necessary.

New WWC Products. The contractor shall develop new products that feature the WWC review process and products in a new, accessible manner. These products may include but are not limited to, new reports or summaries, infographics, videos, webinars, podcasts, guides, or activities, including those carried out on social media.

Commercialization Plan. The contractor shall develop a communications plan to describe the development of new products and supporting materials for the WWC. The contractor shall plan to submit a proposal to the COR and integrate COR revisions into the plan prior to implementation.

Report of Help Desk Inquiries Specific to the R2D2 Contract. The contractor shall manage all R2D2 related inquiries for studies they have reviewed and shall alert the CORs of all inquiries on biweekly conference calls, as the requests arise. A report of all inquiries shall be submitted as part of the monthly progress reports.

VIII. Government Furnished Property (GFP) or Information (GFI)

Required WWC information (reports, data files, file documentation, contacts with the WWC website contractor, and discussions with relevant NCEE staff) will be provided to the contractor, as needed by the project to perform work under individual task orders.

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance

SAMPLE Performance Work Statement – Task Order #1

IES CONTRACT MANAGEMENT, COORDINATION, & VISION FOR DISSEMINATING NEW PRODUCTS

VERSION DATE: 2.23.15

Introduction

IX. Scope of Work: Contract Management, Coordination, & Vision for Disseminating New Products

Section 1: Purpose The contractor shall manage task orders in an efficient manner that fosters communication with the Contracting Officer’s Representative (COR), other What Works Clearinghouse (WWC) contractors, and the grant programs supported. It is expected that the contractor shall have sound project management procedures in place and ensure coordination across contracts to seamlessly facilitate the work of the WWC. The contractor shall also create a strategy for developing reports as appropriate and produce reports approved by NCEE. After the strategy has been approved by the COR, the contractor shall begin to develop reports as specified. Finally, the contractor shall develop a series of products that will disseminate information about particular aspects of the WWC, its work, and/or its content to broad or targeted audiences. New products developed shall be clear, accessible, and depict the WWC content in a fresh and engaging way. The details of these expectations are described in Section 4 below.

Section 2: Contract Type This task order will include a hybrid of firm fixed price (FFP) and cost-reimbursable (CR) tasks, as indicated below in each activity or work requirement.

Section 3: Period of Performance The period of performance for this task order is 24 months.

Section 4: Activity Descriptions and Work Requirements Task 1 Management

Task 1.1 The contractor’s Project Director and Key Project Staff (up to three) shall meet with the WWC Project Manager, COR, and other appropriate Department staff within two weeks following the award date of the Task Order #1 for a project Kickoff meeting. During this meeting, NCEE staff and the contractor shall discuss details regarding the anticipated schedule of activities for the coming year, particularly for evidence review competitions, and other issues related to the task order . The meeting shall be at IES’s offices in Washington, D.C. The contractor shall provide an agenda for the meeting within five business days after the effective date of the initial contract award. The contractor shall come prepared to identify any areas of concern and to suggest ways of responding to these concerns. Within two weeks following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting and how each will be addressed. This task will be firm-fixed-price.

Task 1.2 The contractor shall prepare for biweekly conference calls with the COR and other NCEE leadership. One day prior to each call, the contractor shall create an agenda and request COR input. During the call, the contractor shall present the status of each of the active tasks under this task order. In addition, the contractor shall be prepared to discuss outstanding issues or actions that require follow-up by either the COR or the contractor’s Project Director. Following each call, the contractor shall create notes and submit them to the COR as a record of status, outstanding issues, and next steps. This task will be firm-fixed-price.

Task 1.3 The project director and up to two additional key staff shall meet with the COR and other appropriate NCEE staff in Washington, D.C. on an as needed basis, in person or by conference call, not to exceed two times per year. The purpose of these meetings is to brief NCEE on the progress of the contract and to discuss broad contract strategy (e.g., activities discussed in Task 1.1) or major issues as they arise. Within two weeks following each meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed. This task will be cost-reimbursable.

Task 1.4 The contractor shall create procedures for collecting and analyzing data on the quality and accuracy of task order activities delivered and the products developed under this task order. These quality control measures may include tools such as internal quality checks on WWC study reviews to ensure that all reviewers are accurate and up to date on WWC procedures. This task will be firm-fixed-price.

Task 1.5 As the WWC investment continues to grow, it is critical that all WWC contractors operate in a coordinated, collaborative fashion. The contractor shall establish a process for ensuring seamless coordination with all other WWC contractors on relevant tasks. Key personnel from the R2D2 contract shall have at least bimonthly communication with the Project Directors of the other WWC contracts, as requested by the COR. The content of these phone calls shall include discussions about new developments in the WWC standards, protocol requirements, methodological issues, expected volume of work, issues and challenges, any upcoming changes that may be occurring at the WWC, or other relevant topics. The COR will participate in these phone calls, as needed. The Task Order #1 contractor shall include a memo summarizing key issues and concerns raised at the meeting and how each will be addressed as part of the monthly report.

The contractor shall lead every 4th conference call, providing a conference line for all contractors and summary notes for the COR. These notes must be circulated to all WWC contractors by the Task Order #1 contractor. The contractor shall include a memo summarizing key issues and concerns raised at the meeting and how each will be addressed as part of each call that they lead. This task will be cost-reimbursable.

Task 2.2 The contractor shall update or create new reports as specified in the strategy that is approved by the COR. This task will be cost-reimbursable.

Task 3.2 The contractor shall implement the aforementioned plan and develop new ideas that improve and expand the content, reach, and usefulness of the WWC while also maintaining or enhancing the WWC commitment to rigorous standards and exceptionally high quality. These products may include but are not limited to, new reports or summaries, infographics, videos, webinars, text for website landing pages, podcasts, guides, or activities, including those carried out on social media. All of these items must be created in a format the can be easily disseminated via the WWC website. This task will be cost-reimbursable.

Section 5: Schedule of Deliverables

What Works Clearinghouse Investment

Pre-K to 12 Content

Postsecondary Content

Website

R2D2