TOAST PWS.pdf

WWC-TOAST BPA Performance Work Statement, January 2022

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

WHAT WORKS CLEARINGHOUSETM:

Tools, Online Assistance, Standards, and Training (WWC-TOAST)

Blanket Purchase Agreement (BPA) Performance Work Statement (PWS)

I. Introduction

The U.S. Department of Education (the Department), via the National Center for Education Evaluation and Regional Assistance (NCEE) of the Institute of Education Sciences (IES), needs a contractor to manage the What Works ClearinghouseTM (WWC) website and to provide technical support on the development and review of the products and information contained on the website. The WWC addresses the need for credible, succinct information by identifying existing research in education, assessing the quality of this research, and summarizing and disseminating the evidence from studies that meet WWC standards.

The WWC achieves its mission through the work of a series of contracts. First, the content contractors review studies in a specific area or for a specific purpose. At time of award, there are currently four content contracts: The Guides to Research to Assist Practitioners in Education (GRAPE) task orders are to produce Practice Guides and Intervention Reports. The What Works Clearinghouse - Preschool-to-Postsecondary Evidence Synthesis Task Orders (WWC-PESTO) similarly produce both Practice Guides and Intervention Reports in specific areas. There is currently one task order for PESTO and more task orders are expected in the future. The What Works Clearinghouse Ad-Hoc Reviews of Individual Studies in Education (WWC-ARISE) conducts reviews of individual studies. The Organization of Reporting of Evidence on Graduation, Achievement, and Nonacademic Outcomes (WWC-OREGANO) is slated to end in May 2023 and will have minimal overlap with the current requirement.

In addition to the content contractors, there is a coordination contract. The current requirement is for the coordination requirement. The Tools, Online Assistance, Standards, and Training (WWC-TOAST) contractor achieves the objectives of this contract through six main activities:

1. The WWC develops statistical standards and procedures about how studies should be reviewed. The current set of standards can be found at https://ies.ed.gov/ncee/wwc/Handbooks. The standards and procedures are developed in consultation with methodological experts and WWC content contractors through the Statistical, Technical, and Analysis Team (STAT). The STAT team meets in small groups and as a large group to discuss how to improve the WWC standards and procedures going forward.

2. The contractor will train users on the standards to become certified reviewers for the WWC. This will be done through developing online training, maintaining the certification process, providing annual training and targeted technical assistance, and answering reviewer questions.

3. The TOAST contractor will use those standards to build and maintain the WWC Online Study Review Guide (OSRG) application. WWC content contractors will use the OSRG to review studies. Then content contractors will be working with the TOAST contractor to publish the studies.

4. The contractor will ensure quality of reviews and publications by providing peer review and quality control. All studies are peer reviewed by experts to ensure that https://ies.ed.gov/ncee/wwc/PracticeGuides https://ies.ed.gov/ncee/wwc/AllInterventionReports https://ies.ed.gov/ncee/wwc/ReviewedStudies/ https://ies.ed.gov/ncee/wwc/Handbooks https://youtu.be/W99s8RlNXis they are accurate and do not have errors that could undermine the credibility of the

WWC.

5. The contractor will publish study reviews and reports based on study reviews on the WWC website, found at https://whatworks.ed.gov. They will routinely enhance the website to ensure it is user friendly and accurate.

6. Lastly, the contractor will conduct reviews and produce products that will give them expertise using the system they create.

The purpose of the TOAST BPA is to update and maintain the WWC’s online tools. The WWC uses its public website to communicate with the world and the primary purpose and objective of the contract is to maintain and update the products and tools, including the public tools and the internal tools. The contractor will do this through proposing a strong team that understands both the WWC’s online tools and the WWC standards and how to disseminate the work of the team to broad audiences. All development for TOAST must occur in the IES Data Center.

The TOAST contract is inherently an information technology contract that requires an integrated team with a strong understanding of education research and statistical standards. The contractor needs to understand the statistical standards and the underlying technology of the website in order to provide innovative and cost-effective solutions.

I. Authorizing Legislation

The legislation authorizing this contract is Title I, Part D of the Education Sciences Reform Act of 2002, Section 171(b):

(b) MISSION. —The mission of the National Center for Education Evaluation and Regional Assistance shall be—

(1) to provide technical assistance.

(2) to conduct evaluations of Federal education programs administered by the Secretary (and as time and resources allow, other education programs) to determine the impact of such programs (especially on student academic achievement in the core academic areas of reading, mathematics, and science);

(3) to support synthesis and wide dissemination of results of evaluation, research, and products developed; and

(4) to encourage the use of scientifically valid education research and evaluation throughout the United States.

The 2015 reauthorization of the Elementary and Secondary Education Act (ESEA, P.L.

114-95), the Every Student Succeeds Act (ESSA), emphasized the need for educators and policy makers to identify “evidence-based” practices to improve student outcomes and other outcomes relevant for education. In particular, ESSA amended ESEA to include definitions of “strong evidence” and “moderate evidence” (20 U.S.C. 7801, section 21), corresponding with statistically significant and favorable findings from “well-designed and well-implemented” quasi-experimental and experimental studies.

https://whatworks.ed.gov/

II. WWC Background

The Department established the WWC in 2002 to provide educators, policymakers, researchers, and the public with a transparent way to determine whether there is evidence of effectiveness behind a program or practice and the extent to which those findings can be trusted. A goal for the WWC is to promote better informed education decision-making through a web-based dissemination system that features comprehensive, high-quality systematic reviews of studies on the effectiveness of education interventions. The website, whatworks.ed.gov, is the primary “face” of the WWC, displaying the results of WWC review as well as resources for educators. The WWC is focused on providing relevant and useful reviews of evidence, refining its standards and procedures based on recent improvements in research methods, and disseminating actionable information and related evidence resources to the public.

III. Period of Performance

The ordering period for this BPA contract is 60 months from the date of contract award. There is an optional 3-month transition period that would run from February 2023 to May 2023. During the transition period, the contractor shall only work to understand the existing software and tools to be able to fulfill the responsibilities of the contract with minimal disruption for users. Then the period of performance is scheduled to begin from May 2023 for a period of 57 or 60 months, depending on if the optional transition period was awarded. Individual task orders under this contract may last up to 36 months from the time of award and may extend past the end of the ordering period if they are awarded within those 60 months.

IV. Types of Task Requirements

The Department will issue specific work assignments as task orders under this BPA, specifying the task objectives, the work to be performed, deliverables, schedules, and other requirements. The nature of the work will vary from task order to task order. The Department expects to award firm-fixed-price, cost reimbursement, and hybrid tasks under this BPA to a single prime contractor.

V. Staff Qualifications

The contractor shall propose a well-managed and highly qualified team to manage the work of TOAST. Proposed staff are expected to have sufficient time committed to the project to have a deep understanding of the work. The leadership team, defined as the Primary Investigator, Project Director, and Deputy Directors are required to spend a combined level of effort of at least 2.25 FTE and the minimum levels specified below.

This level of effort is across all open task orders.

IES expects that each position shall be held by one individual. The contractor may https://ies.ed.gov/ncee/wwc/ propose individuals sharing a role if the contractor can convince the government there is sufficient benefit to the government from the arrangement.

- Primary Investigator: The Primary Investigator is responsible for setting the direction and goals of the project. They shall be actively involved in the Statistical, Technical, and Analysis Team (STAT) and all project planning. They shall have expertise in statistical methods, presentation of findings, and conducting study reviews. The Primary Investigator shall be certified in WWC Group Design Standards, version 4.1 at the time of proposal submission and shall be certified in WWC general group designs, advanced group designs, and single-case designs (SCDs), version 5.0, no later than the end of transition and start of the core tasks of the contract1. They shall be committed to at least 0.25 FTE.

- Director: The Director is the lead for Task 1. It is the role of the Director to ensure the work of individual tasks inform each other and the overall work of the project. It is the Director who is ultimately responsible for ensuring that work is submitted error free, on time, and meeting the requirements of the contract. The Director is in daily communication with the COR and the project team and is the main point of contact for all questions about the contract. The Director shall be certified in WWC Group Design Standards, version 4.1 at the time of proposal submission and shall be certified in WWC general group designs and advanced group designs, version 5.0, no later than the end of transition and start of the core tasks of the contract. The Director shall be committed for at least 0.6 FTE for the duration of the project.

- Deputy Director for OSRG and Website: This position is responsible for leading task

2. This person should have a deep technical knowledge in conducting reviews for the WWC and using the Online Study Review Guide (OSRG.) They should understand how the OSRG and website function and technical concerns about improvements and enhancements. The Deputy Director is responsible for ensuring that the OSRG operates smoothly and that contractors are able to complete reviews and publish them to the website with minimal interaction with the TOAST contract. The Deputy Director shall also ensure all timelines for product releases are met and that work is consistently high quality. They shall be in daily communication with the help desk and Director and will frequently be in communication with the COR. The Deputy Director shall be certified in WWC Group Design Standards, version 4.1 at the time of proposal submission and shall be certified in WWC general group designs and advanced group designs, version 5.0, no later than the end of transition and start of the core tasks of the contract. The Deputy Director shall be committed for at least

0.75 FTE during Task Orders 1 and 2 and at least 0.5 thereafter.

- Deputy Director for Reviews, Peer Review, Training, and STAT: This position is responsible for leading tasks 3-6. This person should be an experienced reviewer for the WWC with extensive methodological expertise. The Deputy Director is charged

1 Online training for version 5.0 is expected to be online in early 2023. It will be available at https://ies.ed.gov/ncee/wwc/OnlineTraining with overseeing the day-to-day operations of the tasks and ensure all timelines are met and that work is of high quality. The Deputy Director shall be certified in WWC Group Design Standards, version 4.1 at the time of proposal submission and shall be certified in WWC general group designs, advanced group designs, and single-case designs (SCDs), version 5.0, no later than the end of transition and start of the core tasks of the contract. The Deputy Director shall be committed for at least 0.5 FTE for the duration of the project.

VI. Anticipated Scope of Work

The following tasks and subtasks are included in the scope of work:

Task 1. Management (Hybrid, Firm Fixed Price)

The contractor shall manage task orders in an efficient manner that fosters communication with the COR and other WWC contractors. The contractor shall have sound project management processes and personnel who can manage multiple task orders simultaneously, as it is anticipated that several task orders can have overlapping deadlines.

Typical task order management activities are outlined below.

1.1. Kickoff Meeting (Non-IT)

Within the first 2 business days after the contract start date, the project leadership shall attend a kickoff meeting with the Department to discuss details regarding the tasks outlined in the proposal, scheduling activities, the transition process, and other issues related to the contract. Project leadership is defined as the project investigator (if proposed), project director, deputy project director, and may also include the task leads and/or contract administrators.

The meeting will be held via conference call. The contractor shall provide an agenda for the meeting within 1 business day after the contract start date. Within 5 business days following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting and how each will be addressed. The memo also shall document key decisions made at the meeting, as well as any specific technical directions provided by the Contracting Officer’s Representative (“COR”).

1.2. Transition-in Materials (IT, Optional)

The contract is intended to be awarded with an up to 90-day transition-in period that ends on May 17, 2023. Prior to May 17, 2023, the contractor shall only perform work on Task 1 and subtask 2.3. During this period the contractor is responsible for ensuring that staff has adequate security clearances and permissions to access the WWC websites and tools and that they are adequately trained in the system to meet the metrics specified in the contract.

Within 2 days of contract award the contractor shall submit a request to IES to initiate any staff that do not have security clearances and a request to validate staff members that have existing clearances. These requests shall be in the format specified in the Contractor

Employee Security Screening Requirements section of the PWS. Once initiated, staff will have 14 days to complete all paperwork and return to the COR. The COR will notify the project director when the packages have been accepted by the Security Office in the National Center for Education Statistics and individuals are authorized to have an ELVIS account.

At the kickoff meeting, the COR shall provide a copy of the transition plan developed by the WWC SWAT contractor. Within 21 days of contract award, the contractor shall prepare a transition in memo that uses the dates specified in WWC SWAT contractor’s transition out memo as a basis for the transition. The memo shall document the timeline for each transition activity and responsible parties.

The contractor is responsible for ensuring all staff is adequately trained and has appropriate access to begin accessing the OSRG tools, WWC Website, and the help desk by May 17, 2023. The contractor is expected to meet all contract requirements and have a seamless transition from SWAT to TOAST.

1.3. Biweekly Meetings

The project director and other key staff that the project director deems are necessary shall meet with the COR and other appropriate Department staff via biweekly conference calls to be held at regularly scheduled intervals throughout the duration of the contract. The purpose of the meetings is to brief the Department on progress and challenges related to the tasks in this contract. The Contractor shall send an agenda to the COR 3 days in advance of the meeting and a meeting summary to the COR within 3 business days after the meeting is complete.

In addition, the Contractor shall be available for meetings to address unexpected issues that arise during the contract period. These meetings may occur on an as-needed basis at the request of the COR or the Contractor and will take place via conference call. The contractor should assume one meeting a week in the first six months of the contract and approximately one meeting per month thereafter.

1.4. Monthly Report

The contractor shall submit a monthly report that summarizes progress made in each task and subtask, and shall specifically include the metrics specified in the tasks below and listed in Appendix B. The report shall highlight progress made and any challenges faced.

In addition, the monthly report shall include proposed work and expected challenges for the next month.

Contractor shall provide data on earned value in a specific format specified in Appendix C. Any deviations in schedule or cost shall be fully explained in the risk register.

Additionally, the monthly report shall include the IT reporting sheet and the contractor personnel listing using the templates provided in the solicitation.

The contractor shall submit the report on the sixth business day of every month for the duration of the contract and shall be prepared to discuss it in the next biweekly call with the COR.

1.5. Cross Contractor Meetings

The contractor shall be available to meet up to three times a year for cross-contractor meetings. These meetings shall be held in Washington, DC or virtually. The COR will provide ample notice of the time and the agenda for the meeting. The contractor may be asked to present or facilitate a discussion for part of the agenda.

Task 2: Website, Data, and Online Tools (IT, Cost Plus Fixed Fee) The website is the primary way for the WWC to share their products and resources with the world. It is the responsibility of the TOAST contractor to ensure that the Online SRG is functional. If problems arise, it is the primary responsibility of the TOAST contractor to quickly and accurately resolve any issues.

2.1 . Creation of IT Reporting Materials Within 4 weeks of contract award or the start of an option year, the contractor shall a submit work breakdown structure of all deliverables, down to the individual personnel level for cost plus fixed fee deliverables, in Microsoft Project format, a 2-3 page overview of the quality management plan, and a 2-3 page overview of the risk management plan. Guidance for these documents is provided in Appendix C.2.1.

2.2 Develop a Help Desk for Contractors

The contractor shall be responsible for monitoring the email address WWC.Contractor.Help.Desk@ed.gov. This help desk will receive all inquiries from content contractors about problems with the OSRG and publishing content on the website.

The contractor shall monitor the email address from 9am-5pm ET, excluding federal holidays. Contractors shall check the email at least once an hour and provide a substantive reply, defined as one that solves the problem or follows up with the content contractor about when to expect a resolution, within 90 minutes. A substantive reply shall not be a form response, nor shall it be an acknowledgement of the email with the promise of a later response. Substantive replies shall be specifically tailored and address the needs of users, requests more information, or asks to set up a meeting to discuss the issue further. The contractor shall follow up on all issues raised by the content contractors and promptly inform them when issues have been resolved. If a task requires a fix to the OSRG, that should occur under subtask 2.3.

Up to two individuals will have email addresses and the ability to access the help desk.

These individuals shall be certified in WWC Group Design Standards, version 4.1 and both general group designs and advanced group designs, version 5.0. These individuals have expertise in the functionality and design of the Online Study Review Guide and experience conducting reviews.

With the agenda for the biweekly meeting (see subtask 1.3), the contractor shall submit a table summarizing the work of this task. It should contain a list of emails received since mailto:WWC.Contractor.Help.Desk@ed.gov the previous call, a description of each issue raised, the status of each ticket, and the expected time of resolution. In the monthly report, the contractor shall report on 1) the number of email tickets submitted by content contractors, 2) the proportion of tickets that are work stoppages, 3) the proportion of tickets that are errors versus enhancements, 4) the amount of time from receipt to sending a substantive reply, and 5) the mean and median time from receipt of email to issue resolution.

2.3 Maintain and Update the Online Study Review Guide

The contractor shall be responsible for maintaining and upgrading the WWC’s OSRG.

The OSRG tool is used by all WWC contractors, as well as other IES contractors and grantees to review studies against WWC standards and protocols. It is housed in the NCES Members Site in the IES Data Center at the URL:

https://members.nces.ed.gov/OSRG/Account/Login?ReturnUrl=%2fosrg%2f.

Information about the Online SRG can be found here: https://youtu.be/W99s8RlNXis, https://youtu.be/HT6qkZzpjAw, and https://youtu.be/x3Jy0GaLYf0.

The contractor shall be responsible for ensuring that the tool is functional and is accurately completing reviews. On a weekly basis, the contractor shall release updates to the SRG that fix aspects of the tool that are not working for the reviewers. (See subtask

2.2 for more information). The contractor shall design improvements and enhancements that are cost-effective and are long-term solutions that would work for future reviews, as opposed to short-term patches.

The contractor shall have sufficient capacity to quickly address any issues that cause work stoppages, as well as make long term enhancements. This means that the contractor shall have the ability to bring in additional staff when needed to quickly scale up the level of effort so that the SRG remains functional, and products are not delayed. Contractors are expected to resolve issues that are work stoppages within 1 week and other issues within 4 weeks.

The contractor shall also ensure all functionalities of the OSRG are sufficiently documented through the User’s Guide. The User’s Guide shall be consistently up to date and designed in a way to be useful to reviewers and efficient to update.

The contractor is also responsible for adding and maintaining user accounts and ensuring users have completed necessary cybersecurity training. On a quarterly basis the contractor shall review all accounts to ensure that users are actively on contract and have appropriate trainings to use the tool.

With the agenda for the biweekly meeting (see subtask 1.3), the contractor shall submit a table of open tickets, their status, and estimated level of effort to resolve the issue, and estimated completion date. In the monthly report, the contractor shall report on 1) number of tickets opened, 2) number of tickets closed, 3) mean and median time for ticket resolution, 4) number of labor hours for subtask 2.2, and 5) number of tickets not yet addressed. The contractor shall also provide an Excel file that has a list of all studies currently under review, in the format specified in Appendix B as part of the monthly https://members.nces.ed.gov/OSRG/Account/Login?ReturnUrl=%2fosrg%2f https://youtu.be/W99s8RlNXis https://youtu.be/HT6qkZzpjAw https://youtu.be/x3Jy0GaLYf0 report.

2.4 Build a Publication Workflow System.

The contractor shall build a SharePoint tool to monitor the flow of all WWC publications.

This took shall be built in the IES Data Center, preferably using the same log in credentials as the OSRG application. The tool shall be used to track all Intervention Reports and Practice Guides from 1) initial submission to the COR, 2) peer review to review, 3) review by IES leadership, 4) review by the IES Standards and Review Office (if necessary), 5) final editing and 508 compliance, and 6) publishing to the website. The tool shall have the necessary checks and notifications to ensure that all parties are notified at a report’s status and the actions necessary to keep the report on track. The tool shall also have workflow processes to ensure that all studies from the review and the synthesis are published at the appropriate times and that all steps to publish the report on the website are included in the workflow.

IES will provide the code from the REL tracking system to serve as the basis for this tool.

This video demonstrates the REL tracking system: https://youtu.be/oGnSDNseZ1U.

The WWC tracking tool should be as similar in functionality and design as the REL tool to reduce the learning curve of IES and contractor staff and minimize costs and time to develop the tool. However, the processes for releasing publications are different and the TOAST contractor will have to modify the workflow process to reflect the processes of the WWC.

Within 3 weeks of contract award the contractor shall meet with the REL Website contractor to understand the functionality of REL tracking tool. The TOAST contractor shall produce a memo that outlines the process and timeline for developing the tool. The tool shall be functional to pilot test within 6 months of contract award and fully functional within 8 months of contract award. During the pilot testing the contractor shall ensure that the tool is fully operational and that all staff are trained in how to use the tool.

This training shall occur via webinar, a user’s guide, and a series of FAQs.

The tool is considered fully functional when all products currently in process added to the tool, all users have access to the accounts, and all training is available. While modifications may be needed to enhance the product, the tool should be operational with minimal to no intervention needed by the TOAST contractor. After the tool is fully functional, for up to 12 months after the contract award, the contractor shall be responsible for maintaining the system and addressing any areas of concern.

In the monthly report, the contractor shall provide a status report of each publication, including the contractor, type of publication, publication number, publication name, and status, as well as number of underlying studies, whether they have been published to the IES website, and if they are awaiting any action from TOAST.

2.5 Maintain a Publication Workflow Tool.

Once built, the contractor is responsible for maintaining the publication workflow tool.

This means ensuring that all publications are flowing correctly and addressing any https://youtu.be/oGnSDNseZ1U concerns raised by IES and content contractors through the content contractor help desk (see subtask 2.1). The contractor is also responsible for adding and maintaining user accounts.

On an as needed basis, the contractor shall add enhancements in functionality or design.

Any improvements should be proposed to the COR in advance, with the associated level of effort.

With the agenda for the biweekly meeting (see subtask 1.2), the contractor shall provide a status report of each publication, including the contractor, type of publication, publication number, publication name, and status, as well as number of underlying studies, whether they have been published to the IES website, and if they are awaiting any action from TOAST. The contractor shall use this report to forecast upcoming website releases under subtask 2.5.

2.6 Maintain the WWC Website

The WWC website is designed to be responsive to user needs, efficient, and adaptable to new opportunities available through new technology. The contractor shall maintain the existing site by adding new communications or resource products to the website, announcing events, publishing new reviews, and releasing new products. The COR and TOAST contractor will work with the WWC content contractors to develop a schedule for publishing study reviews and publications, as described in subtasks 2.3 and 2.4. The contractor is expected to meet all dates on the agreed upon schedule and release all reports on the agreed upon date.

For requests to publish new reviews, communications products, events, and other announcements, the content contractors shall submit requests through the contractor help desk (see subtask 2.1). The contractor will be responsible for assigning each product a ticket number and ensuring that new content is posted within 2 weeks.

When developing new content, the contractor shall send the COR a link to preview any pages on the IES Development Server. All updates to the website will be approved by the COR prior to implementation.

In the monthly report, the contractor shall provide: 1) the number of new website changes made, 2) the mean and median amount of time for tickets to be addressed, 3) the number and percentage of tickets that take more than 2 weeks to resolve.

2.7 Enhancements to the WWC Website

The contractor shall periodically review the WWC website and propose enhancements in design or functionality that maximizes usability. Examples of website enhancements include reorganizing the WWC Resources page, improving the design and functionality of the download data from individual studies file, redesigning the study and intervention pages, and/or developing an API of the WWC data.

At the start of each task order, the COR and contractor shall develop an estimated budget and brainstorm topics. The contractor shall use that information to propose a series of potential updates and their level of effort on an ongoing basis. Upon gaining the COR’s approval, the contractor will develop a timeline and budget for development. In the monthly report, the contractor shall provide an update on each enhancement and a discussion of any changes or modifications to the timeline.

2.8 Enhancements to WWC Underlying Data

The contractor shall periodically review the underlying WWC database and propose enhancements that increase transparency or maximize usability. Examples of database improvements include standardizing intervention names or outcome domains, adding additional fields, cleaning existing data, and/or restricting existing fields. In year 1, the contractor shall develop a memo the summarizes existing issues. The known issues include, but are not limited to, a lack of official effect size column, negative effect sizes when there are positive improvement indices, incorrect ESSA tier flags, and missing WWC-calculated effect sizes.

The contractor shall use the information from the memo to propose a series of potential updates and their level of effort. Upon gaining the COR’s approval, the contractor will develop a timeline and budget for development. In the monthly report, the contractor shall provide an update on each enhancement and a discussion of any changes or modifications to the timeline.

2.9 Enhancements to the OSRG

The contractor shall periodically review the OSRG. Enhancements may include improving the reconciler view, improving the way to input information to reduce errors, improving the layout, and adding the ability to review new types of studies. It is also within scope to make major changes to the functionality or design.

At the start of each task order, the COR and contractor shall develop an estimated budget and brainstorm topics. The TOAST contractor shall correspond with existing WWC reviewers to determine issues that they are experiencing and potential enhancements that may be helpful. This information should be used to propose a series of potential updates and their level of effort on an ongoing basis. Upon gaining the COR’s approval, the contractor will develop a timeline and budget for development. In the monthly report, the contractor shall provide an update on each enhancement and a discussion of any changes or modifications to the timeline.

2.10 Convene a Working Group to Advise on WWC Content for Non-Researchers Within 6 weeks of contract award, the contractor shall develop a work group to advise the WWC on the prioritization and presentation of WWC-produced content to non-researchers in WWC publications and on the WWC website. The contractor shall develop a memo that proposes the number of proposed sessions, frequency of sessions, specific objectives, and a roadmap of topics to discuss to achieve those objectives.

In this memo, the contractor shall nominate members for the working group. The proposed members should have broad and diverse expertise. The working group shall have between 6 and 9 individuals, with members including current and former teachers, school and district administrators, intermediaries, usability experts, and policymakers.

IES staff will also be invited to attend.

For each working group meeting, the contractor shall send an agenda and any materials at least one week in advance. The contractor shall include a summary of all calls and the progress of the working group in the monthly report.

At the end of task order, the contractor shall develop a summary of the small working group’s discussion and decision points. Upon COR approval, the summary shall be made 508 compliant and posted to the WWC website. The contractor is responsible for ensuring that the meeting materials are posted in a timely fashion through subtask 2.3.

Task 3. Procedures and Standards Development (Non-IT, Cost Plus Fixed Fee) The contractor shall convene a Statistical, Technical, and Analysis Team (STAT) that will support the reviews of the WWC. The STAT is composed of two sets of groups, small working groups that work on specific issues (see subtask 3.1) and the All-STAT which is a large group comprised of members with diverse methodological expertise. The task will support consideration, by expert methodologists familiar with the WWC, of issues related to WWC standards for assessing the internal validity of findings from different types of impact study designs; WWC procedures for characterizing, presenting, and synthesizing study findings that meet WWC standards; and WWC reviewer guidance for applying current standards and procedures.

3.1. Propose and Convene STAT Small Working Groups

Within the first 3 months of contract award, the contractor shall develop a memo that proposes a series of small working groups with specific purposes. Small working groups are expected to have a narrowly defined mission and differing durations. The contractor shall propose the number of proposed sessions, frequency of sessions, and expected duration of the group. Each group shall have specific objectives and a roadmap of topics to discuss to achieve those objectives.

The proposed members should have broad and diverse expertise. The experts should have strong methodological expertise, as well as specific expertise for the small working group. The small working groups shall each have between 4 and 6 individuals, not counting staff members of current WWC contractors (of both TOAST and content contractors of various levels of seniority and expertise) who will be invited to attend these meetings as budgeted under their separate WWC contracts. IES staff shall also be invited to attend.

The contractor shall propose new working groups as the need arises and shall sunset working groups after they have achieved their goals. The process of proposing a new working group shall begin with a discussion at the biweekly check in calls. If the COR approves of the initial idea, the contractor shall submit an updated proposal for the new working group, proposed members, and scope of work.

For each small working group meeting, the contractor shall send an agenda and any materials at least one week in advance. The contractor shall include a summary of all calls and the progress of the working group in the monthly report.

At the end of each working group, the contractor shall develop a summary of the small working group’s discussion and decision points. Upon COR approval, the summary shall be made 508 compliant and posted to the WWC website. The contractor is responsible for ensuring that the meeting materials are posted in a timely fashion through subtask 2.5.

3.2. Propose and convene an All-STAT

The All-STAT shall be a large working group designed to provide interdisciplinary feedback on methodological issues facing the WWC and review proposed changes to the Procedures and Standards Handbooks. The All-STAT shall include 1 or 2 representatives from each active WWC contract, the IES WWC team, and up to 6 representatives from the small working groups.

The TOAST contractor shall convene the All-STAT at least 4 times a year and up to 6 times a year. The contractor shall have a specific purpose for the meeting and shall develop agendas in a way to seek feedback and have a robust discussion.

Every 6 months the contractor shall develop the proposed objectives for upcoming meetings. Upon COR approval, the contractors shall schedule a date. At least 2 weeks before the meeting, the contractor shall submit the agenda and any meeting materials and slides. Final meeting materials shall be sent to the All-STAT at least 5 days before the meeting.

Ten days after the meeting the contractor shall submit a summary of the meeting. The final version of the meeting summary shall be made 508 compliant and posted on the WWC website with any meeting materials. The contractor is responsible for ensuring that the meeting materials are posted in a timely fashion through subtask 2.5.

3.3. Develop an Updated Procedures and Standards Handbook.

The contractor shall provide an updated Procedures and Standards Handbook that builds upon the discussions and decision points of the small working groups and the All-STAT.

Updates may include changes to the existing content, new chapters, or complete rewriting of sections. The contractor shall produce a memo that summarizes the proposed scope of changes to discuss with IES. Upon agreement on scope and direction, the contractor shall propose a comprehensive update that integrates any changes and improvements.

The contractor shall expect that the handbook shall have multiple rounds of revisions by the COR, WWC content contractors, and IES leadership. Draft handbooks will also be posted for public comment. Each revision should be resubmitted in the form of a tracked changes with an accompanying memo that summarizes major decision points and changes.

The final version of the Handbook shall be posted on the WWC website as a 508 compliant PDF. The contractor is responsible for ensuring that the meeting materials are posted in a timely fashion through subtask 2.5. Once posted, the contractor shall host a webinar that discusses the changes and how reviewers can become recertified.

Task 4. Quality Control (Hybrid) The Quality Control task is responsible for ensuring that the data on the WWC website is accurate and complete. While the content contractors are responsible for ensuring the quality of the studies reviewed, TOAST is responsible for ensuring that the overall quality of the content is high quality and accurate. The contractor achieves this through three functions: quality control metrics, peer review, and audit.

4.1. Monthly Metrics on Quality Control (IT, Firm Fixed Price) In the monthly report, the contractor shall document systematically the number and percentage of study reviews for which the two reviewers originally submitting reviews for a WWC contract through the SRG came to different conclusions regarding the eligibility of the study for review, the evidence ratings for study findings, or the statistical significance or magnitude of findings meeting WWC standards. The contactor shall tabulate the extent of inconsistencies for all reviews completed for the WWC, as well as separately by the type of standards (group design, RDD, or SCD) and study review protocol used.

The contractor shall break down the report for each content contractor and shall summarize recommendations for areas where additional training is needed. Each quarter the contractor shall produce a memo that has a summary of issues seen across contractors and specific training plans to reduce errors in the future. The TOAST contractor shall determine if errors are happening only by specific contracting firms or types of reviewers to determine the most effective training plan. The training will occur through subtask 5.6.

Each quarter, the contractor shall produce a memo on specific, targeted improvements to the SRG that can help reduce user confusion and reduce common errors. Examples of improvements include improved instructions, warnings, and errors aimed at reducing errors in future reports. Through subtask 2.2, the contractor shall make targeted improvements agreed to by the COR.

4.2. Peer Review (Non-IT, Cost Plus Fixed Fee)

All WWC reviews of studies included in WWC publications and any meta-analytic syntheses of findings in those publications are subject to review by an appropriately trained and certified WWC reviewer who serves as a peer reviewer. The contractor shall be responsible for maintaining a listing of reviewers and their areas of certification, updating the list annually. The contractor shall be responsible for ensuring peer reviews are conducted thoroughly, correctly, and promptly. The peer reviewer shall be separate from the team that conducts the review, develops the intervention, and conducts the study. Peer reviewers shall be certified in the appropriate version of the standards and have expertise in conducting multiple reviews for the WWC.

Contractors shall be expected to peer review studies that meet WWC standards. Each peer review shall be conducted within 5 business days and contractors shall be expected to check each Master Review Guide to ensure it is complete, accurate, and follows applicable guidance. The peer review is not a re-review of the study, but instead a quality check for errors against a standardized checklist (see an example of a checklist in Appendix D).

The contractor shall also be responsible for coordinating an appropriate external review of WWC publications, such as WWC Intervention Reports. The TOAST contractor is responsible for peer reviewing the text of the report and the underlying studies and synthesis. The purpose of the peer review is to ensure that the report is complete, accurate, and matches the information in the Master Review Guide. Each publication shall be reviewed by two reviewers and peer reviews shall be conducted within 10 business days.

WWC Practice Guides require external review by IES’ Standards and Review Office and its full text will not be peer reviewed by the contractor. However, the TOAST contractor is responsible for peer reviewing the underlying studies and the meta-analytic synthesis.

The studies shall be reviewed within 5 business days and the meta-analytic synthesis within 10 business days.

4.3 Audit Study Findings (Non-IT, Cost Plus Fixed Fee)

Twice yearly, the contractor shall conduct full re-reviews of three studies conducted in the previous 6 months. Through these re-reviews, or audits, the contractor shall aim to identify discrepancies between reviewer ratings or the characterization of findings from the studies reviewed. The purpose of the audit is to actively correct errors in the database and to find areas where increased reviewer training is needed.

The contractor shall provide a detailed description of, and justification for, the sampling of studies. The contractor shall log all necessary corrections and shall make the corrections, upon request. The contractor shall also document any needs for reviewer training, written reviewer guidance, or improvements to the OSRG implied by the results of the audit. This information shall also be made available to the STAT (as described in Task 5) for review and discussion.

4.4 Make Corrections to Records (IT, Cost Plus Fixed Fee)

Each month the contractor shall make updates or changes to 25 WWC records. This includes correcting any mistakes identified by current contractors, identified in the audit, or submitted to the Help Desk. Corrections include updating any field or fields in the review, but do not include conducting a new review or re-reviewing a study. This could include adding in new intervention names, adding ERIC links, and updating characteristics of a study. The contractor will need to re-publish reviews where corrections are made to the WWC website. This work shall be reported on in the monthly report.

Task 5. Public Help Desk and Training (Hybrid) The contractor shall be responsible for communicating about the WWC and its

Procedures and Standards. They shall do this through operating the public help desk that answers questions from the public, training new reviewers in WWC procedures and standards, and providing training and targeted assistance to current reviewers.

5.1. Operate the Public Help Desk (IT, Firm Fixed Price)

The contractor is responsible for maintaining the public help desk at Contact.WWC@ed.gov. The purpose of his help desk is to answer questions from the public, that is those who are not WWC contractors or IES grantees that have access to the Online SRG.

The contractor shall monitor the email address from 9am-5pm ET, excluding federal holidays. Contractors shall check the email at least every 90 minutes and provide a substantive reply, defined as one that answers the question, within 120 minutes. A substantive reply may be based on a form response but should be specifically tailored and addresses the need of the user. A response acknowledging that a reply will be issued at a later point is not a substantive response.

Up to two individuals will have ED.gov email addresses and the ability to access the help desk. These individuals shall be familiar with the WWC, its processes, and its procedures. The answers shall be closely monitored by a WWC certified reviewer to ensure that the answers are complete and accurate.

In the monthly report, the contractor shall report on 1) the number of email tickets, 2) the type of questions submitted, 3) the amount of time from receipt to sending a substantive reply, and 4) the mean and median time from receipt of email to a substantive reply being sent.

5.2. Provide Online Training (IT, Cost Plus Fixed Fee)

The contractor is responsible for maintaining the WWC Online Training courses. After a new version of the Procedures and Standards Handbook in subtask 3.3, the contractor is responsible for updating the existing online training. This includes developing new content and modifying existing content. All content should be developed by experts in the Procedures and Standards who have expertise in adult learning and have conducted multiple reviews for the WWC. The posting, coding, and maintaining of the training course shall happen through subtask 2.5.

5.3. Certify New Reviewers (IT, Cost Plus Fixed Fee)

The contractor is responsible for developing and maintaining a certification process. The current certification process includes several videos for different levels of certification and several multiple-choice questions. Every 6 months, the contractor shall review the questions potential reviewers frequently missed and why. The contractor shall then either propose changes to the training conducted under subtask 5.2 or alternate questions for the test. The posting, coding, and maintaining of the test shall happen through subtask 2.5.

The current SWAT contractor will develop the training process for version 5.0 standards.

The TOAST contractor will be responsible for maintaining the process created by SWAT mailto:Contact.WWC@ed.gov and then developing any new training under version 6.0.

The contractor shall maintain a list of WWC certified reviewers and send IES an updated list of WWC certified reviewers and the version of the standards that each reviewer is certified in. On a quarterly basis, with the monthly report, the contractor shall submit the updated list of certified reviewers to IES. The contractor is also responsible for updating the list of certified reviewers on the WWC website.

5.4. Recertify Reviewers (IT, Cost Plus Fixed Fee)

After the release of a new version of the handbook, existing reviewers will need to be certified in the latest version of the standards. This process must require reviewers to go through the new reviewer training, as is the case when moving from version 4.1 to version 5.0 or a less intensive process such as a series of videos and answering multiple choice questions as happened from version 4.0 to version 4.1.

Under this task the contractor shall develop a plan for recertification under any new version of the handbook. If necessary, the contractor shall develop any videos, webinars, or testing modules necessary for recertification. The contractor shall answer any questions from those undergoing recertification and grade any assessments. In the monthly report, the contractor shall identify how many new reviewers have been certified.

5.5. Provide Annual Training and Feedback (Non-IT, Firm Fixed Price) Each year, the contractor shall host one annual “refresher” training. This training will last between 1-2 hours and is designed for current reviewers to get a comprehensive overview of training.