Attachment 1 TOAST TO4 PWS.pdf

DRAFT

Task Order 4

The following tasks and subtasks are included in the scope of work. The period of performance for this task is 24 months, with the expectation that the majority of work will be completed from 2025 to 2026. Work may continue in tasks, especially Task 2 and subtask 3.3, to allow for coordination with the digital modernization teams and for other IES staff and contractors.

Task 1. Management (Hybrid, Firm Fixed Price)

The contractor shall manage task orders in an efficient manner that fosters communication with the COR and other WWC contractors. The contractor shall have sound project management processes and personnel who can manage multiple task orders simultaneously, as it is anticipated that several task orders can have overlapping deadlines.

Typical task order management activities are outlined below.

1.3. Biweekly Meetings (Non-IT)

The project director and other key staff that the project director deems are necessary shall meet with the COR and other appropriate Department staff via biweekly conference calls to be held at regularly scheduled intervals throughout the duration of the contract. The purpose of the meetings is to brief the Department on progress and challenges related to the tasks in this contract. The Contractor shall send an agenda to the COR 3 days in advance of the meeting and a meeting summary to the COR within 3 business days after the meeting is complete.

In addition, the Contractor shall be available for meetings to address unexpected issues that arise during the contract period. These meetings may occur on an as-needed basis at the request of the COR or the Contractor and will take place via conference call. The contractor should assume one meeting a week in the first six months of the contract and approximately one meeting per month thereafter.

1.4. Monthly Report (Non-IT)

The contractor shall submit a monthly report that summarizes progress made in each task and subtask, and shall specifically include the metrics specified in the tasks below and listed in Appendix B. The report shall highlight progress made and any challenges faced.

In addition, the monthly report shall include proposed work and expected challenges for the next month.

Contractor shall provide data on earned value in a specific format specified in Appendix C. Any deviations in schedule or cost shall be fully explained in the risk register.

Additionally, the monthly report shall include the IT reporting sheet and the contractor personnel listing using the templates provided in the solicitation.

The contractor shall submit the report on the sixth business day of every month for the duration of the contract and shall be prepared to discuss it in the next biweekly call with the COR.

1.5. Cross Contractor Meetings (Non-IT)

The contractor shall be available to meet up to three times a year for cross-contractor meetings. These meetings shall be held in Washington, DC and/or virtually. The COR will provide ample notice of the time and the agenda for the meeting. The contractor may be asked to present or facilitate a discussion for part of the agenda.

Task 2: Website, Data, and Online Tools (IT, Cost Plus Fixed Fee) The website is the primary way for the WWC to share their products and resources with the world. It is the responsibility of the TOAST contractor to ensure that the Online SRG is functional. If problems arise, it is the primary responsibility of the TOAST contractor to quickly and accurately resolve any issues.

2.1. Creation of IT Reporting Materials

Within 4 weeks of contract award or the start of an option year, the contractor shall a submit work breakdown structure of all deliverables, down to the individual personnel level for cost plus fixed fee deliverables, in Microsoft Project format, a 2-3 page overview of the quality management plan, and a 2-3 page overview of the risk management plan. Guidance for these documents is provided in Appendix C.2.1.

2.2 Develop a Help Desk for Contractors

The contractor shall be responsible for monitoring the email address WWC.Contractor.Help.Desk@ed.gov and Help Desk requests sent through the OSRG.

This help desk will receive all inquiries from content contractors about problems with the OSRG and publishing content on the website.

The contractor shall monitor the email address from 9am-5pm ET, excluding federal holidays. Contractors shall check the email at least once an hour and provide a substantive reply, defined as one that solves the problem or follows up with the content contractor about when to expect a resolution, within 90 minutes. A substantive reply shall not be a form response, nor shall it be an acknowledgement of the email with the promise of a later response. Substantive replies shall be specifically tailored and address the needs of users, requests more information, or asks to set up a meeting to discuss the issue further. The contractor shall follow up on all issues raised by the content contractors and promptly inform them when issues have been resolved. If a task requires a fix to the OSRG, that should occur under subtask 2.3.

Up to two individuals will have email addresses and the ability to access the help desk.

These individuals shall be certified in WWC Group Design Standards, version 4.1 and both general group designs and advanced group designs, version 5.0. These individuals have expertise in the functionality and design of the Online Study Review Guide and experience conducting reviews.

With the agenda for the biweekly meeting (see subtask 1.3), the contractor shall submit a table summarizing the work of this task. It should contain a list of emails received since mailto:WWC.Contractor.Help.Desk@ed.gov the previous call, a description of each issue raised, the status of each ticket, and the expected time of resolution. In the monthly report, the contractor shall report on 1) the number of email tickets submitted by content contractors, 2) the proportion of tickets that are work stoppages, 3) the proportion of tickets that are errors versus enhancements, 4) the amount of time from receipt to sending a substantive reply, and 5) the mean and median time from receipt of email to issue resolution.

2.3 Maintain and Update the Online Study Review Guide

The contractor shall be responsible for maintaining and upgrading the WWC’s OSRG.

The OSRG tool is used by all WWC contractors, as well as other IES contractors and grantees to review studies against WWC standards and protocols. It is housed in the NCES Members Site in the IES Data Center at the URL:

https://members.nces.ed.gov/OSRG/Account/Login?ReturnUrl=%2fosrg%2f.

Information about the Online SRG can be found here: https :/ / youtu.be/ W99s 8RlNXis , https://youtu.be/HT6qkZzpjAw, and https://youtu.be/x3Jy0GaLYf0.

The contractor shall be responsible for ensuring that the tool is functional and is accurately completing reviews. On a weekly basis, the contractor shall release updates to the SRG that fix aspects of the tool that are not working for the reviewers. (See subtask

2.2 for more information). The contractor shall design improvements and enhancements that are cost-effective and are long-term solutions that would work for future reviews, as opposed to short-term patches.

The contractor shall have sufficient capacity to quickly address any issues that cause work stoppages, as well as make long term enhancements. This means that the contractor shall have the ability to bring in additional staff when needed to quickly scale up the level of effort so that the SRG remains functional, and products are not delayed. Contractors are expected to resolve issues that are work stoppages within 1 week and other issues within 4 weeks.

The contractor shall also ensure all functionalities of the OSRG are sufficiently documented through the User’s Guide through subtask 5.8. The User’s Guide shall be consistently up to date and designed in a way to be useful to reviewers and efficient to update.

The contractor is also responsible for adding and maintaining user accounts and ensuring users have completed necessary cybersecurity training. On a quarterly basis the contractor shall review all accounts to ensure that users are actively on contract and have appropriate trainings to use the tool.

With the agenda for the biweekly meeting (see subtask 1.3), the contractor shall submit a table of open tickets, their status, and estimated level of effort to resolve the issue, and estimated completion date. In the monthly report, the contractor shall report on 1) number of tickets opened, 2) number of tickets closed, 3) mean and median time for ticket resolution, 4) number of labor hours for subtask 2.2, and 5) number of tickets not yet addressed. The contractor shall also provide an Excel file that has a list of all studies https://members.nces.ed.gov/OSRG/Account/Login?ReturnUrl=%2fosrg%2f https://youtu.be/W99s8RlNXis https://youtu.be/HT6qkZzpjAw https://youtu.be/x3Jy0GaLYf0 currently under review, in the format specified in Appendix B as part of the monthly report.

2.4 Maintain the WWC Website

The WWC website is designed to be responsive to user needs, efficient, and adaptable.

Until the new WWC website is released by the Digital Modernization team, the WWC contractor shall maintain the existing site. This includes, but is not limited to, adding new communications or resource products to the website, announcing events, publishing new reviews, and releasing new products. The COR and TOAST contractor will work with the WWC content contractors to develop a schedule for publishing study reviews and publications. The contractor is expected to meet all dates on the agreed upon schedule and release all reports on the agreed upon date.

For requests to publish new reviews, communications products, events, and other announcements, the content contractors shall submit requests through the contractor help desk (see subtask 2.1). The contractor will be responsible for assigning each product a ticket number and ensuring that new content is posted within 2 weeks and on the agreed upon date.

When developing new content, the contractor shall send the COR a link to preview any pages on the IES Development Server. All updates to the website will be approved by the COR prior to implementation.

In the monthly report, the contractor shall provide: 1) the number of new website changes made, 2) the mean and median amount of time for tickets to be addressed, 3) the number and percentage of tickets that take more than 2 weeks to resolve.

2.5 Enhancements to the WWC Website

The contractor shall work with the COR and the Digital Modernization website to provide expertise in designing the new website. This includes but is not limited to: 1) provide feedback on wireframes, 2) conducting user testing of the website and CMS, 3) providing requirements on the website, 4) drafting website text, and 5) using the CMS to input and tag content. The contractor shall be available to help with all migration activities. The contractor shall work with the COR to prioritize enhancements.

2.6 Enhancements to WWC Underlying Data

The contractor shall work with the COR and Digital Modernization contractors to prepare the underlying data in the WWC to be migrated to a new website. This includes but is not limited to: 1) migrating the underlying database to PostgreSQL, 2) adding new fields to standardize outcome domains, taxonomy terms, and other controlled vocabularies, 3) address areas of missing data, such as grade level and topic tags, 4) adding versioning variables, and 5) developing an API. The contractor shall work with the COR to prioritize enhancements.

2.7 Enhancements to the OSRG

The contractor shall periodically review the OSRG. Enhancements may include improving the reconciler view, improving the way to input information to reduce errors, improving the layout, and adding the ability to review new types of studies. It is also within scope to make major changes to the functionality or design.

At the start of each task order, the COR and contractor shall develop an estimated budget and brainstorm topics. The TOAST contractor shall correspond with existing WWC reviewers to determine issues that they are experiencing and potential enhancements that may be helpful. This information should be used to propose a series of potential updates and their level of effort on an ongoing basis. Upon gaining the COR’s approval, the contractor will develop a timeline and budget for development. In the monthly report, the contractor shall provide an update on each enhancement and a discussion of any changes or modifications to the timeline.

2.8 Convene a Practitioner Working Group to Advise on WWC Content for Non- Researchers The contractor shall develop a practitioner work group, the SWEEP, to advise the WWC on the prioritization and presentation of WWC-produced content to non-researchers in WWC publications and on the WWC website. During TO4, the SWEEP should meet through 4 formal sessions and 6 informal sessions with a small subset of members. The informal sessions should have 3-6 members that meet when IES needs rapid feedback on specific wireframes or materials.

For each meeting, the contractor shall send an agenda and any materials at least two weeks in advance. The contractor shall include a summary of all calls and the progress in the monthly report.

At the end of task order, the contractor shall develop a summary of the practitioner working group’s discussion and decision points. Upon COR approval, the summary shall be made 508 compliant and posted to the WWC website. The contractor is responsible for ensuring that the meeting materials are posted in a timely fashion through subtask 2.3.

Task 3. Procedures and Standards Development (Non-IT, Cost Plus Fixed Fee) The contractor shall convene a Statistical, Technical, and Analysis Team (STAT) that will support the reviews of the WWC. The STAT is composed of two sets of groups, small working groups that work on specific issues (see subtask 3.1) and the All-STAT which is a large group comprised of members with diverse methodological expertise.

The task will support consideration, by expert methodologists familiar with the WWC, of issues related to WWC standards for assessing the internal validity of findings from different types of impact study designs; WWC procedures for characterizing, presenting, and synthesizing study findings that meet WWC standards; and WWC reviewer guidance for applying current standards and procedures. Members of the STAT will include employees of the contractor team and consultants to the TOAST contract. Other WWC contractors and IES staff will be members of the STAT and their time will be budgeted for under their respective contracts and salaries.

3.1. Propose and Convene STAT Small Working Groups

The contractor shall continue to host the previously approved small working groups. For each small working group meeting, the contractor shall send an agenda and any materials to the COR at least one week in advance and to the working group at least 24 hours in advance of the meeting. The contractor shall include a summary of all calls and the progress of the working group in the monthly report.

At the end of each working group, the contractor shall develop a summary of the small working group’s discussion and decision points. Upon COR approval, the summary shall be made 508 compliant and posted to the WWC website. The contractor is responsible for ensuring that the meeting materials are posted in a timely fashion through subtask 2.5.

3.2. Propose and convene an All-STAT

The All-STAT shall be a large working group designed to provide interdisciplinary feedback on methodological issues facing the WWC and review proposed changes to the Procedures and Standards Handbooks. The All-STAT shall include 1 or 2 representatives from each active WWC contract, the IES WWC team, and up to 6 representatives from the small working groups.

The TOAST contractor shall convene the All-STAT at least 4 times a year and up to 6 times a year. The contractor shall have a specific purpose for the meeting and shall develop agendas in a way to seek feedback and have a robust discussion.

Every 6 months the contractor shall develop the proposed objectives for upcoming meetings. Upon COR approval, the contractors shall schedule a date. At least 2 weeks before the meeting, the contractor shall submit the agenda and any meeting materials and slides. Final meeting materials shall be sent to the All-STAT at least 5 days before the meeting.

Ten days after the meeting the contractor shall submit a summary of the meeting. The final version of the meeting summary shall be made 508 compliant and posted on the WWC website with any meeting materials. The contractor is responsible for ensuring that the meeting materials are posted in a timely fashion through subtask 2.5.

3.3. Develop an Updated Procedures and Standards Handbook.

The contractor shall develop an updated Procedures and Standards Handbook that builds upon the discussions and decision points of the small working groups and the All-STAT.

Updates may include changes to the existing content, new chapters, or complete rewriting of sections.

During TO4, the contractor is expected to develop a memo that outlines potential changes and the impact of the changes. This memo will be the basis of an all-STAT meeting.

Once feedback from the STAT is obtained and the memo is approved by IES, the contractor shall develop an outline of the handbook and a detailed timeline for completion. The contractor may begin work in drafting the handbook under TO4.

If work begins under this task, the contractor shall expect to deliver the handbook on a chapter-by-chapter basis. There will be at least 2 rounds of IES feedback at the chapter level, as well as at least 2 rounds of feedback when the handbook is complete. The handbook will then be sent to IES staff and the STAT for feedback and the contractor shall expect at least 3 rounds of review after receiving feedback to get final approval to post. The final handbook will be posted as a 508 compliant PDF.

Task 4. Quality Control (Hybrid) The Quality Control task is responsible for ensuring that the data on the WWC website is accurate and complete. While the content contractors are responsible for ensuring the quality of the studies reviewed, TOAST is responsible for ensuring that the overall quality of the content is high quality and accurate. The contractor achieves this through three functions: quality control metrics, peer review, and audit.

4.1. Monthly Metrics on Quality Control (IT, Firm Fixed Price) In the monthly report, the contractor shall document systematically the number and percentage of study reviews for which the two reviewers originally submitting reviews for a WWC contract through the SRG came to different conclusions regarding the eligibility of the study for review, the evidence ratings for study findings, or the statistical significance or magnitude of findings meeting WWC standards. The contactor shall tabulate the extent of inconsistencies for all reviews completed for the WWC, as well as separately by the type of standards (group design, RDD, or SCD) and study review protocol used.

The contractor shall break down the report for each content contractor and shall summarize recommendations for areas where additional training is needed. Each quarter the contractor shall produce a memo that has a summary of issues seen across contractors and specific training plans to reduce errors in the future. The TOAST contractor shall determine if errors are happening only by specific contracting firms or types of reviewers to determine the most effective training plan. The training will occur through subtask 5.6.

Each quarter, the contractor shall produce a memo on specific, targeted improvements to the SRG that can help reduce user confusion and reduce common errors. Examples of improvements include improved instructions, warnings, and errors aimed at reducing errors in future reports. Through subtask 2.2, the contractor shall make targeted improvements agreed to by the COR.

4.2. Peer Review (Non-IT, Cost Plus Fixed Fee)

All WWC reviews of studies included in WWC publications and any systematic review or meta-analytic syntheses of findings in those publications are subject to review by an appropriately trained and certified WWC reviewer who serves as a peer reviewer. The contractor shall be responsible for maintaining a listing of reviewers and their areas of certification, updating the list annually. The contractor shall be responsible for ensuring peer reviews are conducted thoroughly, correctly, and promptly. The peer reviewer shall be separate from the team that conducts the review, develops the intervention, and conducts the study. Peer reviewers shall be certified in the appropriate version of the standards and have expertise in conducting multiple reviews for the WWC.

Contractors shall be expected to peer review studies that meet WWC standards. Each peer review shall be conducted within 5 business days and contractors shall be expected to check each Master Review Guide to ensure it is complete, accurate, and follows applicable guidance. The peer review is not a re-review of the study, but instead a quality check for errors against a standardized checklist (see an example of a checklist in Appendix D).

The contractor shall also be responsible for coordinating an appropriate external review of WWC publications, such as WWC Intervention Reports. The TOAST contractor is responsible for peer reviewing the text of the report and the underlying studies and synthesis. The purpose of the peer review is to ensure that the report is complete, accurate, and matches the information in the Master Review Guide. Each publication shall be reviewed by two reviewers and peer reviews shall be conducted within 10 business days. The contractor should expect up to 5 publications to review under this task order.

WWC Practice Guides require external review by IES’ Standards and Review Office and will not be peer reviewed by the contractor. However, the TOAST contractor is responsible for peer reviewing the underlying studies and the meta-analytic synthesis.

The studies shall be reviewed within 5 business days and the meta-analytic synthesis within 10 business days.

4.3 Audit Study Findings (Non-IT, Cost Plus Fixed Fee)

Twice yearly, the contractor shall conduct full re-reviews of three studies conducted in the previous 6 months. Through these re-reviews, or audits, the contractor shall aim to identify discrepancies between reviewer ratings or the characterization of findings from the studies reviewed. The purpose of the audit is to actively correct errors in the database and to find areas where increased reviewer training is needed.

The contractor shall provide a detailed description of, and justification for, the sampling of studies. The contractor shall log all necessary corrections and shall make the corrections, upon request. The contractor shall also document any needs for reviewer training, written reviewer guidance, or improvements to the OSRG implied by the results of the audit. This information shall also be made available to the STAT (as described in Task 5) for review and discussion.

4.4 Make Corrections to Records (IT, Cost Plus Fixed Fee)

Each month the contractor shall make updates or changes to approximately 25 WWC records. This includes correcting any mistakes identified by current contractors, identified in the audit, or submitted to the Help Desk. Corrections include updating any field or fields in the review, but do not include conducting a new review or re-reviewing a study.

This could include adding in new intervention names, adding ERIC links, and updating characteristics of a study. The contractor will need to re-publish reviews where corrections are made to the WWC website. This work shall be reported on in the monthly report.

Task 5. Public Help Desk and Training (Hybrid) The contractor shall be responsible for communicating about the WWC and its Procedures and Standards. They shall do this through operating the public help desk that answers questions from the public, training new reviewers in WWC procedures and standards, and providing training and targeted assistance to current reviewers.

5.1. Operate the Public Help Desk (IT, Firm Fixed Price)

The contractor is responsible for maintaining the public help desk at Contact.WWC@ed.gov. The purpose of his help desk is to answer questions from the public, that is those who are not WWC contractors or IES grantees that have access to the Online SRG.

The contractor shall monitor the email address from 9am-5pm ET, excluding federal holidays. Contractors shall check the email at least every 90 minutes and provide a substantive reply, defined as one that answers the question, within 120 minutes. A substantive reply may be based on a form response but should be specifically tailored and addresses the need of the user. A response acknowledging that a reply will be issued at a later point is not a substantive response.

Up to two individuals will have ED.gov email addresses and the ability to access the help desk. These individuals shall be familiar with the WWC, its processes, and its procedures. The answers shall be closely monitored by a WWC certified reviewer to ensure that the answers are complete and accurate.

In the monthly report, the contractor shall report on 1) the number of email tickets, 2) the type of questions submitted, 3) the amount of time from receipt to sending a substantive reply, and 4) the mean and median time from receipt of email to a substantive reply being sent.

5.2. Provide Online Training (IT, Cost Plus Fixed Fee)

The contractor is responsible for maintaining the WWC Online Training courses. After a new version of the Procedures and Standards Handbook in subtask 3.3, the contractor is responsible for updating the existing online training. This includes developing new content and modifying existing content. All content should be developed by experts in the Procedures and Standards who have expertise in adult learning and have conducted multiple reviews for the WWC. The posting, coding, and maintaining of the training course shall happen through subtask 2.5.

5.3. Certify New Reviewers (IT, Cost Plus Fixed Fee)

The contractor is responsible for developing and maintaining a certification process. The current certification process includes several videos for different levels of certification and several multiple-choice questions. Every 6 months, the contractor shall review the questions potential reviewers frequently missed and why. The contractor shall then either mailto:Contact.WWC@ed.gov propose changes to the training conducted under subtask 5.2 or alternate questions for the test. The posting, coding, and maintaining of the test shall happen through subtask 2.5.

The 2018-2023 SWAT contractor will develop the training process for version 5.0 standards. The TOAST contractor will be responsible for maintaining the process created by SWAT and then developing any new training under future versions of the standards.

The contractor shall maintain a list of WWC certified reviewers and send IES an updated list of WWC certified reviewers and the version of the standards that each reviewer is certified in. On a quarterly basis, with the monthly report, the contractor shall submit the updated list of certified reviewers to IES. The contractor is also responsible for updating the list of certified reviewers on the WWC website.

5.5. Provide Annual Training and Feedback (Non-IT, Firm Fixed Price) Each year, the contractor shall host one annual “refresher” training. This training will last between 1-2 hours and is designed for current reviewers to get a comprehensive overview of training. This differs from targeted training under subtask 5.6 because, unlike the targeted training, the annual refresher is broader and is designed to walk reviewers through a complete review. The contractor shall briefly focus on areas where task 4 has indicated that reviewers need additional guidance or training.

This training shall occur in the spring of each year, ideally around March 15th. By January 15th each year the contactor shall submit an outline of the propose training. The COR and the contractor shall discuss the outline in the biweekly meeting. By March 1, the contractor shall submit the slides and materials for the training and shall expect at least one round of revision. The training shall take live questions and answers and shall be archived on the WWC website no later than 2 weeks after the training.

5.6. Provide Targeted Training and Answer Targeted Questions from Reviewers (Non-IT, Cost Plus Fixed Fee) The contactor shall provide targeted training to reviewers. This training may be in the form of general training available to all reviewers, such as a brief webinar, video, memo, or email. Trainings may also be consultations between TOAST and reviewers to answer specific technical questions. All trainings shall be approved by the COR in advance.

5.7 Communicate About WWC Standards and Review Processes (Non-IT, Cost Plus Fixed Fee) The contractor shall produce dissemination activities that communicate about the processes that the WWC uses to review studies. The contractor shall do this through webinars, videos, briefs, updated website information (including FAQs), or infographics.

The contractor shall budget for one webinar, one video, one brief, and one infographic.

Six weeks after contract award the contractor shall propose the topic and audience for these materials. The proposal shall be sufficiently tailored and based on documented needs of reviewers. The COR and contractor shall discuss the proposed ideas and agree on topics, timeline, and milestones.

5.8 Develop and Maintain a Users Guide (IT, Cost Plus Fixed Fee) The contractor shall create and maintain a comprehensive reviewers guide that is the primary source for reviewer guidance. This should be built in a way that is easy to enhance and improve—that is that it should be coded as website text, as opposed to a PDF or word document. It should include all relevant information from the Handbook on how to conduct a review in a well-designed, user-friendly manner. The guide shall be updated quarterly with new measures, additionally, the guide may be updated more frequently if there is a major update.

Task 6. Reviews and Syntheses (Non-IT, Cost Plus Fixed Fee) The TOAST contractor shall periodically review studies for the WWC and synthesize the findings from multiple reviews to generate syntheses. These reviews will occur at the discretion of IES and shall be done by a separate team as those performing peer reviews and quality checks. Depending on the scope and purpose of the review, not all subtasks may be required for each project and the number of reviews and syntheses will vary widely throughout the year.

6.3 Screen Studies for the WWC

The TOAST contractor shall screen publicly available, ED funded studies in ERIC to determine if they are eligible for WWC review. This will be done on a routine basis to determine which studies content contractors should prioritize for review. The COR will work with the contractor to determine the scope and volume of studies to screen. Eligible studies should be included in the OSRG database as “Eligible, but not yet assigned for review” so that other contractors may review the studies or the TOAST contractor may review the studies under subtask 6.4.

For budgeting assumptions, the contractor shall assume that they shall screen approximately 2,500 studies under this task order. The contractor shall prioritize studies in order of publication date. They shall assume that the majority they will be able to screen as “not eligible” based on the abstract. Others may require the reviewer to view the full text. The contractor may access the full text through the National Library of Education.

6.4 Review Studies for the WWC

The contractor shall review up to 50 studies funded by the Department and screened under Task 6.1 above. The contractor shall use WWC certified reviewers to review the studies using the specified version of WWC standards and the online SRG. Per the WWC Procedures and Standards Handbook, each study shall be reviewed by two WWC certified reviewers and then the reviews shall be reconciled by a senior reviewer. The senior reviewer shall generate a master review guide, or MRG. Some studies may have been previously reviewed under another WWC protocol and the contractor can import the information from the previous review to review against the new protocol.

After the master SRGs are complete, the contractor shall submit the SRGs for peer review. The contractor is expected to address all concerns raised in the review. After the peer review is complete, the contractor shall publish the reviews to the WWC website.

The contractor shall prepare, once per year, a memo summarizing the results of the reviews of Department-funded studies, including the research ratings of each study, the populations and settings included in the studies, the intervention and comparison group conditions, and the findings meeting WWC standards. The contractor shall revise this memo within 2 weeks of receiving comments on the draft memo from the COR.