Attachment 1 - PWS.docx

WWC-PESTO PWS, Version Date: 3.16.2021 PWS, Version Date: 3.16.2021

U.S. Department of Education Institute of Education Sciences National Center for Education Evaluation and Regional Assistance

PERFORMANCE WORK STATEMENT

VERSION 1.00

MARCH 16, 2021

What Works ClearinghouseTM Preschool-to-Postsecondary Evidence Synthesis Task Orders (“WWC-PESTO”) Task Order 01: Assisting Students Struggling with Behavior in Grades K-12

I. Introduction

The United States Department of Education (the Department), Institute of Education Sciences (IES), National Center for Education Evaluation and Regional Assistance (NCEE) is procuring contractor support to conduct systematic reviews of research evidence and develop a series of What Works ClearinghouseTM (WWC) publications on various topics.

To support the development of WWC practice guides, intervention reports, and related products in various topic areas, the Department plans to award a multiple-award Blanket Purchase Agreement (BPA) for Preschool-to-Postsecondary Evidence Synthesis Task Orders (WWC-PESTO). The Department expects to award at least eight WWC-PESTO task orders during a 60-month period between Fiscal Years 2021 and 2025 to support the development of evidence-based practice recommendations to educators in preschool, elementary, secondary, or postsecondary education. Contractor proposals for the first task order will be the basis for determining eligibility to bid on subsequent task orders. The value of each task order will not exceed $3.75 million. No prime contractor awarded a BPA will be eligible to be awarded PESTO task order contracts exceeding a ceiling of $7.5 million per contractor.

WWC-PESTO Task Order 01 will focus on the topic of “Assisting Students Struggling with Behavior in Grades K-12.” The period of performance for Task Order 01 will have a Base Period of 24 months with two partially overlapping optional periods, each also 24 months in length and beginning 13 months and 25 months after award, that may extend the period of performance to 48 months. The Base Period will focus on conducting a systematic review of evidence and meetings of a panel of researchers and practitioners for a new WWC practice guide on “Assisting Students Struggling with Behavior in the Elementary Grades.” In subsequent optional periods, if authorized by the Department, the contractor shall build upon the reviews done for the first practice guide and conduct additional study reviews and panel meetings to develop a second practice guide or related products of interest to educators, researchers, and program developers.

The contractor shall prepare publications under Task Order 01 that are informed by a systematic review of evidence on the impacts of school-based behavior interventions on students in kindergarten through Grade 12 who have been identified as struggling with behavior, at risk of social and behavioral problems, or in need of interventions to address externalizing behaviors, internalizing behaviors, or both. In the context of Multi-Tiered Systems of Support (MTSS) in behavior, the focus of this task order will include targeted (Tier 2) or intensive (Tier 3) interventions as well as related whole-class and whole-school (Tier 1) supports for all students, but with a particular focus on assisting students at elevated risk of behavior problems and in likely need of Tier 2 or Tier 3 interventions. The student populations of interest--those in need of behavioral supports--may or may not have Individualized Education Programs (IEPs) specifying behavior interventions or supports.

The primary target audiences for publications and products prepared under PESTO Task Order 01 will include: classroom educators, special education teachers, school psychologists, social workers, paraprofessionals, parents, school leaders, teacher educators, technical assistance providers, and school district personnel responsible for preservice or in-service training of educators.

II. Authorizing Legislation

This contract is authorized by Title I, Part D of the Education Sciences Reform Act of 2002 (Public Law 107-279), Section 171(b).

(b) MISSION.—The mission of the National Center for Education Evaluation and Regional Assistance shall be—

(1) to provide technical assistance;

(2) to conduct evaluations of Federal education programs administered by the Secretary (and as time and resources allow, other education programs) to determine the impact of such programs (especially on student academic achievement in the core academic areas of reading, mathematics, and science);

(3) to support synthesis and wide dissemination of results of evaluation, research, and products developed; and

(4) to encourage the use of scientifically valid education research and evaluation throughout the United States.

The need for educators and policy makers to identify “evidence-based” practices to improve student outcomes and other outcomes relevant for education was also emphasized in the 2015 reauthorization of the Elementary and Secondary Education Act (Public Law 114-95). In particular, ESEA was amended to include definitions of “moderate evidence” and “strong evidence” (as defined in 20 U.S.C. 7801, section 21), corresponding with statistically significant and favorable findings from “well-designed and well-implemented” quasi-experimental and experimental studies. In addition, WWC-PESTO task orders including a focus on the effectiveness of interventions for students with disabilities or at risk of identification for special education may be supported with funds authorized under Section 664 of the Individuals with Disabilities Education Act (Public Law 101-476), Section 664(e).

III. Existing What Works Clearinghouse Contracts

As of March 2021, the WWC investment operates through eight distinct contracts:

· ED-IES-16-C-0024 (with Abt Associates): Postsecondary Education and Postsecondary Preparation Evidence Reporting (the WWC-PEPPER Contract), is focused on conducting study reviews and preparing intervention reports and practice guides on topics related to postsecondary education. Following the completion of the WWC-PEPPER contract, the Department expects to rely primarily on WWC-PESTO task orders to conduct systematic reviews of evidence in the corresponding topic areas.

· 91990018C0019 (with the American Institutes for Research): What Works Clearinghouse Statistics, Website, and Training (herein referred to as the WWC-SWAT or SWAT Contract), is focused on supporting management and improvement of the WWC website, WWC Help Desk, WWC standards and procedures development, WWC reviewer training and certification, and WWC study review tools including the online Study Review Guide (SRG)

· 91990018C0009 (with Mathematica): What Works Clearinghouse Organization of Reporting of Evidence on Graduation, Achievement, and Nonacademic Outcomes (the WWC-OREGANO Contract), is focused on conducting study reviews and preparing intervention reports on topics in early childhood through Grade 12 education. Following the completion of the WWC-OREGANO contract, the Department expects to rely primarily on WWC-PESTO task orders to conduct systematic reviews of evidence in the corresponding topic areas.

· 91990020D0005 (with Development Services Group): What Works Clearinghouse Ad-hoc Reviews of Individual Studies in Education Indefinite Delivery, Indefinite Quantity Contract (the WWC-ARISE Contract), is focused on conducting reviews of individual studies at the request of the Department, for example for ED grant competitions, as well as on helping IES create new or updated WWC products other than practice guides

· Task Order 0001 of blanket purchase agreement 91990018A0006, contract 91990018F0375 (with Instructional Research Group): What Works Clearinghouse Guides to Research that Assist Practitioners in Education (WWC-GRAPE) Task Order 1 has been focused on creating a WWC practice guide for release in March 2021, Assisting Students Struggling with Mathematics: Intervention in the Elementary Grades

· Task Order 0002 of blanket purchase agreement 91990018A0005, contract 91990019F0319 (with 2M Research): What Works Clearinghouse Guides to Research that Assist Practitioners (WWC-GRAPE) Task Order 2 is focused on creating a WWC practice guide on Promoting Social and Behavioral Success for Learning in Elementary Schools. In contrast with PESTO Task Order 1, GRAPE Task Order 0002 is focused on whole-school and whole-class (Tier 1) practices for the general student population, rather than those identified as at elevated risk of behavior problems.

· Task Order 0003 of blanket purchase agreement 91990018A0006, contract 91990019F0399 (with Instructional Research Group): What Works Clearinghouse Guides to Research that Assist Practitioners in Education (WWC-GRAPE) Task Order 3 is focused on creating a WWC practice guide on Assisting Students Struggling with Reading: Intervention in Grades 4–9

· Task Order 0004 of blanket purchase agreement 91990018A0006, contract 91990020F325 (with Instructional Research Group): What Works Clearinghouse Guides to Research that Assist Practitioners in Education (WWC-GRAPE) Task Order 4 is focused on creating a WWC practice guide and related intervention reports on Preparing Young Children for School

IV. WWC Preschool-to-Postsecondary Evidence Synthesis Task Orders (WWC-PESTO) Scope of Work

Through WWC-PESTO task orders, IES intends to support the preparation of practice guides, intervention reports, and related products in preschool, elementary, secondary, or postsecondary education. These products will provide stakeholders with research-based, actionable information to inform choices regarding education practices or the adoption of interventions to improve student outcomes or other outcomes relevant to education. WWC practice guides (https://ies.ed.gov/ncee/wwc/PracticeGuides) provide educators with a coherent set of actionable recommendations on practices with respect to a particular issue, problem, or challenge that educators face. The practice guide connects each recommendation to the supporting research evidence, and describes the quality of that evidence. Other WWC systematic reviews (https://ies.ed.gov/ncee/wwc/Publication#/ContentTypeId:1) summarize evidence of the effectiveness of particular interventions--“branded” or otherwise--in the field of education, whether commercial or non-commercial.

Under each PESTO task order, the contractor shall organize and partner with a panel of researchers and practitioners with relevant content expertise to develop the topic area review protocol to identify and assess research for the WWC to review in the topic area that is the focus of each task order. Reviews of studies shall occur in a timely manner using reviewers certified in the most recent version of the WWC design standards (https://ies.ed.gov/ncee/wwc/Handbooks) who have access to the corresponding SRG.

The contractor shall prepare practice guides and related products to incorporate the best available research evidence on the topic, supplementing WWC’s systematic reviews of studies with the content knowledge and professional expertise of the panel members. Once the contractor has reported on and synthesized a body of relevant studies reviewed according to WWC procedures and standards, the contractor shall meet with the panel to assess where there is evidence that specific practices or intervention components improve student outcomes and will consider which components of effective interventions are feasible for educators to implement with the help of a practice guide. Practice guide authors—who generally include key contractor personnel distinct from practice guide panel members— shall combine panel recommendations with their own expertise with the findings of rigorous research, when available, to describe the evidence base informing the panel’s recommendations for addressing educators’ challenges in the topic area of interest. In addition, the contractor shall include in the practice guide information on how to carry out each recommendation and address possible implementation roadblocks. For specific interventions for which there is evidence of effectiveness from multiple studies, the contractor may also, on the advice of the panel and with IES approval, review additional studies relevant to the topic area focus of the task order but not used to inform recommendations in the practice guide, and prepare additional reports characterizing the effectiveness of corresponding interventions.

IES-sponsored practice guides and other publications are subject to rigorous external peer review. There are two phases of peer review. First, all study reviews contributing to these publications will receive peer review with support from the SWAT contractor to ensure that those reviews are accurate and complete. This ensures that the publications are based on the best available evidence and adequately address the topic of the guide. Second, there will be external peer review, through the IES Standards and Review Office (SRO), of the text of each practice guide. In order to ensure the best quality publication possible, the contractor shall revise each practice guide as necessary to meet the concerns of external peer reviews and gain the approval of SRO. In the case of intervention reports, the SWAT contractor supports the peer review of the text of the publication.

Written summaries, presentations, webinars, and videos highlighting a new practice guide and related products are also within the scope of this task order.

V. Period of Performance

This task order will have a 24-month Base Period where the contractor is expected to develop and prepare for publication an initial practice guide. Two partially overlapping optional periods—Option Period 1 running from months 13 to 36 after award, and Option Period 2 running from months 25 to 48 after award—would allow the contractor to prepare a second practice guide, related intervention reports, and other products and dissemination activities that complement the first practice guide.

VI. Task Order Type

The Department envisions each task order including a combination of 9 Firm Fixed Price tasks and 2 Cost Reimbursement tasks, as specified below.

VII. Task-Specific Requirements

Task 1: Meetings with the Department (Base Period, Option Period 1, Option Period 2—All Firm Fixed Price)

1.1 Kickoff Meeting

The contractor’s project director and one other key project staff member shall meet with the Contracting Officer’s Representative (COR) and other appropriate Department staff within two weeks following the award date to discuss details regarding the tasks and subtasks outlined in the proposal, scheduling of activities, and other issues related to the contract. The contractor shall bring a proposed list of panelists to discuss with the COR.

The meeting shall be either in person or via conference call. The contractor shall be prepared to identify any task areas requiring clarification and suggest ways of responding to these concerns. The contractor shall provide an agenda for the meeting electronically to the COR within five business days after the effective date of the contract. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting and how each will be addressed.

The contractor shall present and be prepared to discuss the practice guide work plan and schedule of deliverables at this initial meeting.

1.2 Monthly Conference Calls

The contractor’s project director shall electronically submit monthly abbreviated notes to the COR on the status of each task and subtask under the contract, including details on the status of each review underway. Outstanding issues or actions requiring follow up by either the COR or the contract’s project director will be discussed in monthly conference calls between the contractor’s project director and the COR. The contractor shall update the abbreviated status report based on discussions and submit the report to the COR within one workday following the call. The Base Period will support 23 conference calls for the task order, corresponding with months 2 to 24 after award, while Option Period 1 and Option Period 2 will each support 12 conference calls corresponding with months 25 to 48 after award.

1.3 Monthly Reports

The contractor shall prepare monthly progress reports due the 10th business day of the month. The reports shall summarize the major activities and accomplishments for the reporting period. In addition, the reports shall provide information for each project task regarding significant findings and events, problems encountered, and use of staff time. The reports shall also specify the extent to which the project is on schedule and briefly describe the activities planned for the next month. The contractor shall identify and discuss significant deviations from the substantive and time factors in the management plan and identify and discuss any decisions which may be needed from the Department. The report shall also include a copy of the deliverables schedule with columns for anticipated submission dates, actual submission dates, and actual acceptance dates for all deliverables in the task order. The Base Period will support progress reports for the first 24 months of the task order, while Option Period 1 and Option Period 2 will each support 12 progress reports corresponding with months 25 to 48 after award.

1.4 Periodic Meetings

The contractor’s project director and one other key staff member shall meet with the COR and other appropriate Department staff on an as-needed basis, in person or by a video conference call, not to exceed four times during the Base Period, two times during Option Period 1, and two times during Option Period 2. The purpose of the meetings is to brief Department on the progress of developing the practice guide(s) and related products and to discuss issues as they arise. These meetings may require the contractor to prepare agendas, meeting materials, and summaries.

The contractor’s project director and one other key staff member shall also join select cross-contractor meetings with Department staff and representatives of the other WWC contracts once per year. For budgeting purposes, the contractor may assume that these meetings will be virtual and will occur in two 3-hour afternoon sessions.

The contractor is also expected to have a representative participate in the WWC’s Statistical and Technical Analysis (STAT) team and associated small working groups. This participation involves periodic 60- to 90-minute working group meetings, up to 2 per month, to discuss methodological issues. These meetings are virtual.

Task 2: Form the Panel (Base Period—Firm Fixed Price)

The contractor shall submit a recommendation memo that confirms the participation of the panel chair and panelists approved by the COR within four weeks after the award date. If the first choice for the panel chair or a panelist is unable or unwilling to serve in that the role, the contractor shall propose, no later than four weeks after the award date, a qualified replacement for approval by the COR. If necessary, replacements for panelists may be nominated by the contractor for approval by the COR throughout the period of performance.

Panel Chair. The panel chair is a national expert in the topic that is the focus of the task order and is responsible for leading panel members in developing the topic area review protocol and any practice guides that will be informed by the evidence review. In particular, the panel chair leads panel members in developing research-based recommendations, articulating how to implement the recommendations, and addressing potential roadblocks for implementation. To accomplish these tasks, the panel chair shall have a working knowledge of the WWC evidence standards and the WWC process for reviewing studies and support the use of evidence to develop recommendations. The panel chair should also be adept at serving as a meeting moderator and discussant and working with groups with differing opinions.

Panelists. The practice guide panel shall be composed of seven content and methodological experts in addition to the panel chair, of whom at least four shall be researchers, at least two shall be current practitioners, and at least two shall be involved in the pre-service or in-service development of education personnel. Panelists are researchers, educators, and/or practitioners identified as experts in their field who, along with the panel chair, recommend research-based practices, suggest how to implement those practices, and address potential roadblocks for implementing the recommendations. Panelists are expected to contribute to the topic area protocol development, development of the practice guide recommendations, and reviewing the practice guide. The chair and panelists shall also be interested in participating in dissemination events, including, but not limited to, being included in videos, webinars, and in-person dissemination events.

Conflict of Interest Disclosure. Given the central importance of the WWC, NCEE has established separate guidelines regarding actual or perceived conflicts of interest specific to the WWC and the characterization of evidence in WWC publications. The intention of this process is to protect the WWC-PESTO team from situations in which reports and products reasonably could be questioned, discredited, or dismissed due to apparent or actual conflicts of interest, and to maintain standards for high-quality, unbiased policy research and analysis.

All WWC-PESTO team members contributing to practice guides or intervention reports, including the panel chair, panelists, coordinators, and reviewers are required to complete and sign a form disclosing potential conflicts of interest exist. Conflicts for all tasks shall be disclosed before any work is started, and practice guide team members will again be asked to complete the conflict of interest form before the guide is published.

Task 3: Develop a Topic Area Review Protocol (Base Period—Firm Fixed Price)

The topic area review protocol defines both the scope of the literature search and the subsequent systematic review process. The topic area review protocol will specify which kinds of studies, populations, interventions, and findings are of interest for the topic area review corresponding with the task order. The protocol shall be sufficiently broad to cover the field of school-based behavior interventions for students in kindergarten through Grade 12, with the intent that the protocol can be the basis for two practice guides and related intervention reports across the base and option period of the task order. The contractor should assume that individual studies within the scope of the topic area review protocol will be reviewed using a standard study review protocol used for all WWC study reviews, regardless of topic area, under version 4.1 standards and available at https://ies.ed.gov/ncee/wwc/Protocols. Relevant studies the WWC previously reviewed under the same study review protocol should not ordinarily need an additional review.

The contractor’s development of the topic area review protocol should be informed, where appropriate, by the review protocols used in previous WWC practice guides and intervention reports. The contractor shall work with academic librarians and researchers to identify a search strategy to exhaustively identify eligible literature in an efficient non-duplicative manner. The COR will also provide information on research grants from the Department supporting potentially relevant studies that could be eligible for review. The contractor shall include in the topic area review protocol the motivation and goals of the systematic review, the screening criteria for including studies in the scope of review, the relevant outcomes for inclusion in the synthesis of evidence, databases to be consulted for the review, search terms for reviews of the literature of publicly available research, and criteria for prioritizing particular studies for review more than others.

Eligible studies will include manuscripts already in, or eligible to be added to, the Education Resources Information Center (ERIC, https://eric.ed.gov/), although the contractor shall also plan to search other relevant public databases to identify relevant research eligible to review. The evidence review will include education research released during the preceding 15 years and focused on students in the United States or other nations in which English is a primary language. Among eligible studies, some of which may have been reviewed previously by the WWC, those with the highest priority for review will document the practices or core components included in the intervention, as opposed to studies of “black box” interventions.

In collaboration with the panel and the COR, the contractor shall develop a draft topic area review protocol within ten weeks after award. The contractor shall make revisions and submit a final review protocol within 2 weeks of receiving IES feedback from the COR.

Task 4: Review the Literature and Synthesize Findings (Base Period and Option Period 1—Cost Reimbursement)

The contractor shall work with academic librarians and the panel to conduct a thorough review of the literature using the procedures specified in the topic area review protocol. The contractor shall screen studies to ensure their eligibility for review and shall prioritize studies for review in a manner consistent with the protocol.

The contractor shall have reviewers certified in version 4.1 group design standards and version 4.1 single-case design standards in time to begin new reviews of studies. Through the SWAT contract, IES supports the provision of online training to WWC reviewers as well as an online SRG for conducting reviews of individual studies according to the WWC Handbooks. Online reviewer training and certification will be available throughout calendar year 2021, including training in single-case design standards as well as group design standards and, later in 2021, regression discontinuity design standards. To have access to the online SRG, reviewers need to (1) have reviewer certification in WWC group design standards, and (2) complete the security screening requirements described in the Appendix to this PWS. Reviews of single-case design studies and regression discontinuity design studies occur using Excel-based SRG files rather than the online SRG. The contractor shall enter findings from any Excel-based reviews into the online SRG as needed to display the results of those reviews on the WWC website. Costs of training reviewers and obtaining online SRG access are allowable under this task, as are costs of responding to any audits of reviewed studies.

For budgeting purposes, the contractor may assume that, under this task, they will review up to 180 studies that are eligible and in need of review, of which 120 will be group design studies and 60 SCDs. The contractor shall assume that one-third of the studies eligible for review for each practice guide will have been reviewed previously under another WWC protocol and standards version, and that the contractor can import the information from the previous review to use for reviews under the version 4.1 study review protocol. The contractor shall assume 120 study reviews will be prioritized under the Base Period to inform preparation of the initial practice guide, and an additional 60 study reviews will be completed under Option Period 1 to inform a second practice guide. If the contractor expects to need to review more studies, they shall inform the Department as soon as possible. Note that Task 9 is an option to support additional study reviews.

After the master SRGs are complete, the contractor shall submit the SRGs to the COR for peer review supported by the SWAT contractor. The contractor is expected to address all concerns raised in the review and prepare the study reviews to be published on the WWC website with support from the SWAT contractor and approval by the COR. After the peer review is complete, the contractor shall make the master SRGs and summary of studies that meet standards available to the panel members as part of meeting materials.

When summarizing relevant studies that meet WWC standards—including studies reviewed previously by the WWC and not in need of re-review under this task--the contractor shall work with content experts to describe the components of the intervention and the comparison condition as reported by the study authors. Previous practice guides have had between 20 to 50 studies that met WWC standards with or without reservations.

After completing the coding of the components of the interventions in the research meeting WWC standards, the contractor shall conduct a meta-analytic synthesis of effects by outcome domain across all relevant studies sharing the same component, as described in the version 4.1 Procedures Handbook. The contractor shall submit a memo to the COR that describes each study reviewed that meets WWC standards, the components of the intervention in each study, and the meta-analytic findings by recommendation and outcome domain. Within 2 weeks of receiving COR comments on that memo, the contractor shall revise the meta-analytic synthesis memo to share with the panel to inform their development of practice recommendations.

Task 5: Convene the Panel (Base Period and Option Period 1—Firm Fixed Price)

The main work of the panel occurs through a series of formal panel meetings. The contractor shall plan for panel meetings that are each 6-8 hours in duration. If these meetings are held virtually, they may be scheduled on multiple days. Four panel meetings shall occur under the Base Period of the contract and three panel meetings shall occur under Option Period 1. The first panel meeting shall be to introduce the panel and to help define the scope of the topic area review protocol. The second panel meeting shall include a review of the literature screened for WWC review that has been found to meet WWC standards and Task 4’s meta-analytic synthesis of findings by outcome and intervention component that may be the basis for practice recommendations. The contractor’s preliminary syntheses of evidence under Task 4, which the contractor shall refine for inclusion in the practice guide, shall help the panel to identify components of interventions in the underlying studies that could be the basis for practice recommendations for the practice guide. The third panel meeting shall be to finalize the practice recommendations, identify associated implementation steps, consider potential obstacles and roadblocks, and develop an outline of the text for the guide. The fourth panel meeting shall be to refine the text of the guide in response to IES feedback. The fifth, sixth, and seventh panel meetings shall repeat the focus of the second, third, and fourth panel meetings, respectively, but with the inclusion of additional studies not reviewed for the first practice guide. For budgeting purposes, the contractor may assume that all of the panel meetings occur virtually rather than in a single location. The contractor shall submit an agenda and all meeting materials to the COR for comment three weeks prior to each panel meeting, shall revise those materials within one week of receiving COR comments, shall provide an accurate summary of decisions and action items from the meeting within two weeks of the conclusion of each meeting.

Task 6: Develop the Guide (Base Period and Option Period 1—Firm Fixed Price)

Four weeks after the second (and fifth) panel meetings, the contractor shall submit draft recommendations and supporting evidence to the COR, based on the contractor’s refinement to the preliminary meta-analytic synthesis completed under Task 4. The COR and IES leadership will provide comments and the contractor shall discuss and reconcile the IES feedback with the panel at the third (and sixth) panel meeting.

Seven weeks before the fourth (and seventh) panel meeting, the contractor shall submit a first draft of the practice guide. Unless specified otherwise by the COR, the contractor shall include the following components in the guide:

· Introduction

The introduction begins with an Overview of the rationale for the guide, the education challenge being addressed, and expectations for how readers will use the recommendations to address the challenge. The introduction then discusses the Scope of the Practice Guide, including the target population for the practices, student outcomes considered in reviewing the evidence, the range of evidence reviewed or the guide, and any considerations readers should keep in mind when interpreting recommendations. The Summary of Recommendations section clearly presents the text of the practice recommendations and the corresponding level of evidence supporting each practice recommendation. The introductory section on Using Evidence to Develop the Recommendations describes the process for assigning evidence levels to the recommendations. The introduction concludes with a discussion of Who Might Find This Guide Useful and a summary of How to Use This Practice Guide including the recommendations, implementation steps, and discussion of potential obstacles and roadblocks.

· Recommendations Each recommendation is a distinct practice that the panel suggests educators implement to address an education challenge. For most guides, the panel crafts three to six recommendations to address what steps should be taken to address the challenge or to improve current practices. Consistent with current WWC procedures and related guidance issued by IES, the WWC assigns each recommendation a level of evidence rating (strong evidence, moderate evidence or minimal evidence) that indicates the validity of the evidence base that supports the recommendations.

· How to Carry Out Recommendations This section differentiates a practice guide from other research-based documents because it specifies strategies for implementing the recommendation based on the best available evidence combined with expert panel opinion. Each recommendation typically has between three and five discrete activities that educators could do to implement the recommendation. This section is written in straightforward and nontechnical language, and it incorporates examples, figures, and vignettes as appropriate. The panel also may suggest additional resources for consideration when implementing recommendations.

· Roadblocks and Solutions Each recommendation ends with a collection of roadblocks and solutions that acknowledge potential barriers to implementation and suggest approaches for proactively addressing these barriers.

· Practice Guide Appendices A and B Appendix A is a postscript from IES describing practice guides, including the criteria used to distinguish the level of evidence supporting each practice recommendation as strong evidence, moderate evidence, or minimal evidence. Appendix B describes methods and processes for developing the practice guide.

· Practice Guide Appendix C: Rationale for Evidence Rating This appendix begins with a table describing the studies relevant to each of the practice recommendations in the guide, including links to the corresponding study pages on the WWC website. The appendix also describes the relevant outcomes in the studies reviewed for the practice guide. The appendix then provides details about the research studies used to develop and support each practice recommendation, beginning with the size and statistical significance of each meta-analytic effect size estimate pooling findings on the same outcome across all studies relevant to each recommendation. A table breaking down the evidence by study reports, for each study contributing to a recommendation, the study design, sample characteristics and settings, intervention condition, comparison condition, outcome domain, WWC-calculated effect size, and statistical significance of the estimate.

· Practice Guide Appendices D and E Appendix D describes each of the panelists, and Appendix E notes any financial or other conflicts of interest that panelists or contractor staff may have with the content of the guide and how these were addressed and mitigated to ensure the impartiality and objectivity of the information included in the guide.

Each practice guide will consist of approximately 60 pages of practice recommendations and related action steps and roadblocks, together with a glossary of relevant terms, followed by up to 100 pages of appendices, references, and endnotes. The contractor shall also prepare a summary of the practice guide and its recommendations, not to exceed 15 pages in length, written specifically for a nontechnical audience.

The COR will review the first draft of the guide within 2 weeks, and the contractor shall make any necessary edits within 2 weeks before sharing a second draft of the guide with the COR and the panel 3 weeks prior to the panel meeting. The COR will then share this draft with IES colleagues and provide written feedback to the contractor 1 week prior to the panel meeting, which the contractor shall share with the panel.

The contractor shall submit a third draft of the practice guide to the COR within 4 weeks of the fourth (and seventh) panel meeting. This shall be a polished draft that has been thoroughly edited and checked for accuracy.

The COR will review the third draft of the practice guide and provide feedback from the NCEE leadership to ensure the acceptability of this draft for external peer review. The contractor shall be prepared to make all revisions necessary to the practice guide in order to address the comments from the COR and from the NCEE leadership on the third draft and permit the guide to be submitted for peer review.

The COR will submit the acceptable revised practice guide to the IES Standards and Review Office (SRO) for an independent peer review. After the first round of review, the contractor shall prepare a memo addressing satisfactorily each of the comments raised by the peer reviewers or the IES action editor, as well as tracked and clean versions of the practice guide reviews to address these comments. If these responses are acceptable to the COR, the COR shall return the responses and the revised guide for a second round of SRO peer review. This iterative process occurs until the practice guide is approved by IES for production.

For budgeting purposes, from the time when the COR submits the guide to SRO, the contractor shall plan for up to 4 rounds of revisions to the draft practice guide over a period of 4 months in order for the guide to be accepted by SRO as an IES publication.

Task 7: Production (Base Period and Option Period 1—Firm Fixed Price)

Each practice guide will be posted on the WWC website with support from the SWAT contractor. Within 3 weeks of being accepted by SRO, the contractor shall prepare a properly formatted, error-free PDF version of the practice guide to be shared with the office of the Secretary of Education. This PDF shall conform to a practice guide template provided by the COR and shall include hyperlinks to the study protocol and reviews of studies for the guide on the WWC website. No more than 2 weeks later, the contractor shall prepare a 508-compliant PDF version of each practice guide to the COR. In addition, the contractor shall develop a summary paragraph and image to accompany the guide, create a rotating banner image to advertise the release of the guide on the WWC website home page, write text for the “What’s New” announcement, and create the text an electronic newsflash announcing the new practice guide.

Task 8: Summaries (Option Period 1 and Option Period 2—Firm Fixed Price)

The contractor shall develop a summary, no more than 15 pages in length, of each practice guide. This summary shall be written for nontechnical audiences and shall be drawn from the text and images included in the approved practice guide.

The contractor shall submit a draft summary to the COR within 2 weeks of SRO approval of a practice guide. The COR will provide feedback and the contractor shall produce a properly formatted PDF version within 2 weeks of receiving COR feedback and acceptance of the practice guide by SRO. Prior to the scheduled release of the practice guide, the contractor shall deliver 508-compliant PDF version of the summary to the COR, so the summary can be released at the same time as the practice guide.

Task 9: Conduct Additional Study Reviews (Option Period 1 and Option Period 2—Cost Reimbursement)

The contractor may need to review more studies than anticipated in the assumptions of Task 4 or to contribute to intervention reports prepared under Task 10, below, or to other products relevant to the task order. All reviews shall occur using the same topic area review protocol and study review protocol used to review the initial batch of studies for the first practice guide. For budgeting purposes, the contractor may assume the review of 60 additional group design studies and 30 additional single-case designs studies, with two-thirds of them reviewed under Option Period 1 and one-third under Option Period 2.

Task 10: Intervention Reports (Option Period 1 and Option Period 2—Firm Fixed Price)

Using the reviews conducted under Task 4 and/or Task 9 within the scope of the topic area synthesis protocol, the contractor shall develop 6 intervention reports—3 per option period--that synthesize evidence on branded interventions or programs. The contractor may update existing WWC intervention reports or develop new intervention reports using the approved report templates provided by the COR. Working with the COR and in accordance with the topic area review protocol, the contractor shall prioritize the interventions for reporting based on the number of eligible studies that meet with or without reservations, that describe interventions in wide use, and that include information from study authors on the core components of the corresponding intervention.

Intervention reports are a summary of all studies published during a specific time period that examine the effectiveness of an intervention. Using its objective and transparent standards, the WWC rates the quality of the evidence in each study. For studies that meet WWC standards, the WWC combines the findings to generate overall estimates of the size of effects for the intervention. The WWC also provides ratings of the extent of evidence and of the intervention’s effectiveness, taking into consideration the number of studies, their sample sizes, the WWC’s ratings of the quality of the studies, and the statistical significance of the meta-analytic estimates of effects.

The contractor shall be responsible for developing an intervention report of no more than 15 pages and a 1-page evidence snapshot. The contractor shall expect at least 1 round of peer review involving the SWAT contractor, 1 round of review by the COR and WWC team lead, and 1 round of review by the IES leadership. The contractor will be responsible for making any revisions necessary in the judgment of the COR to address issues identified during these reviews.

Within 3 weeks of being accepted by the COR, the contractor shall prepare a properly formatted, error-free PDF version of each intervention report and snapshot to be shared with the office of the Secretary of Education. This PDF shall conform to the intervention report template provided by the COR and shall include hyperlinks to the study protocol and reviews of studies for the report on the WWC website. No more than 2 weeks later, the contractor shall prepare a 508-compliant PDF version of each report document to the COR. In addition, the contractor shall develop a summary paragraph, create a rotating banner image to advertise the release of the report(s) on the WWC website home page, write text for the “What’s New” announcement, and create the text an electronic newsflash announcing each new intervention report or set of intervention reports.

Task 11: Webinars, Videos, or Dissemination Events (Option Period 1 and Option Period 2—Firm Fixed Price)

The contractor shall support the Department’s dissemination activities of the practice guides and related products through three distinct activities during each option period. For budgeting purposes, the contractor may assume that one dissemination event will be in person at Department headquarters, one in person at another location such as a professional conference, and one a virtual event, such as a webinar or a series of brief videos. The contractor should plan on the panel chair and a practitioner from the panel joining these events. For each event, the contractor shall prepare draft and final presentation slides for discussing a practice guide and related products with the audience. The contractor shall add subtitles to any recorded webinar or videos and make appropriate metadata available within one month of the release of the practice guide, so the webinar or videos may be posted by the SWAT contractor to the WWC website.

VIII. SCHEDULE OF DELIVERABLES

In their proposal, the contractor shall develop a proposed deliverables schedule that includes the deliverables mentioned above. Additional interim deliverables may be added, provided they have a tangible value to the Government.

Appendix: Compliance with ED IT Security Policy The contractor shall:

* Complete/update the appropriate level of Security Accreditation (SA) documentation per NIST Risk Management Framework guidance, security controls testing, interagency security agreements (ISAs), and risk assessments in support of government issuance of security assessment and authorization to operate (ATO) decisions

* Ensure that systems/products/applications have the ability to facilitate single-sign-on capabilities and required support for HSPD 12 Personal Identity Verification (PIV) enablement and integration

* Include the capability for network traffic that flows between externally hosted systems and networks, to/from Department systems and networks, to be routed through one of the Department’s Trusted Internet Connections (TIC) gateways as part of the solution configuration. Implement controls to ensuring all possible traffic, including mobile and cloud, goes through a TIC. Implement connections between Department systems and networks with externally hosted systems that are in compliance with the requirements of the Trusted Internet Connections (TIC) initiative

* Architect contractor hosting environments to use security isolation and network segmentation principles in order to ensure that the environments are properly protected against an unauthorized access and threat from adversaries who may strive to move laterally across internal Department or contractor hosted systems and network segments.

* Provide an automated capability and process to scan and assess all systems and assets, and associated logs for malicious indicators of compromise (IOCs) identified by the Department regarding priority threat-actor Techniques, Tactics, and Procedures (TTPs); the contractor is required to have the capability to scan for indicators of compromise within 24 hours of receipt of the indicators provided by the Department of Education from the Department of Homeland Security

* Implement and maintain capabilities and processes to support the timely detection of, reporting, and rapid response and recovery to cyber incidents in accordance with timelines and requirements specified in Federal guidance and Department cybersecurity incident reporting policy guidance

* In support of cybersecurity performance measure reporting, the contractor shall implement and maintain an automated software asset management/inventory and hardware asset inventory capability (e.g. scans/device discovery processes) at the enterprise-level.

* Implement capabilities to rapidly deploy emergency security patches and implement specific security control enhancements as directed by the Department of Homeland Security to all Federal Departments via mechanisms such as the DHS Cybersecurity Coordination, Assessment, and Response (C-CAR) action items, and DHS Binding Operational Directives (BODs).

* Implement capabilities and processes to patch all critical vulnerabilities identified to the Department of Education by DHS immediately or, at a minimum, within 30 days of patch release.

* Ensure robust physical and cybersecurity protections are in place for all of the Department’s high value assets (HVAs). The identification of HVAs by the Department will be an ongoing activity due to the dynamic nature of cybersecurity risks.

* Implement remote access solutions that only use multi-factor authentication solutions and that prohibit the use of split tunneling and/or dual-connected remote hosts where the connecting device has two active connections.

* Implement remote access solutions that scan for malware before allowing full connections and that time out after 30 minutes (or less) of inactivity and require re-authentication to re-establish a session

* Implement capabilities for all incoming email traffic to pass through anti-phishing and anti-spam filtration at the outermost border mail agent or server

* Implement capabilities for all incoming email traffic to be analyzed using sender authentication protocols (e.g., DKIM, DMARC, VBR, SPF, iprev)

* Implement capabilities that ensure that incoming email traffic is analyzed using a reputation filter (to perform threat assessment of sender)

* Implement capabilities that ensure that incoming email traffic is analyzed for detection of clickable URLs, embedded content, and attachments; and incoming email traffic is first analyzed for suspicious or potentially nefarious attachments and opened in a sandboxed environment or detonation chamber

* Implement capabilities for all outbound communications traffic to be checked at the external boundaries to detect encrypted exfiltration of information…