WWC_GRAPE_-_PWS_06.14.18.pdf

ATTACHMENT A

U.S. Department of Education Institute of Education Sciences

National Center for Education Evaluation and Regional Assistance

DRAFT PERFORMANCE WORK STATEMENT (46/1814/2018)

What Works ClearinghouseTM Guides to Research that Assists Practitioners in Education (“WWC-GRAPE”)

I. Introduction

The United States Department of Education (the “Department”), Institute of Education Sciences (IES), National Center for Education Evaluation and Regional Assistance (NCEE) requires a contractors to convene a panels of education practitioners and national experts in research, education policy, and education practice and to develop a series of What Works ClearinghouseTM (WWC) practice guides on various topics. In at least some instances, these practice guides will update earlier WWC practice guides (https://ies.ed.gov/ncee/wwc/PracticeGuideshttps://ies.ed.gov/ncee/wwc/Publication# /ContentTypeId:3).

The This Blanket Purchase Agreement (BPA) shall have an ordering period of 60 months.

The period of performance for Task Order 0001 will be 24 months and will focus on preparing an updated WWC practice guide on Assisting Students Struggling with Mathematics: Response to Intervention (RtI) for Elementary and Middle Schools (https://ies.ed.gov/ncee/wwc/PracticeGuide/2). Up to six aAdditional 24-month task orders may be authorized by the Department to produce additional practice guides, ;

these task orders shall all of which shall be completedbegin within the 60 month ordering period of the BPA. The Department does not expect to award more than 7 such task orders fromto any single contractor on the BPA.

A. Authorizing Legislation This contract is authorized by Title I, Part D of the Education Sciences Reform Act of 2002, Section 171(b).

(b) MISSION.—The mission of the National Center for Education Evaluation and Regional Assistance shall be—

(1) to provide technical assistance;

(2) to conduct evaluations of Federal education programs administered by the Secretary (and as time and resources allow, other education programs) to determine the impact of such programs (especially on student academic achievement in the core academic areas of reading, mathematics, and science);

https://ies.ed.gov/ncee/wwc/PracticeGuide/2

(3) to support synthesis and wide dissemination of results of evaluation, research, and products developed; and

(4) to encourage the use of scientifically valid education research and evaluation throughout the United States.

The need for educators and policy makers to identify “evidence-based” practices to improve student outcomes and other outcomes relevant for education was also emphasized in the 2015 reauthorization of the Elementary and Secondary Education Act (P.L. 114-95). In particular, ESEA was amended to include definitions of “moderate evidence” and “strong evidence” (as defined in 20 U.S.C. 7801, section 21), corresponding with statistically significant and favorable findings from “well-designed and well-implemented” quasi-experimental and experimental studies.

B. WWC Background

The WWC was established in 2002 by the Department to provide educators, policymakers, researchers, and the public with a transparent way to determine whether there is evidence of effectiveness behind a program or practice and the extent to which those findings can be trusted. The goal was to promote better informed education decision-making through a web-based dissemination system that features comprehensive, systematic, high-quality reviews1 of studies on the effectiveness of educational interventions (i.e., programs, products, practices, and policies). The website, whatworks.ed.gov is the primary “face” of the WWC, displaying the results of the systematic review process. The WWC is focused on providing timely, comprehensive reviews of interventions, establishing and refining state of the art evidence standards for the field of education, providing data on study reviews to a wide range of stakeholders, and disseminating information to the public.

The WWC investment currently operates through four distinct contracts:

• ED-IES-13-C-0010: What Works Clearinghouse Content (contract with Mathematica Policy Research, ending February 27, 2018) focused on reviews related to pre-kindergarten through Grade 12 education, as well as WWC standards and procedures, WWC reviewer training and certification, and the WWC Help Desk

• ED-IES-13-C-0013: What Works Clearinghouse Website (contract with Sanametrix, ending April 28, 2018)

• ED-IES-16-C-0024: Postsecondary Education and Postsecondary Preparation Evidence Reporting (herein referred to as the WWC-PEPPER Contract, with

1 A systematic review is a review of the evidence on a clearly formulated question that uses systematic and explicit methods to identify, select and critically appraise relevant primary research, and to extract and analyze data from the studies that are included in the review.

Abt Associates), focused on reviews related to student access to, and success in, postsecondary education

• ED-IES-15-D-0003: What Works Clearinghouse Reviews, Reporting, Dissemination, & Development Indefinite Delivery, Indefinite Quantity Contract (with Development Services Group), focused on conducting timely reviews of individual studies submitted as part of the Department’s evidence-based grant competition, as well as reviewing other studies at the request of IES and helping IES create new WWC products

Additionally, two additional contracts will be added to the WWC portfolio in the near future:

• 91990018C0019 (with the American Institutes for Research)Contract Number To Be Determined: What Works Clearinghouse Statistics, Website, and Training (herein referred to as the SWAT Contract), focused on the WWC website, WWC Help Desk, WWC standards and procedures, and WWC reviewer training and certification

• 91990018C0009 (with Mathematica Policy Research)Contract Number To Be Determined: What Works Clearinghouse Organization of Reporting of Evidence on Graduation, Achievement, and Nonacademic Outcomes (herein referred to as WWC-OREGANO Contract), focused on reviews on education interventions to pre-kindergartenfrom early childhood through Grade 12 education

The SWAT contract is expected to supports the provision of online training to WWC reviewers as well as an online Study Review Guide (SRG) for entering information on the reviews of individual studies using the WWC Standards and Procedures Handbooks (https://ies.ed.gov/ncee/wwc/Handbooks), ). To have access to the online SRG, reviewers working on the WWC-GRAPE contract will need to (1) have reviewer certification in the most recent WWC standards for group designs, and (2) complete the security screening requirements described in the Appendix to this PWS.

C. WWC Guides to Research that Assists Practitioners in Education (WWC-GRAPE)

Through the WWC-GRAPE contract, IES intends to support the preparation of new or updated practice guides to provide educators in pre-kindergartenearly childhood through Grade 12postsecondary education and other stakeholders with research-based, actionable recommendations for improving student outcomes and other outcomes relevant to education. WWC practice guides (https://ies.ed.gov/ncee/wwc/PracticeGuideshttp://ies.ed.gov/ncee/wwc/publications/ practiceguides/) provide a coherent set of recommendations on practices with respect to a particular issue, problem, or challenge that schools educators face, connect that advice to the supporting evidence, and describe the quality of that evidence. A panel of experts develops an IES practice guide to provide practical, evidence-based recommendations for educators (such as teachers and or administrators or parents) on https://ies.ed.gov/ncee/wwc/Handbooks how to address multifaceted challenges that lack developed or evaluated packaged approaches (commercial or non-commercial). Authors of practice guides—who generally include key contractor personnel distinct from practice guide panel members— combine their expertise with the findings of rigorous research, when available, to develop specific recommendations for addressing these challenges. The authors rate the strength of the research evidence supporting each of their recommendations. In addition, practice guides provide information on how to carry out each recommendation and address possible implementation roadblocks.

Practice guides incorporate the best available research evidence on the topic, supplementing WWC’s systematic review of scientifically-based research with other levels and types of evidence. The other evidence may include that obtained from experimental and quasi-experimental studies that do not meet WWC standards, correlational studies with statistical controls, comparative studies with matching, simple correlational studies, case studies, pre-post studies, beat-the-odds studies without matching, and the panel’s expertise supported by conceptual models and generalizations from related topic areas but without direct supporting evidence. All evidence used to inform the “how-to” guidance in a practice guide is organized in a hierarchy defined by its quality and relevance.

IES-sponsored practice guides are subject to rigorous external peer review through the same office that is responsible for independent review of other IES publications. This ensures that the recommendations in the guide are based on the best available evidence and adequately address the topic of the guide. Peer reviewers assess whether the evidence cited in support of particular recommendations is up-to-date and that studies of similar or better quality that point in a different direction have not been ignored. They also are asked to evaluate whether the “evidence grades” assigned to particular recommendations by the practice guide authors are consistent with WWC standards. The external peer review is carried out independent of the IES staff that instigated the practice guide. In order to ensure the best quality publication possible, WWC practice guides are revised as necessary to meet the concerns of external peer reviews and gain the approval of the standards and review staff at IES.

The contractor shall include the following components in this IES practice guide (although the arrangement of the components may be revised at the direction of IES or with IES approval):

• Review of Recommendations

The practice guide opens with a one-page overview of the recommendations and action steps on how to carry them out.

• IES Levels of Evidence for Practice Guides

This section describes the criteria for strong evidence, moderate evidence, and minimal evidence recommendations, as well as the process the panel undertakes for determining each recommendation’s level of evidence.

• Introduction

The introduction begins with an Overview of the rationale for the guide, the education challenge being addressed, and expectations for how readers will use the recommendations to address the challenge. The introduction then discusses the Scope of the Practice Guide, including the target population for the practices, student outcomes considered in reviewing the evidence, the range of evidence reviewed and topics covered by studies submitted for review, and any considerations readers should keep in mind when interpreting recommendations.

The Summary of Recommendations section clearly describes how the panel conceptualized the recommendations, including information about whether recommendations should be implemented alone or should build upon one another, a discussion of challenges or suggestions readers should consider in implementing recommendations, and a discussion of challenges encountered by the panel in developing recommendations and how the panel addressed these challenges. The Use of Research component of the introduction (which may be moved to an appendix to shorten the body of the guide) describes the body of research that contributed to the practice guide and levels of evidence for each recommendation.

It also includes a summary of the research that contributed to the development of the practice guide, the strengths and limitations of the practice guide based on the research reviewed for the guide, and suggestions for further research or information needed about the practice guide topic. The introduction concludes with Recommendations and Corresponding Levels of Evidence to Support Each Recommendation, which are a compact version of each recommendation and the level of evidence to support the recommendation.

• Recommendations

Each recommendation is a distinct practice that the panel suggests schools or districtseducators implement to address an education challenge. In most cases, the panel crafts recommendations to address what steps should be taken to address the challenge or to improve current practices. The panel assigns each recommendation a level of evidence rating (strong evidence, moderate evidence or minimal evidence) that indicates the internal and external validity of the evidence base that supports the recommendations.

• Summary of Evidence

For each recommendation, all studies of the practice are examined to form the evidence base that the panel uses to determine the level of evidence for the recommendation: strong evidence, moderate evidence, or minimal evidence. This section (which may be moved to an appendix to shorten the body of the guide) justifies the recommendation’s level of evidence by describing the studies the panel used as the basis for the recommendation. However, this section clearly links the practices described in the recommendation with their effects on the outcomes of interest. The section also describes the generalizability of the findings from the studies for larger audiences or the limitations of the studies. Moderate evidence and minimal ratings often incorporate brief discussions of evidence beyond studies rated by the WWC.

• How to Carry Out Recommendations

This section differentiates a practice guide from other research-based documents because it specifies strategies for implementing the recommendation based on the best available evidence combined with expert panel opinion. Each recommendation typically has between three and five discrete activities that teachers, schools, or districtseducators could do to implement the recommendation. This section is written in straightforward and nontechnical language, and it incorporates examples, figures, and vignettes as appropriate. The panel also may suggest additional resources for consideration when implementing recommendations.

• Roadblocks and Solutions

Each recommendation ends with a collection of roadblocks and solutions that acknowledge potential barriers to implementation and suggest approaches for proactively addressing these barriers.

• Practice Guide Appendices A, B, and C

Appendix A is a postscript from IES describing practice guides. Appendices B and C, respectively, introduce the panelists and note any financial or other conflicts of interest that panelists or contractor staff may have with the content of the guide and how these were addressed and mitigated to ensure the impartiality and objectivity of the information included in the guide.

• Practice Guide Appendix D: Rationale for Evidence Rating

The evidence base for each recommendation is described in detail in this appendix.

Each recommendation in the appendix begins with a description of the panel’s justification for choosing a particular level of evidence, such as the types of studies that constitute the evidence base and any caveats or special considerations. After building a case for a particular level of evidence, the panel provides details about the research studies used to develop and support the recommendation. In the case of a strong evidence or moderate evidence, details on the samples, the intervention and comparison groups, the magnitude of the effects, and the limitations of the studies may be provided. Research that demonstrates a relationship between the practice and outcomes—above and beyond what is already demonstrated by experimental (randomized control trial (RCT), quasi-experimental design (QED), single-case design (SCD), or regression discontinuity design (RDD) research—also may be summarized in the appendix. Study details typically are grouped by topic;

e.g., all the research that relates to a particular action step is summarized together).

An overview of the studies and key details often are included in a summary table for each recommendation. In the case of minimal evidence rating, research that attempts to link the practices to outcomes or systematic analyses of the implementation of practices are described. In all cases, extrapolations made by the panel and limitations to the evidence base are explicitly acknowledged in the appendix.

• Practice Guide Appendix E: Evidence Heuristic

This appendix contains the heuristic currently used for categorizing the evidence base for practice guide recommendations. The heuristic may be updated during the period of the performance of the contract if the WWC Handbooks are updated. The intent is for the heuristic to serve as a framework to ensure that the levels of evidence are consistently applied across practice guides while at the same time clarifying the levels for panelists and educators. Following this heuristic, evidence is characterized as strong evidence, moderate evidence, or minimal evidence.

• Practice Guide Appendix F: Supplemental materials

Supplemental materials may include worksheets or additional activities to supplement the recommendations and action steps for carrying out the recommendations.

IES seeks to have a panel of experts develop new or updated practice guides to characterize the evidence base that supports practice recommendations in certain topic areas of interest to educators. In the case of updated practice guides, the panel will examine whether the focus of the guide needs to be expanded to include what has been learned from research completed since the earlier guide was completed. For example, based on an examination of the research, the panel may determine that new recommendations are warranted. Similarly, the panel may determine that findings from new research would result in a change in the level of evidence for an existing recommendation.

The practice guide will consist of a one-page and four-page summary summaries of recommendations accompanied by a document of approximately 50 pages (e.g., approximately 15 pages written for practitioners, and approximately 35 pages with supporting documentation for researchers) that provides narrative support in the form of rationales, suggested sequences of activities, connections between recommendations, procedural details on how to implement the recommendations, and descriptions of supporting evidence, including its limitations and complications. An updated practice guide may include recommendations that were not in the earlier practice guide, different evidence ratings based on new research and additional grade levels if warranted by the evidence. An updated guide shall include references to the earlier practice guide when appropriate.

II. Scope of Work

This section specifies the tasks and subtasks that the contractor shall perform for this 24-month task order authorizing the preparation of an updated or new WWC practice guide. All tasks and subtasks of each task order are expected to be Firm Fixed Price (hereafter referred to as FFP). As noted above, the first practice guide to be updated under this contract BPA will be Assisting Students Struggling with Mathematics:

Response to Intervention (RtI) for Elementary and Middle Schools (https://ies.ed.gov/ncee/wwc/PracticeGuide/2). The topics of additional practice guides prepared through task orders under this contract BPA may include updates to the guides on Reducing Behavior Problems in the Elementary School Classroom (https://ies.ed.gov/ncee/wwc/PracticeGuide/4) and Assisting Students Struggling with Reading: Response to Intervention (RtI) and Multi-Tier Intervention in the Primary Grades (https://ies.ed.gov/ncee/wwc/PracticeGuide/3) and Reducing Behavior Problems in the Elementary School Classroom (https://ies.ed.gov/ncee/wwc/PracticeGuide/4).

Other practice guides authorized under this BPA may focus on topics in early childhood education, elementary education, secondary education, or postsecondary education, including topics not previously covered by a WWC practice guide.

Task 1: Meetings with the Department (Firm-Fixed-Price)

1.1 Kickoff Meeting

The contractor’s project director and one other key project staff member shall meet with the Contracting Officer’s Representative (“COR”) and other appropriate Department staff in Washington, DC, within two weeks following the award date to discuss details regarding the tasks and subtasks outlined in the proposal, scheduling of activities, and other issues related to the contract. The meeting will be at the Department’s offices in Washington, D.C. The contractor shall be prepared to identify any task areas requiring clarification and suggest ways of responding to these concerns.

The contractor shall provide an agenda for the meeting electronically to the COR within five business days after the effective date of the contract. Within one week following the meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting and how each will be addressed. The COR will respond to this memo with comments, as necessary, within two weeks of receiving it.

https://ies.ed.gov/ncee/wwc/PracticeGuide/2 https://ies.ed.gov/ncee/wwc/PracticeGuide/4 https://ies.ed.gov/ncee/wwc/PracticeGuide/3

The contractor shall present and be prepared to discuss the practice guide work plan (Subtask 2.1) at this initial meeting.

1.2 Monthly Conference Calls

The contractor’s project director shall electronically submit monthly abbreviated notes to the COR on the status of each task and subtask under the contract, including details on the status of each review underway. The contractor shall submit the notes at least 24 hours prior to regularly scheduled monthly telephone updates. Outstanding issues or actions requiring follow up by either the COR or the contract’s project director will be discussed in monthly conference calls between the contractor’s project director and the COR. The contractor shall update the abbreviated status report based on discussions and submit the report to the COR within one workday following the call.

1.3 Periodic Meetings

The contractor’s project director and one other key staff member shall meet with the COR and other appropriate Department staff in Washington, DC on an as-needed basis, in person or by conference call, not to exceed four times during the contract period.

The purpose of the meetings is to brief Department on the progress of developing the practice guide and to discuss issues as they arise. The contractor shall submit draft agenda and meeting materials three work days prior to the meeting. Within one week following each meeting, the contractor shall prepare a memo summarizing the key issues and concerns raised at the meeting, and how each will be addressed.

The contractor’s project director and one other key staff member shall also join select cross-contractor meetings with Department staff and representatives of the other WWC contracts up to four times per year. (For budgeting purposes, the contractor may assume that these meetings will be held via teleconference.) The COR will alert the project director to the specific meetings at which the project director’s attendance is required and will provide an agenda in advance of the meeting.

Deliverables (Task 1): Meeting agenda (subtask 1.1) Meeting memo (subtask 1.1) Status notes (subtask 1.2) Updated status notes (subtask 1.2) Draft agendas and meeting materials (subtask 1.3) Summary memo (subtask 1.3)

Task 2: Planning and Start-Up (Firm-Fixed-Price)

The contractor shall submit a plan to conduct the practice guide preparation process.

The contractor shall prepare a work plan for the entire 24-month contract period, informed by the contractor’s technical proposal and subsequent comments from the

Department. The contractor shall develop a purpose statement, conduct a preliminary scan of available literature, propose a panel chair, and identify potential panelists for the practice guide, all of which shall be submitted to IES for approval.

By the end of this stage, the focus of the practice guide shall be clearly defined so that, once convened, the panel can begin discussions. The contractor shall submit the draft practice guide work plan within two weeks of award, using the work plan in the contractor’s proposal as a starting point. The COR will review and provide comments to the contractor on the draft practice guide work plan within two weeks. The comments on the draft work plan should inform the development of the final practice guide work plan. The final work plan should serve as a reflection document, comparing the draft plan to the how the production of the guide actually developed, with key take-ways and “lessons learned.” The contractor shall make revisions and submit a final plan along with the final draft of the practice guide (see Task 5). The total number of revisions shall not exceed four.

2.1 Practice Guide Work Plan

The contractor shall submit a draft of the practice guide work plan (based on the plan that was originally submitted in the proposal) two weeks after the award date. The contractor shall pay particular attention in the work plan to the following:

• A statement of the educational significance of the new or updated practice guide, including an explanation of the reasoning for changing the title or focus of the practice guide;

• The number and a brief description of methodologically rigorous and substantively relevant studies on the topic;

• Gaps in evidence to support recommendations; and,

• The challenges that arise in updating the practice guide, and proposed strategies that will be used to overcome them.

The contractor shall propose and justify the plans for administering and managing the study reviews and for updating the practice guide. This plan shall include details regarding how the project will be staffed and managed.

For example, the plan might propose to rely primarily on the panel, primarily on core contractor staff to both review the studies and develop the new or updated practice guide; it could rely heavily on consultants and/or subcontractors for substantial tasks or portions thereof, or a combination of these options. Under any of these scenarios, the contractor has primary responsibility for managing and overseeing the work on the project, including identifying relevant consultants, subcontractors, and/or experts, and ensuring the quality of their work.

In the work plan, the contractor shall describe the specific activities that the contractor will carry out in the performance of each task and subtask. At a minimum, the plan shall include detailed descriptions for the following:

(1) the systematic review process;

(2) the practice guide development and review process, including a plan for how studies will be prioritized for review; and

(3) how the contractor will support the work of the panel including the role of the contractor.

The work plan also shall include:

(1) the production schedule with timelines and deliverables;

(2) the plan for recruiting and supporting the panel;

(3) the proposed contractor personnel who will staff the team;

(4) the proposed tracking procedures for deliverables; and

(5) the proposed quality assurance procedures.

The COR will review the draft work plan and return with comments within two weeks.

The contractor shall make revisions and submit a final practice guide work plan within 6 weeks of award.

2.2 Develop a Purpose Statement

If necessary, the contractor shall revise the purpose statement submitted in the proposal that defines the education challenge to be addressed by the practice guide, provides a framework for decisions about the scope of the practice guide, discusses the existing evidence base, and identifies areas of expertise that should be represented on the panel. The practice guide panel and the contractor use the purpose statement to focus the literature search and to develop the literature review protocol. Once the panel is assembled, the purpose statement helps panelists focus their recommendations on a well-defined problem.

The contractor shall submit the draft purpose statement within three weeks from the award date. The COR will review the draft purpose statement within two weeks. The contractor shall make revisions and submit the revised draft purpose statement within 7 weeks.

2.3 Select the Panel

The contractor shall revise the recommendation memo submitted in the proposal of the practice guide work plan recommending a panel chair and panel members within four weeks after the award date.

The COR will provide comments within two weeks. The contractor shall make revisions and submit a final recommendation memo within 8 weeks.

Panel Chair. The panel chair is a national expert in the topic that is the focus of the guide and is responsible for leading practice guide panel members in developing the practice guide. The panel chair assists in identifying and recruiting panel members, and developing the review protocol and literature search parameters. The panel chair leads practice guide panel members in developing research-based recommendations, articulating how to implement the recommendations, and addressing potential roadblocks for implementation. To accomplish these tasks, the panel chair shall have a working knowledge of the WWC evidence standards and the WWC process for reviewing studies, and support the use of evidence to develop recommendations, largely with moderate to strong levels of evidence for improving student outcomes and other relevant education outcomes in the topic area of the practice guide. The panel chair should also be adept at running meetings and keeping the panel focused on developing a consensus-based product within the specified time frame.

Expectations and responsibilities of the panel chair include but are not limited to:

1) assisting in identifying and recruiting panel members;

2) developing the review protocol and literature search parameters;

3) providing citations of studies for review;

4) communicating panelists’ responsibilities;

5) overseeing conference calls and in-person meetings of the panel;

6) assisting in establishing practice guide recommendations;

7) facilitating panel discussions on the extent of evidence for practice guide recommendations;

8) working with key staff on the contractor’s team to address disagreements among panelists regarding the content of the practice guide;

9) providing feedback and approval on all outlines and drafts of the practice guide prepared by the authors on the contractor’s team;

10) providing feedback for responding to peer review comments;

11) providing recommendations for key dissemination sources; and

12) participating in dissemination of the content of the practice guide at conferences and seminars, as specified in the Communications Plan, Task 7.

Within 4 weeks of award, the contractor shall submit a memorandum recommending at least three leading, nationally recognized researchers on the practice guide topic who are potential panel chairs with supporting documentation of their credentials. After COR approval or substitution and ordering of the sequence of the recommendations, the contractor shall contact the candidates and determine availability and interest in serving as panel chair.

Panelists. The practice guide panel is composed of content and methodological experts.

Panelists are researchers and/or practitioners identified as experts in their field who, along with the panel chair, recommend research-based practices, suggest how to implement those practices, and address potential roadblocks for implementing the recommendations. Panel members may include some of the experts who served on an earlier, related practice guide panel, as well as new panel members who can provide an up-to-date perspective on the topic and round out the areas of expertise to bear on the development of the guide.

Expectations and responsibilities of a panel member include, but are not limited to:

1) providing citations of studies for review;

2) actively participating in conference calls and in-person meetings;

3) assisting in establishing practice guide recommendations;

4) commenting on all practice guide outlines and drafts prepared by the authors on the contractor’s team;

5) participating in discussions about levels of evidence for recommendations;

6) providing feedback for responding to peer review comments; and

7) providing suggestions for key dissemination sources.

At least one panel member shall participate in dissemination activities, as specified in the Communication Plan, Task 7.

Within 6 weeks of award, the contractor, working in collaboration with the panel chair, shall recommend up to seven panel members who are leading researchers and/or practitioners in the topic area. The memorandum shall include primary and alternate candidate recommendations, grouped by the type of expertise this panel needs, and with supporting documentation of their credentials. After COR approval or substitution and ordering of the sequence of the recommendations, the contractor shall contact the candidates and determine availability and interest in serving as panelists. Within 8 weeks of award, the contractor shall provide a final memo describing the panel members and summarizing their qualifications. Note that the participation of the panel chair and all panelists is contingent on the completion and submission of a NCEE Conflict of Interest disclosure form to the COR.

Conflict of Interest Disclosure. Given the central importance of the WWC, NCEE has established separate guidelines regarding actual or perceived conflicts of interest specific to the WWC and the characterization of evidence in WWC publications. The intention of this process is to protect the WWC and practice guide team from situations in which reports and products reasonably could be questioned, discredited, or dismissed due to apparent or actual conflicts of interest, and to maintain standards for high-quality, unbiased policy research and analysis.

All WWC practice guide team members, including the panel chair, panelists, coordinators, and reviewers are required to complete and sign a form identifying whether or not potential conflicts of interest exist. Conflicts for all tasks shall be disclosed before any work is started, and practice guide team members will again be asked to complete the conflict of interest form before the guide is published on the WWC website. The contractor shall administer the Conflict of Interest form and the Authorship form to the panel members who have contributed significantly to the actual writing of the guide, and submit completed forms to the COR.

Deliverables (Task 2): Draft practice guide work plan (subtask 2.1) Final practice guide work plan (subtask 2.1) Draft purpose statement (subtask 2.2) Final purpose statement (subtask 2.2) Draft panel chair recommendations memo (subtask 2.3) Draft panel member recommendations memo (subtask 2.3) Final panel member summary memo (subtask 2.3)

Task 3: Identify and Review the Evidence (Firm-Fixed-Price)

Since practice guides provide specific and coherent evidence-based recommendations, researching and assessing available evidence is the cornerstone for developing a practical and actionable practice guide. The contractor shall work with the panel chair to develop a review protocol, building on that used in the development of the existing guide. The review protocol shall include the literature search terms used in the preparation of the earlier guide. The practice guide review protocol defines both the scope of the literature search and the subsequent systematic review process. The contractor shall conduct a literature search and screen the results. The contractor shall prioritize studies and assign them to be reviewed by certified WWC reviewers who are certified in the most recent version of the WWC Procedures and Standardsversion of the WWC Procedures and Standards26 . This process begins after the COR approves the purpose statement and may continue through completion of the first draft of the guide.

3.1 Prepare a Review Protocol

The practice guide review protocol defines both the scope of the literature search and the subsequent systematic review process. The contractor’s development of the review protocol should be informed, where appropriate, by the review protocols used in previous WWC practice guides. The contractor shall include the following key components in the review protocol:

Component Description of Review Protocol Component Screening Criteria

Defines key parameters for:

• Relevant time frame for publication

• Relevant ages/grades of students

• Other defining student characteristics

Study Design Describes different types of studies including those eligible for WWC review such as:

• Randomized controlled trial (RCT) studies

• Quasi-experimental design (QED) studies

• Single case design (SCD) studies

• Regression discontinuity design (RDD) studies

• Also describes studies that can be included in the guide but are not eligible for WWC review such as:

• Correlational studies

• Case studies

Outcomes Defines all domains and relevant outcomes of interest Attrition Standards for RCTs

Identifies thresholds for overall and differential attrition in accordance with WWC standards

Baseline Characteristics for Equivalence

Describes that all QEDs (and RCTs with high attrition) must demonstrate that groups being compared post-attrition are equivalent on relevant characteristics at baseline

Statistical and Analytical Issues

Explains the process for adjusting for pre-intervention differences and multiple comparison adjustments in a way that is consistent with WWC standards

Missing Information

Defines how to proceed with reviewing studies when insufficient information is provided by the author

Component Description of Review Protocol Component Study Ratings Defines study ratings as they relate to the current version of WWC evidence standards

• Meets WWC evidence standards

• Meets WWC evidence standards with reservations

• Does not meet WWC evidence standards

• Uncertain/potentially meets standards pending author query

In collaboration with the panel chair, the contractor shall develop a draft review protocol based on discussions with IES at the kick-off meeting within ten weeks after award of Task Order 0001. The COR will review the draft protocol and return it to the contractor with comments within two weeks. The contractor shall make revisions and submit a final review protocol within 18 weeks. For budgeting purposes, the contractor may assume one additional revision of the draft protocol will occur prior to COR acceptance of the final protocol.

3.2 Training on Conducting WWC Practice Guide Reviews and Assurance of Compliance with WWC Standards

The contractor shall ensure that its reviewers are certified to conduct reviews using the most recent version of WWC standards and procedures, utilizing the online training available from the SWAT contractor. The contractor shall provide, by 1 month following the start of Task Order 0001, a draft memo to the COR summarizing any WWC training needed in order to ensure a sufficient number of reviewers for work on the task order.

By 4 months following the start of the task order, the contractor shall provide a final memo confirming the certification of the individuals who received training.

3.3 Review the Literature

The purpose statement and review protocol provide the parameters to search relevant databases and store all references in a study tracking database. The contractor shall use multiple strategies to assemble relevant studies for review, including searches of electronic databases and incorporation of studies suggested by the panel. During the development of the practice guide, the panel’s discussion of potential recommendations may lead to an expanded literature search, or the panel may expand the scope of the recommendations. The contractor shall conduct additional literature searches, as warranted, to ensure that relevant studies are reviewed. This process ensures that panelists are not constrained to recommending only those practices that are identified through an initial review of literature.

The contractor shall screen the literature according to the review protocol, which includes the following steps:

(1) identify studies that are eligible for WWC review;

(2) identify studies that have non-eligible designs but can be summarized to support the recommendations; and

(3) document the relevance of the study to the practice guide topic, the relevance of the study sample to the population of interest, the timeliness of the study, the relevance and validity of the outcome measure, and other criteria specified in the review protocol.

In screening the literature, the contractor shall apply these four general criteria:

(1) Was the study published in the relevant time range?

(2) Is the study a primary analysis of the effect of a practice?

(3) Does the study have an eligible design?

(4) Does the study fall within the review protocol parameters?

The contractor shall prepare draft and final literature review memos that document the details for each of the above elements and include, from the online study review guide (SRG) maintained by the SWAT contractor, a final, master SRG for each study as it is reviewed during development of the new or updated practice guide. The study summaries included in the SRGs are particularly useful in providing panelists with easily accessible information about which studies can be used to determine the level of evidence for practice guide recommendations. At those points in the development of the practice guide when consultations occur with panelists in accordance with the schedule developed under subtask 2.1, the contractor shall summarize the work of the reviewers so that panelists can see how individual studies might map to proposed recommendations, action steps, and roadblocks. The contractor shall make the master SRGs and summary of studies that do and do not meet standards available to the panel members for information collected during the review process. In general, the due dates for the draft and final literature review memos summarizing the SRGs prepared for the practice guide shall correspond with the schedule of practice guide preparation developed under subtask 2.1.

Deliverables (Task 3): Draft review protocol (subtask 3.1) Final review protocol (subtask 3.1) Draft training memorandum (subtask 3.2) Final training memorandum (subtask 3.2) Draft literature review (subtask 3.3) Final literature review (subtask 3.3)

Task 4: Convene the Panel and Develop the Practice Guide (Firm-Fixed-Price)

The main work of the panel occurs through a series of in-person meetings and conference calls. To ensure that the recommendations balance research evidence with expert judgment, the goal of these meetings and calls is to arrive at consensus about the practice guide recommendations and content.

The contractor shall convene the panel and develop the practice guide in accordance with the timelines and practice guide development process specified in the COR approved practice guide work plan, subtask 2.1.

Prior to submitting the first full draft of the guide, the contractor shall submit draft recommendations and supporting evidence to the COR. The COR will provide comments within two weeks. The contractor shall discuss and reconcile the COR feedback with the panel.

The contractor shall conduct a thorough quality review and edit of the full draft practice guide, and have the authors on the contractor’s team (who generally are distinct from the panelists) review and approve the draft guide and sign an IES Authorship Form prior to submitting the draft to the COR.

The COR will review the draft practice guide and provide feedback within two weeks.

The contractor shall make revisions and submit a second draft within 3 weeks. The COR and NCEE leadership will review and provide feedback on the second draft within 3 weeks. The contractor shall be prepared to make all revisions necessary to the practice guide in order to address all of the comments from the COR and from the NCEE leadership on the second draft and permit the guide to be submitted for peer review.

The COR will submit the acceptable revised practice guide to the IES Standards and Review Office (SRO) for an independent peer review. A critical task for peer reviewers of a practice guide is to determine whether the evidence cited in support of particular recommendations is up to date and that studies of similar or better quality that point in a different direction have not been overlooked. After the first round of review, the contractor shall share peer review comments with the panel and discuss and resolve substantive issues and comments raised by the peer reviewers during peer review. The contractor shall rely on panel’s feedback and suggestions to revise the draft guide. To document all revisions of the practice guide, the contractor shall draft a memo that links each proposed change made to the guide with the reviewers’ comments. The contractor shall review revisions to the practice guide and re-submit the guide to IES for a second round of SRO peer review. This iterative process occurs until the practice guide is approved by IES for production.

For price-estimation purposes, the contractor shall plan for five rounds of revisions to the draft practice guide, two with the COR (prior to the submission of the revised guide to SRO) and 3 based on SRO review in order for the guide to be accepted as an IES publication.

Deliverables (Task 4): Panel meeting, schedule, agenda, materials, minutes Authorship forms First draft practice guide Second draft practice guide

Revised practice guide for peer review Memos responding to each round of peer review comments, with proposed revisions to the practice guide

Final practice guide

Task 5: Production (Firm-Fixed-Price)

The production process includes web and print production. The practice guide will be posted on the WWC website which is hosted by IES. The contractor shall prepare all electronic files, website copy, and meet the Policies, Guidelines, Standards and Procedures for web posting for IES and NCES. The contractor shall plan to submit a 508-compliant version of the practice guide to the IES to begin production.

In addition, content that is posted to the web should follow these guidelines:

• The contractor shall ensure that the site complies with OMB Memorandum M- 03-22 (https://www.whitehouse.gov/omb/memoranda_m03-22),which requires the posting of a privacy policy.

• A ".gov" hostname (typically hostname.ed.gov) is required for all government funded web sites under OMB Memorandum M-17-06 (https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/201 7/m-17-06.pdf).

• The contractor shall ensure that any "Department-Sponsored" content is identified prominently with Department of Education, and the ED.gov logo should be displayed prominently and made a link to the ED.gov home page. The complete policy is described in the “ED.gov Management and Publishing Policies” at http://www2.ed.gov/internal/wwwstds.html#sponsor.

• The contractor shall ensure that any non-Federal sponsored websites or content (outside of .gov, .mil, .Fed.us, etc.) that is linked to complies with “ED.gov Management and Publishing Policies” at http://www2.ed.gov/internal/wwwstds.html#link and the OMB Memorandum M-17-06, “Policies for Federal Agency Public Web Sites” at https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/201 7/m-17-06.pdf.

The contractor shall submit all practice guide production materials for COR approvals as specified in the practice guide work plan, Practice Guide Work Plan 2.1 subtask.

The contractor shall develop a 508-compliant portable document format (PDF) version of the guide in the approved IES Practice Guide format. The contractor will also provide the deliverable in XML output. The contractor shall submit the final PDF and XML files to https://www.whitehouse.gov/omb/memoranda_m03-22 https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2017/m-17-06.pdf https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2017/m-17-06.pdf http://www2.ed.gov/internal/wwwstds.html#sponsor http://www2.ed.gov/internal/wwwstds.html#link https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2017/m-17-06.pdf https://www.whitehouse.gov/sites/whitehouse.gov/files/omb/memoranda/2017/m-17-06.pdf the IES web team for approval. In addition, the contractor shall develop a summary paragraph and image to accompany the guide, create a rotating banner image to advertise the release of the guide on the WWC website home page, write text for the “What’s New” announcement, and create an attractive HTML-based electronic newsflash.

The contractor shall coordinate with the WWC contractor who will send the electronic newsflash to the WWC’s electronic mailing list, all IES electronic mailing list subscribers, and relevant and potentially interested organizations and individuals, as those identified by the panelists.

5.1 Production of Practice Guide Summary Summaries

The contractor shall submit, with the 508-compliant PDFs of the body of the practice guide and the appendices for researchers, a one- and four-page nontechnical summary summaries of the practice guide that explains how education practitioners can use the practice guide information for decision making at the school and district level, as specified in the Practice Guide Work Plan 2.1 subtask, that summarizes the guide’s recommendations and action steps. The COR will review and recommend approval or edits to the summary summaries within two weeks.