91990023R0002 Amendment 0001 SF30.pdf

Offers must acknowledge receipt of this amendment prior to the hour and date specified in the solicitation or as amended, by one of the following methods:

(a) By completing items 8 and 15, and returning or (c) By separate letter or electronic communication which includes a reference to the solicitation and amendment numbers. FAILURE OF YOUR ACKNOWLEDGMENT TO BE RECEIVED AT THE PLACE DESIGNATED FOR THE RECEIPT OF OFFERS PRIOR TO THE HOUR AND DATE SPECIFIED MAY RESULT IN REJECTION OF YOUR OFFER. If by virtue of this amendment you desire to change an offer already submitted, such change may be made by letter or electronic communication, provided each letter or electronic communication makes reference to the solicitation and this amendment, and is received prior to the opening hour and date specified.

E. IMPORTANT: Contractor is not is required to sign this document and return copies to the issuing office.

AMENDMENT OF SOLICITATION/MODIFICATION OF CONTRACT 1. CONTRACT ID CODE

2. AMENDMENT/MODIFICATION NUMBER 3. EFFECTIVE DATE 4. REQUISITION/PURCHASE REQUISITION NUMBER 5. PROJECT NUMBER (If applicable)

7. ADMINISTERED BY (If other than Item 6) CODE

STANDARD FORM 30 (REV. 11/2016)

Prescribed by GSA FAR (48 CFR) 53.243

FACILITY CODE

9A. AMENDMENT OF SOLICITATION NUMBER

9B. DATED (SEE ITEM 11)

10A. MODIFICATION OF CONTRACT/ORDER NUMBER

10B. DATED (SEE ITEM 13)

11. THIS ITEM ONLY APPLIES TO AMENDMENTS OF SOLICITATIONS

The above numbered solicitation is amended as set forth in Item 14. The hour and date specified for receipt of Offers is extended. is not extended.

12. ACCOUNTING AND APPROPRIATION DATA (If required) copies of the amendment; (b) By acknowledging receipt of this amendment on each copy of the offer submitted;

13. THIS ITEM APPLIES ONLY TO MODIFICATIONS OF CONTRACTS/ORDERS. IT MODIFIES THE CONTRACT/ORDER NUMBER AS DESCRIBED IN ITEM 14.

CHECK ONE A. THIS CHANGE ORDER IS ISSUED PURSUANT TO: (Specify authority) THE CHANGES SET FORTH IN ITEM 14 ARE MADE IN THE CONTRACT ORDER NUMBER

IN ITEM 10A.

B. THE ABOVE NUMBERED CONTRACT/ORDER IS MODIFIED TO REFLECT THE ADMINISTRATIVE CHANGES (such as changes in paying office, appropriation data, etc.) SET FORTH IN ITEM 14, PURSUANT TO THE AUTHORITY OF FAR 43.103(b).

C. THIS SUPPLEMENTAL AGREEMENT IS ENTERED INTO PURSUANT TO AUTHORITY OF:

D. OTHER (Specify type of modification and authority)

Except as provided herein, all terms and conditions of the document referenced in Item 9A or 10A, as heretofore changed, remains unchanged and in full force and effect.

15C. DATE SIGNED

15A. NAME AND TITLE OF SIGNER (Type or print)

16C. DATE SIGNED

16A. NAME AND TITLE OF CONTRACTING OFFICER (Type or print)

14. DESCRIPTION OF AMENDMENT/MODIFICATION (Organized by UCF section headings, including solicitation/contract subject matter where feasible.)

PAGE OF PAGES

6. ISSUED BY CODE

8. NAME AND ADDRESS OF CONTRACTOR (Number, street, county, State and ZIP Code) (X)

CODE

15B. CONTRACTOR/OFFEROR

(Signature of person authorized to sign)

16B. UNITED STATES OF AMERICA

(Signature of Contracting Officer)

Previous edition unusable

The purpose of this Solicitation Amendment 0001 is to provide responses to clarification questions, provide revised documents based on question responses, and extend the due date for offeror response to November 14, 2022 at 1:00PM EST.

1 102

91990023R00020001 OCT 20, 2022

CAMPCP

US Dept of ED-550 12th Street SW - Room 7169 Washington DC 20065

See Block 6

91990023R0002

OCT 04, 2022

See Schedule

Table of Contents

SECTION C DESCRIPTION/SPECIFICATIONS/WORK STATEMENT

C.1 Questions and Responses

C.2 Performance Work Statement (PWS)

SECTION L INSTRUCTIONS, CONDITIONS, AND NOTICES TO OFFERORS

L.1 Instructions to Offerors

SECTION M EVALUATION FACTORS FOR AWARD

M.1 Evaluation Criteria

PAGE 2 OF 102 91990023R00020001

SECTION C

DESCRIPTION/SPECIFICATIONS/WORK STATEMENT

C.1 Questions and Responses

PAGE 3 OF 102 91990023R00020001

WHAT WORKS CLEARINGHOU SE TM :

Tools , Onlin e Assi s ta n c e , Stan dard s , and Trainin g (WWC-

TOAST)

Quest io n s and Answers

Tech ni c a l Propos a l:

Ques t io n: Does IES expect the Technical Plan to addres s Task Order 1, Task Order 2, and subtasks not curren tly listed under either task order but listed under the anticipa t ed scope of work?

Answer: The technical plan should briefly discuss all tasks in the PWS. This can be at a high level and does not need to have the same level of detail as the workplan for Task Order 1 and 2. Please see the revised section in the RFP for clarity.

Quest io n: In the Instruc t ions to Offerors , Point 8 (pg. 114) says “The contrac to r shall copy the table highlighted in green into the technical proposal and submit the Excel file as par t of the business proposal .”

Please confirm that bidders need to copy each table highlighted in green from the Named Staff, Task Order 1, and Task Order 2 tabs to the technical proposal.

Answer: Yes. Only these tables shall appea r in the technical proposal.

No cost information should appea r in the technical proposal.

Quest io n: Please confirm the time commitm en t table from Excel should be within the technical narra t ive volume with 10 pt font and single-spaced.

Answer: Yes. This should be format t ed as a table in the technical proposal .

Quest io n: Would it be possible to change the Deliverable schedule from 12 pt font double- space to 10 pt single- space?

Answer: Yes. See revised instruc tions .

Quest io n: Requiring tables to be double- spaced makes it difficult to presen t some mate r ial that would be visually and conceptua lly well-suited to a table. While we apprecia t e the desire for a clear and concise nar ra t ive in the technical proposal, would ED be willing to allow a limited number of single- spaced tables beyond what is allowed in the curren t instruc t ions? For example, would ED permit up to four

PAGE 4 OF 102 91990023R00020001

tables that are single- spaced, with 10- point font, and no longer than one page with the 75 page limit?

Answer: Offerors may include up to four tables that are single-spaced, with 10- point font, and no longer than one page combined.

That is, all four tables must fit on an 8 x 12 sheet of paper but may be spread out throughout the proposal.

Quest io n: Several subtasks in Task Orders 1 and 2 have differen t subtask numbe r s from those in the BPA. For example, Certify New Reviewers is subtask 5.3 in the BPA and is subtask 5.2 in Task Order 1. Should bidders follow the subtask number s in the BPA? Or should they follow the subtask number s in the individual task orders?

Answer: ED has revised the task number s to be consisten t with the BPA. For Task Orders 1 and 2 this means that the number s of tasks will not be consecu tive because some tasks in the BPA are not curren t ly awarde d.

Quest io n: The Inst ruc t ion to Offerors for the Technical Volume notes that the technical plan should include "A descrip tion of work that the offeror, including key personnel , has conducted in the past that demonst r a t e s that the offeror has the experience to carry out the task as described." However , the evaluat ion criteria for the genera l approach to the Performa nc e Work Statem e n t do not include organiza t ional capabilities or staff experience . Can ED confirm that descrip tions of past experience should be included in the personnel and organizational capabili ties sections rathe r than in the technical plan?

Answer: Please see evaluation crite ria numbe r 1.

Quest io n: Should bidders include letter s of commitme n t for proposed member s of the practi tione r working group in the Appendix?

Answer: Yes. Please submit lette rs of commitme n t .

Quest io n: PWS Task Orders 1 and 2, Task 2, Task 2.1 Creation of IT Reporting Mater ials require s the use of the IES KU IT Reporting Template . This templat e is embedd e d in Appendix C on Page 56.

However , we have been unable to open this file. Can the file be provided separa t e ly?

Answer: Yes, see attachm e n t s to RFP.

PAGE 5 OF 102 91990023R00020001

Task 1:

Ques t io n: Should we budge t for the cross contrac to r meetings to be two times a year or three times a year? The scope of work and Task Order 1 note differen t frequencie s .

Answer: For Task Order 1, there will be up to two cross contrac to r meetings during the 12- month period of perform anc e . In future task orders , there may be up to three meetings . The exact number will be specified in future task orders .

Quest io n: Is the Manage m e n t task a hybrid or a non- IT task? It’s a hybrid task for the BPA (RFP page 27), but it’s a non- IT task for Task Order 2 (RFP page 40).

Answer: Manage m e n t has both IT and non- IT subtasks . Only non- IT subtasks are exercised for Task Order 2.

Task 2:

Ques t io n: The Build Tools to Track Workflow subtask is not in the BPA’s Anticipated Scope of Work but is in Task Order 2’s SOW for Task 2.

Please confirm that Task 2 for Task Order 2 should include this subtask.

Answer: Yes. It is conside red to be within scope of Subtask 2.2 of the

BPA.

Quest io n: The PWS, Task Order 2, Task 2.5 Enhance m e n t s to the OSRG notes that for the OSRG, "One functionali ty that the contrac to r shall need to develop is the ability to assign multiple manusc rip t s to a single study ID and multiple study IDs to a manusc rip t with multiple contras t s ." Can ED clarify how this added functionality differs from the curren t functionality to assign related sources to reviews and the ability to assign multiple study IDs to a manusc rip t using the "Relevant Comparison" field in the OSRG?

Answer: The curren t functionality does not allow all related manusc rip t s to share the same study ID. Additionally, several manusc rip t s have multiple study IDs for each contras t . The curren t functionality is not sufficient to allow for the complexity of grouping needed.

Task 3:

PAGE 6 OF 102 91990023R00020001

Quest io n: What is the contrac t type for Subtask 3.1 – Propose and Convene STAT Small Working Groups? It is CPFF on page 19 of the RFP and FFP on page 33.

Answer: It is Cost Plus Fixed Fee.

Quest io n: What is the contrac t type for Subtask 3.2 – Propose and Convene an All-STAT? It is CPFF on page 20 of the RFP and FFP on page 34.

Answer: It is Cost Plus Fixed Fee.

Quest io n: Is the contrac t type for subtasks 3.1 and 3.2 and Task 3 CPFF or FFP? It is CPFF on pages 19 and 20 of the RFP and FFP on pages 33 and 34. In addition, if it is to be trea ted as FFP, it’s not listed in the Pricing Matrix tab which includes FFP tasks.

Answer: See above

Task 4:

Ques t io n: Does PWS Task Order 1, Task 4.2 entail conduc ting peer review of all studies, all studies that meet standa rd s , or only studies that meet standa rd s and are included in WWC publications? BPA, PWS Task 4.2 states "All WWC reviews of studies included in publications. . .

are subject to review" but also states later that "Contrac to r s shall be expected to peer review studies that meet WWC standa r ds ." PWS Task Order 1, Task 4.2 states that "All WWC studies and publica tions are subject to review" and the introduc tion of the PWS states that "All studies are peer reviewed."

Answer: All studies that meet standa r ds

Task 5:

Ques t io n: The curren t PWS does not include language for updating the FAQs, should we assum e that work will be included in this task?

Answer: Please see the revised PWS. This is considere d in scope of Task 5.

Quest io n: Is the Provide Annual Training and Feedback an IT or a non- IT task? It is a non- IT task for the BPA (RFP Page 24), but it is an IT task for Task Order 1 (RFP page 37).

Answer: It is not IT.

PAGE 7 OF 102 91990023R00020001

Quest io n: Is the Provide Target ed Training and Answer Target ed Questions from Reviewers an IT or a non- IT task? It is a non- IT task for the BPA (RFP page 25), but it is an IT task for Task Order 1 (RFP page 38).

Answer: It is not IT.

Quest io n: Is the Communica t e About WWC Standa r ds and Review Process an IT or a non- IT task? It is a non- IT task for the BPA (RFP page 25), but it is an IT task for Task Order 1 (RFP page 38).

Answer: It is not IT.

Instru c t i o n s to Offerors:

Ques t io n: Can you please clarify what is required for the concept paper referenc ed on page 110?

Answer: This was included in error . There are no concept papers for this procure m e n t .

Quest io n: If Subtasks do not curren tly include deliverables , should we propose them along with a potential due date?

Answer: Yes, for Task Orders 1 and 2.

Busin e s s Propo s a l s :

Ques t io n: Business proposal instruc t ions – Price Informat ion (RFP page

116) states that “offeror shall submit proposed costs for each task and subtask, and for the project overall, to encompas s the entire period of perform anc e” and “the business proposal shall include a separa t e cost estimate for each task and subtask for all five years of performa nc e .”

The period of performa nc e for Task Orders 1 and 2 is 12 or 15 months and 24 months , respec t ively. In addition, the TOAST+Budg e t + T e m pl a t e .xlsx file only has tabs to submit a full budge t (for CPFF and FFP subtasks and tasks) for Task Orders 1 and 2 only. Please clarify what budge t bidders need to submit in the business proposal.

Answer: The budge t templat e is required to provide in-depth price information for Task Order 1 and 2 and offerors are required to use the templat e for those tasks. Pricing informat ion is also needed for all

PAGE 8 OF 102 91990023R00020001

five years of the contrac t . Please see the revised budge t templat e for specific tables.

Quest io n: Please clarify what bidders need to submit at the BPA level in the business proposal.

Answer: The business proposal shall include a separa t e cost estimate for each task and subtask for all five years of perform anc e . Business proposals that do not provide separa t e breakou t s for each task and subtask are unaccep ta ble, and the Depar t m e n t will not conside r your proposal for award. Please see the revised budge t templat e for specific tables.

Quest io n: Business proposal instruc t ions – Labor Rates (RFP page 118) states that “labor hours shall be proposed for the developme n t of costs, in the event that it becomes necessa ry to extend the period of perform anc e for up to 6 months beyond the stated period of perform anc e .” Will the Depar t m e n t please clarify the meaning of the “stated period of performa nc e”? And is the informat ion to be submit ted in an Excel file separa t e from the TOAST+Budg e t + T e m pl a t e .xlsx file?

Answer: Please see the revised instruc t ions . Offerors shall assum e that the level of effort for any given year will be similar to Task Order

1. Offerors can include this information in their TOAST budge t templa t e .

Quest io n: Please confirm bidders should submit the Excel version of the budge t using the TOAST+Budge t + T e m pl a t e .xlsx file and not submit a separa t e Excel file that includes our full budge t and bid

Answer: Offerors must submit the TOAST budge t templa t e . They may supplem en t the information in the templat e with additional Excel files, if needed.

Quest io n: Are bidders allowed to add lines and tabs to the Excel file?

Answer: Offerors may alter the templa te , as long as the informa tion in the green shaded cells is accura t e and complete . If modifica tions to the templat e fail to the provide ED with the information needed in the reques t e d format, it will be seen as a weakness in the proposal.

PAGE 9 OF 102 91990023R00020001

C.2 Performance Work Statement (PWS)

PAGE 10 OF 102 91990023R00020001

WWC- TOAST BPA Performa n c e Work State m e n t , Octobe r 19, 2022

U.S. Depart m e n t of Educat i o n Insti tu t e of Educat i o n Scien c e s

Natio n a l Center for Educat io n Evaluat i o n and Regio n a l Assis ta n c e

WHAT WORKS CLEARINGHOU SE TM :

Tools , Onlin e Assi s ta n c e , Stan dard s , and Trainin g (WWC-

TOAST)

Blank e t Purcha s e Agree m e n t (BPA) Perform a n c e Work State m e n t (PWS)

PAGE 11 OF 102 91990023R00020001

WWC- TOAST BPA Performa n c e Work State m e n t , Octobe r 19, 2022

I. Introduc tion

The U.S. Depar tm e n t of Education (the Depar t me n t) , via the National Center for Education Evaluation and Regional Assistance (NCEE) of the Institute of Education Sciences (IES), needs a contrac to r to manage the What Works Clearinghous e TM (WWC) website and to provide technical suppor t on the developme n t and review of the produc ts and information contained on the website . The WWC address e s the need for credible, succinct informa tion by identifying existing resea rc h in educa tion, assessing the quality of this resea rc h , and summa rizing and dissemina ting the evidence from studies that meet WWC standa r ds .

The WWC achieves its mission through the work of a series of contrac t s . First , the conten t contrac to r s review studies in a specific area or for a specific purpose . At time of award, there are curren tly four conten t contrac t s : The Guides to Research to Assist Practit ione rs in Educa tion (GRAPE) task orders are to produce Practice Guides and Interven t ion Repor t s . The What Works Clearinghous e - Preschool- to- Postseconda ry Evidence Synthesis Task Orders (WWC- PESTO) similarly produce both Practice Guides and Interven t ion Repor t s in specific areas. There are curren t ly two task orders for PESTO and more task orders are expected in the future .

The What Works Clearinghous e Ad-Hoc Reviews of Individual Studies in Educa tion (WWC- ARISE) conduc ts reviews of individual studies .

The Organiza tion of Repor ting of Evidence on Gradua tion, Achievemen t , and Nonacad e mic Outcomes (WWC- OREGANO) is slated to end in May 2023 and will have minimal overlap with the curren t require m e n t .

In addition to the conten t contrac to r s , there is a coordina t ion contrac t . The curren t require m e n t is for the coordina t ion require m e n t . The Tools, Online Assistance , Standa r ds , and Training (WWC- TOAST) contrac to r achieves the objectives of this contrac t through six main activities:

1. The WWC develops statis tical standa r ds and procedu r es about how studies should be reviewed. The curren t set of standa rd s can be found at https://ies.ed.gov/ncee/wwc/H an d books . The standa r ds and procedu r es are developed in consulta t ion with methodological exper t s and WWC conten t contrac to r s through the Statis tical, Technical, and Analysis Team (STAT). The STAT team meets in small groups and as a large group to discuss how to improve the WWC standa r ds and procedu re s going forward.

PAGE 12 OF 102 91990023R00020001

2. The contrac to r will train users on the standa r ds to become cer tified reviewer s for the WWC. This will be done through developing online training, maintaining the certification process , providing annual training and targe t e d technical assistanc e , and answering reviewer questions.

3. The TOAST contrac to r will use those standa rd s to build and maintain the WWC Online Study Review Guide (OSRG) applica tion. WWC conten t contrac to r s will use the OSRG to review studies. Then conten t contrac to r s will be working with the TOAST contrac to r to publish the studies.

4. The contrac to r will ensure quality of reviews and publica tions by providing peer review and quality control. All studies that meet WWC standa rd s are peer reviewed by exper t s to ensure that they are accura t e and do not have errors that could undermine the credibility of the WWC.

5. The contrac to r will publish study reviews and repor t s based on study reviews on the WWC website , found at https://wha tworks .ed .gov . They will routinely enhance the website to ensure it is user friendly and accura t e .

6. Lastly, the contrac to r will conduc t reviews and produce produc ts that will give them exper tise using the system they create .

The WWC uses its public websi te to communica t e with the world and the primary purpose and objective of the contrac t is to maintain and update the produc ts and tools, including the public tools and the internal tools. The contrac to r will do this through proposing a strong team that unders t a n ds both the WWC’s online tools and the WWC standa rd s and how to dissemina te the work of the team to broad audiences . All developme n t for TOAST must occur in the IES Data Cente r and must conform to the Data Cente r’s Authori ty to Opera t e .

The TOAST contrac t is inheren t ly an informa tion technology contrac t that requires an integra t e d team with a strong unders t a n ding of education resea rch and statistical standa rds . The contrac to r needs to unders t a n d the statis tical standa r ds and the underlying technology of the website in order to provide innovative and cost- effective solutions. A successful team will be integra t e d with team member s with multiple skillsets and have sufficient experience and exper tise to respond to problems and develop effective solutions.

I. Authorizing Legislation

The legislation authorizing this contrac t is Title I, Part D of the Educa tion Sciences Reform Act of 2002, Section 171(b):

(b) MISSION. —The mission of the National Cente r for Educa tion Evaluat ion and Regional Assistance shall be—

(1) to provide technical assistance .

(2) to conduct evaluations of Federal education progra m s

PAGE 13 OF 102 91990023R00020001

adminis te r e d by the Secre t a ry (and as time and resources allow, other education progra m s) to dete rmine the impact of such progra m s (especially on studen t academic achievem e n t in the core academic areas of reading, mathe m a t ics , and science);

(3) to suppor t synthesis and wide dissemina tion of resul ts of evalua tion, resear ch , and produc ts developed; and

(4) to encourag e the use of scientifically valid educa tion research and evaluat ion throughout the United States .

The 2015 reautho riza t ion of the Element a ry and Seconda ry Educa tion Act (ESEA, P.L. 114- 95), the Every Studen t Succeeds Act (ESSA), emphasized the need for educa to r s and policy makers to identify “evidence- based” practices to improve studen t outcomes and other outcomes relevan t for educa tion. In par ticula r , ESSA amende d ESEA to include definitions of “strong evidence” and “modera t e evidence” (20 U.S.C. 7801, section 21), corresponding with statis tically significan t and favorable findings from “well-designed and well- implemen t e d” quasi- experime n t a l and experime n t a l studies.

II. WWC Background

The Depar t m e n t established the WWC in 2002 to provide educato r s , policymake r s , resea rch e r s , and the public with a transpa r e n t way to dete rmine whethe r there is evidence of effectiveness behind a progra m or practice and the extent to which those findings can be trus ted . A goal for the WWC is to promote bet te r informed education decision- making through a web- based dissemina t ion system that featu re s compre he n s ive, high- quality system a t ic reviews of studies on the effectiveness of education interven tions . The websi te , whatworks .ed .gov , is the primary “face” of the WWC, displaying the resul ts of WWC review as well as resources for educa to r s . The WWC is focused on providing relevan t and useful reviews of evidence, refining its standa rd s and procedu re s based on recen t improvem e n t s in resear ch methods, and dissemina ting actionable information and related evidence resources to the public.

III. Period of Performa nce

The ordering period for this BPA is 60 months from the date of contrac t award. Individual task orders under this contrac t may last up to 24 months from the time of award and may extend past the end of the ordering period if they are awarded within those 60 months.

PAGE 14 OF 102 91990023R00020001

The contrac t is expected to begin in Februa ry 2023 with an optional 3-month transi tion period that would run from Februa ry 2023 to May 2023. During the transi tion period, the contrac to r shall only work to unders t a nd the existing software and tools to be able to fulfill the responsibilities of the contrac t with minimal disruption for users. After the 3-month transi tion, a 12- month period of performanc e would begin and would run from May 2023 to May 2024 for Task Order 1. Task Order 2 would have an estimated period of performanc e from May 2023 to May 2025.

IV. Types of Task Requireme n t s

The Depar tm e n t will issue specific work assignmen t s as task orders under this BPA, specifying the task objectives, the work to be performed, deliverables , schedules , and other requirem e n t s . The nature of the work will vary from task order to task order. The Depar tme n t expects to award firm- fixed- price, cost reimburse m e n t , and hybrid tasks under this BPA to a single prime contrac to r .

V. Staff Qualifications

The contrac to r shall propose a well- manage d and highly qualified team to manage the work of TOAST. Proposed staff are expected to have sufficient time committed to the project , have a deep unders t a n ding of the work, and adequa t e time to be responsive to the needs of IES. The leade rs hip team, defined as the Principal Investiga to r (optional), Project Director , and Deputy Project Director s are required to spend a combined level of effort of at least 2.2 FTE and the minimum levels specified below. This level of effort is across all open task orders . For planning purposes , the contrac to r shall assum e that a task order similar in scale to Task Order 1 will be exercised each year and that the WWC will develop a version 6.0 handbook in years 3 and 4 of the contrac t and develop the associa ted training and new Online SRG in year 5 of the contrac t .

IES expects that each position shall be held by one individual. Having one individual ensure s that there is a sole point of communica t ion and there are clear lines of account abili ty. The contrac to r may propose individuals sharing a role if the contrac to r can convince the governm e n t there is sufficient benefit to the governm e n t from the arrange m e n t and the associa ted risks of multiple individuals have been mitigated .

PAGE 15 OF 102 91990023R00020001

- Principal Inves tigator (Optional): The Principal Investiga to r is responsible for set ting the direc tion and goals of the project . They shall be actively involved in the Statis tical, Technical, and Analysis Team (STAT) and all project planning. They shall have exper tise in statis tical methods, presen t a t ion of findings, and conduc ting study reviews. The Principal Investiga to r shall be cer tified in WWC Group Design Standa r ds , version 4.1 at the time of proposal submission and shall be certified in WWC general group designs, advanced group designs , and single- case designs (SCDs), version 5.0, no later than the end of transi tion and star t of the core tasks of the contrac t 1 . They shall be commit ted to at leas t 0.4 FTE.

- Project Director: The Project Directo r is the lead for Task 1. It is the role of the Director to ensure the work of individual tasks inform each other and the overall work of the project. It is the Project Director who is ultimately responsible for ensuring that work is submitt ed error free, on time, and meeting the require m e n t s of the contrac t . The Project Directo r is expected to be in daily communica t ion with the COR and the project team and is the main point of contac t for all questions about the contrac t .

The Project Director shall be certified in WWC Group Design Standa rd s , version 4.1 at the time of proposal submission and shall be cer tified in WWC genera l group designs and advanced group designs , version 5.0, no later than the end of transi tion and star t of the core tasks of the contrac t . The Project Director shall be commit t ed for at leas t 0.6 FTE for the dura tion of the project .

- Deputy Project Director for OSRG and Website: This position is responsible for leading task 2. This person should have a deep technical knowledge in conduc ting reviews for the WWC and using the Online Study Review Guide (OSRG). They should unders t an d how the OSRG and website function and technical concerns about improvem e n t s and enhance m e n t s . The Deputy Project Directo r is responsible for ensuring that the OSRG opera te s smoothly and that contrac to r s can complete reviews and publish them to the website with minimal interac t ion with the TOAST contrac to r . The Deputy Project Director shall also ensure all timelines for product release s are met and that work is consisten t ly high quality. They shall be in daily communica t ion with relevan t task and project leads and the project direc to r to ensure that IES and project leadership are informed of all relevant work on the contrac t . The Deputy Project Director shall be certified in WWC Group Design Standa rd s , version 4.1 at the time of proposal submission and shall be cer tified in WWC genera l group designs and advanced group

1 Online training for version 5.0 is expected to be online in early 2023. It will be available at https://ies.ed.gov/ncee/wwc/OnlineTraining

PAGE 16 OF 102 91990023R00020001

designs , version 5.0, no later than the end of transi tion and star t of the core tasks of the contrac t . The Deputy Project Directo r shall be commit t ed for at leas t 0.8 FTE.

- Deputy Project Director for Reviews, Peer Review, Training, and STAT: This position is responsible for leading tasks 3-6. This person should be an experienced reviewer for the WWC with extensive methodological exper t ise . The Deputy Project Director is charged with overseeing the day- to- day opera t ions of the tasks and ensure all timelines are met and that work is of high quality. They shall be in daily communica t ion with relevan t task and project leads and the project direc to r to ensure that IES and project leadership are informed of all relevant work on the contrac t . The Deputy Project Director shall be certified in WWC Group Design Standa rd s , version 4.1 at the time of proposal submission and shall be cer tified in WWC genera l group designs , advanced group designs , and single- case designs (SCDs), version 5.0, no later than the end of transi tion and star t of the core tasks of the contrac t .

The Deputy Director shall be commit ted for at least 0.4 FTE for the dura t ion of the project.

VI. Anticipated Scope of Work

The following tasks and subtasks are included in the scope of work:

Task 1. Manag e m e n t (Hybrid, Firm Fixed Price)

The contrac to r shall manage task orders in an efficient manne r that fosters communica t ion with the COR and other WWC contrac to r s . The contrac to r shall have sound project manage m e n t processe s and personnel who can manage multiple task orders simultaneously, as it is anticipated that several task orders can have overlapping deadlines.

Typical task order manage m e n t activities are outlined below.

1.1. Kickoff Meet i n g (Non- IT)

Within the first 2 business days after the contrac t star t date, the project leadership shall attend a kickoff meeting with the Depar tm e n t to discuss details regarding the tasks outlined in the proposal, scheduling activities, the transi tion process , and other issues related to the contrac t . Project leadership is defined as the primary investigato r , project direc tor , deputy project director , and may also include the task leads and/or contrac t adminis t ra to r s .

The meeting will be held via conferenc e call. The contrac to r shall provide an agenda for the meeting within 1 business day after the contrac t star t date . Within 5 business days following the meeting, the

PAGE 17 OF 102 91990023R00020001

contrac to r shall prepa re a memo summarizing the key issues and concerns raised at the meeting and how each will be addre ss ed . The memo also shall documen t key decisions made at the meeting, as well as any specific technical direc tions provided by the Contrac t ing Officer’s Represe n t a t ive (“COR”).

1.2. Transi t io n - in Materia l s (IT, Option a l )

The contrac t is intended to be awarded with an up to 90- day transi tion- in period that ends on May 17, 2023. Prior to May 17, 2023, the contrac to r shall only perform work on subtask 1.2. During this period the contrac to r is responsible for ensuring that staff has adequa t e securi ty clearanc es and permissions to access the WWC websites and tools and that they are adequa t e ly trained in the system to meet the metrics specified in the contrac t .

Within 2 days of contrac t award the contrac to r shall submit a reques t to IES to initiate any staff that do not have securi ty clearances and a reques t to validate staff member s that have existing clearanc es . These reques t s shall be in the format specified in the Contrac to r Employee Securi ty Screening Requirem e n t s section of the PWS. Once initiated , staff will have 14 days to complete all paperwork and retu rn to the COR. The COR will notify the project direc tor when the package s have been accep ted by the Securi ty Office in the National Cente r for Educa tion Statistics and individuals are authorized to have access to the applica tions .

At the kickoff meeting, the COR shall provide a copy of the transi tion plan developed by the WWC SWAT contrac to r . Within 21 days of contrac t award, the contrac to r shall prepa re a transi tion in memo that uses the dates specified in WWC SWAT contrac to r’s transi tion out memo as a basis for the transi tion. The memo shall documen t the timeline for each transi tion activity and responsible par ties.

The contrac to r is responsible for ensuring all staff is adequa t e ly trained and has approp ria t e access to begin accessing the OSRG tools, WWC Website , and the help desk by May 17, 2023. The contrac to r is expected to meet all contrac t require m e n t s and have a seamless transi tion from SWAT to TOAST. The contrac to r shall submit a memo confirming that the transi tion is complete no later than May 17, 2023.

1.3. Biwe e k ly Meet i n g s (Non- IT)

The project direc tor and other key staff that the project direc tor deems are necessa ry shall meet with the COR and other appropria t e Depar t m e n t staff via biweekly conference calls to be held at regula rly

PAGE 18 OF 102 91990023R00020001

scheduled intervals throughout the dura tion of the contrac t . The purpose of the meetings is to brief the Depar tm e n t on progress and challenges related to the tasks in this contrac t . The Contrac to r shall send an agenda to the COR 3 days in advance of the meeting and a meeting summa ry to the COR within 3 business days after the meeting is complete .

In addition, the Contrac to r shall be available for meetings to address unexpec t ed issues that arise during the contrac t period. These meetings may occur on an as- needed basis at the reques t of the COR or the Contrac to r and will take place via conferenc e call. The contrac to r should assum e one meeting a week in the first six months of the contrac t and approximat ely one meeting per month thereaf t e r .

1.4. Month ly Report (Non- IT)

The contrac to r shall submit a monthly repor t that summa rizes progress made in each task and subtask, and shall specifically include the metrics specified in the tasks below and listed in Appendix B. The repor t shall highlight progress made and any challenges faced. In addition, the monthly repor t shall include proposed work and expected challenges for the next month.

Contrac to r shall provide data on earned value in a specific format specified in Appendix C. Any deviations in schedule or cost shall be fully explained in the risk regis te r . Additionally, the monthly repor t shall include the IT repor ting shee t and the contrac to r personnel listing using the templat e s provided in the solicita tion.

The contrac to r shall submit the repor t on the sixth business day of every month for the dura tion of the contrac t and shall be prepa re d to discuss it in the next biweekly call with the COR.

1.5. Cross Contrac t or Meet i n g s (Non- IT)

The contrac to r shall be available to meet up to three times a year for cross- contrac to r meetings . These meetings shall be held in Washington, DC and/or virtually. The COR will provide ample notice of the time and the agenda for the meeting. The contrac to r may be asked to presen t or facilita te a discussion for par t of the agenda.

Task 2: Webs i t e , Data, and Online Tools (IT, Cost Plus Fixed Fee) The websi te is the primary way for the WWC to share their products and resources with the world. It is the responsibility of the TOAST contrac to r to ensure that the Online SRG is functional. If problems arise, it is the primary responsibility of the TOAST contrac to r to quickly and accura t e ly resolve any issues.

PAGE 19 OF 102 91990023R00020001

2.1. Creatio n of IT Report i n g Materia l s

Within 4 weeks of contrac t award, the contrac to r shall a submit work breakdow n structu r e of all deliverable s , down to the individual personnel level for cost plus fixed fee deliverables , in Microsoft Project format, a 2-3 page overview of the quality manage m e n t plan, and a 2-3 page overview of the risk manage m e n t plan. Guidance for these documen t s is provided in Appendix C.2.1.

2.2 Develo p a Help Desk for Contrac t or s

The contrac to r shall be responsible for monitoring the email addre ss WWC.Cont rac to r .Help.Desk@ed.gov and Help Desk reques t s sent through the OSRG. This help desk will receive all inquiries from conten t contrac to r s about problems with the OSRG and publishing conten t on the website .

The contrac to r shall monitor the email addre ss from 9am- 5pm ET, excluding federal holidays. Contrac to r s shall check the email at least once an hour and provide a subst an t ive reply, defined as one that solves the problem or follows up with the conten t contrac to r about when to expect a resolution, within 90 minutes . A subs tan t ive reply shall not be a form response , nor shall it be an acknowledge m e n t of the email with the promise of a later response . Substan t ive replies shall be specifically tailored and addre ss the needs of users, reques t s more informa tion, or asks to set up a meeting to discuss the issue furthe r . The contrac to r shall follow up on all issues raised by the conten t contrac to r s and promptly inform them when issues have been resolved. If a task requires a fix to the OSRG, that should occur under subtask 2.3.

Up to two individuals will have email addres se s and the ability to access the help desk. These individuals shall be certified in WWC Group Design Standa r ds , version 4.1 and both genera l group designs and advanced group designs , version 5.0. These individuals have exper tise in the functionali ty and design of the Online Study Review Guide and experience conducting reviews.

With the agenda for the biweekly meeting (see subtask 1.3), the contrac to r shall submit a table summa rizing the work of this task. It should contain a list of emails received since the previous call, a descrip tion of each issue raised, the status of each ticket, and the expected time of resolution. In the monthly repor t , the contrac to r shall repor t on 1) the number of email tickets submit ted by conten t contrac to r s , 2) the propor t ion of tickets that are work stoppag es , 3) the propor tion of tickets that are error s versus enhance m e n t s , 4) the

PAGE 20 OF 102 91990023R00020001

amount of time from receipt to sending a substan tive reply, and 5) the mean and median time from receipt of email to issue resolution.

2.3 Mainta i n and Updat e the Onlin e Study Review Guide

The contrac to r shall be responsible for maintaining and upgrading the WWC’s OSRG. The OSRG tool is used by all WWC contrac to r s , as well as other IES contrac to r s and grantee s to review studies against WWC standa rd s and protocols. It is housed in the NCES Member s Site in the IES Data Cente r at the URL:

https://me mb e r s .nc e s . ed .gov/OSRG/Account/Login?Retu rn U rl = %2fosrg%2f . Information about the Online SRG can be found here:

https://youtu.be/W99s8RlNXis , https://youtu.be/HT6qkZzpjAw , and https://youtu.be/x3Jy0GaLYf0 .

The contrac to r shall be responsible for ensuring that the tool is functional and is accura t e ly completing reviews. On a weekly basis, the contrac to r shall release updates to the SRG that fix aspect s of the tool that are not working for the reviewers . (See subtask 2.2 for more information). The contrac to r shall design improveme n t s and enhance m e n t s that are cost- effective and are long- term solutions that would work for future reviews, as opposed to short- term patches .

The contrac to r shall have sufficient capacity to quickly address any issues that cause work stoppag es , as well as make long term enhance m e n t s . This means that the contrac to r shall have the ability to bring in additional staff when needed to quickly scale up the level of effort so that the SRG remains functional, and product s are not delayed. Contrac to r s are expected to resolve issues that are work stoppage s within 1 week and other issues within 4 weeks.

The contrac to r shall also ensure all functionali ties of the OSRG are sufficiently docume n te d through the User’s Guide through subtask

5.8. The User’s Guide shall be consis ten t ly up to date and designed in a way to be useful to reviewers and efficient to update.

The contrac to r is also responsible for adding and maintaining user accoun ts and ensuring users have complet ed necessa ry cybersec u r i ty training. On a quart e r ly basis the contrac to r shall review all accoun ts to ensure that users are actively on contrac t and have approp ria t e trainings to use the tool.

With the agenda for the biweekly meeting (see subtask 1.3), the contrac to r shall submit a table of open tickets , their status, and estimated level of effort to resolve the issue, and estimate d completion date. In the monthly repor t , the contrac to r shall repor t on

1) number of tickets opened, 2) number of tickets closed, 3) mean and

PAGE 21 OF 102 91990023R00020001

median time for ticket resolution, 4) number of labor hours for subtask 2.2, and 5) number of tickets not yet addre ss e d . The contrac to r shall also provide an Excel file that has a list of all studies curren t ly under review, in the format specified in Appendix B as par t of the monthly repor t .

2.4 Mainta i n the WWC Websi t e

The WWC website is designed to be responsive to user needs, efficient, and adaptable to new oppor tuni t ies available through new technology. The contrac to r shall maintain the existing site by adding new communica t ions or resource produc ts to the website , announcing events , publishing new reviews, and releasing new produc ts . The COR and TOAST contrac to r will work with the WWC conten t contrac to r s to develop a schedule for publishing study reviews and publica tions , as described in subtasks 2.3 and 2.4. The contrac to r is expected to meet all dates on the agreed upon schedule and release all repor ts on the agreed upon date.

For reques t s to publish new reviews, communica t ions produc ts , events , and other announc e m e n t s , the conten t contrac to r s shall submit reques t s through the contrac to r help desk (see subtask 2.1).

The contrac to r will be responsible for assigning each produc t a ticket numbe r and ensuring that new conten t is posted within 2 weeks.

When developing new conten t , the contrac to r shall send the COR a link to preview any pages on the IES Developme n t Server . All updates to the website will be approved by the COR prior to implemen t a t ion.

In the monthly repor t , the contrac to r shall provide: 1) the numbe r of new website changes made, 2) the mean and median amount of time for tickets to be address e d , 3) the number and percen t ag e of tickets that take more than 2 weeks to resolve.

2.5 Enhan c e m e n t s to the WWC Websi t e

The contrac to r shall periodically review the WWC website and propose enhance m e n t s in design or functionality that maximizes usability. Examples of websi te enhance m e n t s include reorganizing the WWC Resources page, improving the design and functionality of the download data from individual studies file, redesigning study pages , improving interven t ion search resul ts, and/or developing an API of the WWC data.

At the star t of each task order , the COR and contrac to r shall develop an estimated budge t and brains to rm topics. The contrac to r shall use that information to propose a series of poten tial updates and their level of effort on an ongoing basis. Upon gaining the COR’s approval, PAGE 22 OF 102 91990023R00020001 the contrac to r will develop a timeline and budge t for developm en t . In the monthly repor t , the contrac to r shall provide an update on each enhance m e n t and a discussion of any changes or modifications to the timeline.

2.6 Enhan c e m e n t s to WWC Und erlyin g Data

The contrac to r shall periodically review the underlying WWC databas e and propose enhance m e n t s that increas e transpa r e ncy or maximize usabili ty. Examples of databas e improveme n t s include standa rdizing interven tion names or outcome domains , adding additional fields, cleaning existing data, and/or rest ric ting existing fields. In year 1, the contrac to r shall develop a memo the summarizes existing issues. The known issues include, but are not limited to, a lack of official effect size column, nega tive effect sizes when there are positive improveme n t indices, incorrec t ESSA tier flags, and missing WWC- calcula ted effect sizes.

The contrac to r shall use the information from the memo to propose a series of potential update s and their level of effort. Upon gaining the COR’s approval, the contrac to r will develop a timeline and budge t for developm e n t . In the monthly repor t , the contrac to r shall provide an update on each enhanc e m e n t and a discussion of any changes or modifica tions to the timeline.

2.7 Enhan c e m e n t s to the OSRG

The contrac to r shall periodically review the OSRG. Enhance m e n t s may include improving the reconciler view, improving the way to input informat ion to reduce error s, improving the layout, and adding the ability to review new types of studies. It is also within scope to make major changes to the functionality or design.

At the star t of each task order , the COR and contrac to r shall develop an estimated budge t and brains to rm topics. The TOAST contrac to r shall correspond with existing WWC reviewers to dete rmine issues that they are experiencing and potential enhance m e n t s that may be helpful. This informa tion should be used to propose a series of potential updates and their level of effort on an ongoing basis. Upon gaining the COR’s approval, the contrac to r will develop a timeline and budge t for developme n t . In the monthly repor t , the contrac to r shall provide an upda te on each enhance m e n t and a discussion of any changes or modifica tions to the timeline.

2.8 Conven e a Practi t io n e r Workin g Group to Advise on WWC Conte n t for Non- Rese ar c h e r s Within 6 weeks of contrac t award, the contrac to r shall develop a practi tione r work group to advise the WWC on the prioritiza tion and

PAGE 23 OF 102 91990023R00020001

presen t a t ion of WWC- produce d conten t to non- resea rch e r s in WWC publications and on the WWC website . The contrac to r shall develop a memo that proposes the number of proposed sessions , frequency of sessions, specific objectives, and a roadm a p of topics to discuss to achieve those objectives.

In this memo, the contrac to r shall nomina te membe r s for the practi tione r working group. The proposed member s should have broad and diverse exper tise . The working group shall have between 6 and 9 individuals, with member s including curren t and former teache r s , school and district adminis t r a to r s , interm edia r ies , usability exper ts , and policymake r s . IES staff will also attend.

For each working group meeting, the contrac to r shall send an agenda and any mate rials at least one week in advance. The contrac to r shall include a summary of all calls and the progre ss of the working group in the monthly repor t .

At the end of task order , the contrac to r shall develop a summa ry of the practit ioner working group’s discussion and decision points. Upon COR approval, the summary shall be made 508 compliant and posted to the WWC website . The contrac to r is responsible for ensuring that the meeting mate rials are posted in a timely fashion through subtask 2.3.

Task 3. Proce d u r e s and Stan dard s Deve lop m e n t (Non- IT, Cost Plus Fixed Fee) The contrac to r shall convene a Statis tical, Technical, and Analysis Team (STAT) that will suppor t the reviews of the WWC. The STAT is composed of two sets of groups, small working groups that work on specific issues (see subtask 3.1) and the All-STAT which is a large group comprised of membe r s with diverse methodological exper t ise .

The task will suppor t considera t ion, by exper t methodologis t s familiar with the WWC, of issues related to WWC standa r ds for assessing the internal validity of findings from differen t types of impact study designs; WWC procedu r e s for charac t e r izing, presen t ing , and synthesizing study findings that meet WWC standa rd s ; and WWC reviewer guidance for applying curren t standa rd s and procedu re s .

Member s of the STAT will include employees of the contrac to r team and consultan t s to the TOAST contrac t . Other WWC contrac to r s and IES staff will be member s of the STAT and their time will be budge ted for under their respec tive contrac t s and salaries.

3.1. Propos e and Conven e STAT Small Workin g Group s

Within the first 3 months of contrac t award, the contrac to r shall develop a memo that proposes a series of small working groups with

PAGE 24 OF 102 91990023R00020001

specific purposes . STAT small working groups are expected to have a nar rowly defined mission and differing dura tions.