Responses to Vendor Questions from the RFI.pdf

Responses to Vendor Questions from the RFI

1. The PWS states that “within the first 18 months of the REL contract, each working group shall develop a brief, public-facing product such as a fact sheet or brief memo providing an overview of the topic that includes relevant REL and non-REL research and technical support products” (p.

35). Will a rubric or set of criteria be supplied for RELs to use in assessing which non-REL research and technical support products should be included? Or should RELs plan to develop their own set of criteria?

RELs can develop their own criteria, post-award in the product proposal, for identifying non-REL research and products to be included in the working group fact sheet/memo.

2. We appreciate the increased flexibility for partnerships under REL 2022. Does the SEA need to be involved in at least one of the partnerships within each state in the region?

As members of the REL Governing Board, State Education Agencies (SEAs) need to be aware of all REL projects. SEAs are not required to be directly involved as project partners.

3. PWS, page 17, partnership, see also subtask 3.3, page 41 Please provide clarification about the requirement that all partnerships be within a single state. While these types of partnerships are often deeper than multi-state, several examples of current REL partnerships exist that cross state lines and have active engagement across multiple states and are creating products that are useful in multiple states. Perhaps these are the exception, but they are impactful investments, consistent with the current PWS. Further, when multiple states or regions across states have a common high-leverage issue, there may be important opportunities to work with several communities across a few states to develop tools that are useful to a larger group than in one state. Is this required to be part of a community of practice, as described in subtask 3.3, as opposed to a partnership, which could be focused on building tools or products, or carrying out research in multiple districts in different states that use similar measures and have created similar policies (or use the same intervention/tool to improve practice)?

As stated in the PWS, “RELs shall work in partnership with stakeholder groups within a single state or jurisdiction. The work of REL partnerships should generally focus on statewide or local needs to increase the likelihood of achieving partners’ desired outcomes.” (page 17). As described on page 41, under Subtask 3.3, RELs may bring their partners together in a cross-state community of practice. Applied research, training, coaching, or technical support work may be conducted with a community of practice.

4. PWS, page 24, Does IES anticipate that RELs will collect data for their own measurement plan?

To what extent will RELs use IES-developed and OMB-approved surveys versus their own measures, which may need to go through OMB approval, to assess outcomes?

RELs will collect data for their partnership measurement plans. Page 40 and Footnote 22 of the PWS explain that RELs may use the Stakeholder Feedback Surveys (which are Office of Management and Budget (OMB)-approved customer feedback surveys) to measure progress toward their intended outcomes. RELs may also use other data sources, including extant data or data the REL collects outside of the Stakeholder Feedback Surveys. The type of data most appropriate for a measurement plan will depend on the types of outcomes the plan is designed to measure.

5. As part of Subtask 5.3, the “RELs must formatively evaluate the usability of the tool with its intended audience during development” (p. 53). We understand that the formative usability test is part of the development process and does not require OMB clearance. Please confirm if this understanding is accurate.

Page 74 of the RFI (Instructions for Offerors, under the instructions for Part 3A of the technical plan) states “If the usability testing requires it, offerors should plan to develop and submit an OMB package.”

See footnote 43 as well.

6. The PWS states that “proposals must include information about the individuals participating in the research, how they will be selected and recruited, whether they will be compensated…” (p. 75).

Are participant incentives going to be acceptable under REL 2022?

RELs may propose participant incentives in project proposals, which will be subject to Institute of Education Sciences (IES) approval.

7. Under subtask 5.3, each REL is required to develop at least one toolkit based on one or more WWC Practice Guide recommendations. Beyond this requirement, would IES consider allowing RELs to develop toolkits based on practice guides developed by other federal agencies (e.g., HHS) provided that the guides are based on the most rigorous evidence available, the same information is not in an existing WWC Practice Guides, and it serves a need in the region?

RELs may propose to develop tools based on practice guides developed by other federal agencies under Subtask 5.5. Toolkits developed under Subtask 5.3 must be based on the What Works Clearinghouse (WWC) Practice Guide assigned to that REL region. (See the response to question 46 for the updated Practice Guide assignments.)

8. Would IES consider allowing more than two Just-in-Time studies per year if each one addressed a stakeholder need that required a relatively quick turnaround to inform policy? Alternatively, could the JITs opportunities “roll over” each year so that a total of ten could be proposed by a single REL over the life of the contact without the two per year restriction.

To streamline the various proposal and report review timelines included in Appendix E in the Instructions to Offerors, IES has made the decision to discontinue Just-in-Time products and create a single review process for all research proposals and reports that is intended to get report findings into partners’ hands as quickly as possible. The process is as follows:

- Research proposals will be reviewed by the Contracting Officer’s Representative (COR), the REL Peer Review (RPR) contractor, and the National Center for Education Evaluation and Regional Assistance (NCEE) Associate Commissioner. With COR discretion, the proposal may be considered conditionally approved after one round of RPR review. The contractor may submit data requests and seek Institutional Review Board (IRB) approval upon conditional approval from the COR. The proposal may only receive final approval from the Associate Commissioner.

- Research Reports:

o Draft reports will follow this review and production process: REL COR review > RPR review > NCEE Associate Commissioner and Commissioner review > editing > Associate Commissioner review > final production and release.

When necessary to address stakeholders’ needs, and at COR discretion, the contractor may opt to release an early version of the report as a white paper after Associate Commissioner and Commissioner review. White papers serve as interim reports, which will be submitted to the Secretary’s Packet and then published on the REL website and indexed in ERIC. The REL may share the white paper once it is available on the REL website and has been submitted to

ERIC.

o All reports, whether there is an associated white paper or not, will continue the review and production process to be released as an IES report.

Additionally, RELs do not have a limit on the number of white papers produced each year. Decisions about white papers will be made with COR discretion and based on the needs of the stakeholders. Please see the revised version of Appendix E from the Instructions to Offerors – “Review Process Timelines for Research Proposals and Reports” for additional information.

9. PWS, subtask 5.5., page 56, If a REL proposes to develop web-based tools, is it accurate that the REL website contractor will cover costs of programming and web development, whereas the REL contractor is only responsible for directing, supporting, and validating the programming?

If a REL gets approval to develop a web-based tool, the REL Website contractor will cover the programming and web development costs up to a predetermined amount. The REL Website contract will include funds to cover a set amount of tool development costs for each REL. If a REL proposes to develop a tool (or tools) that will require more programming work than can be covered by the Website contract’s budget for that REL, the REL contractor will pay for the additional required programming costs using REL funds. The REL will also be responsible for the costs of developing the content of the tool, as well as their own costs related to collaborating with the Website contractor to determine the technical requirements and test the programming.

10. At the beginning of the Announcement (un-numbered page) under General Information, Classification, it states the NAICS Code for this solicitation is 541720. Later, on page 7, under Proposed NAICS Information, two NAICS Codes are listed, 541720 and 611710. Can two NAICS Codes be used for a solicitation? If not, which Code does the Department intend to use?

Both North American Industry Classification System (NAICS) codes are listed in order to gather information from vendors under both. The REL is the type of requirement that falls under both NAICS codes. As we move forward into the solicitation phase, the Department will decide which NAICS best suits the requirement or may use both to increase the pool of vendors. However, when an award is made, the contract will be entered under one NAICS code so we may accurately report in the Federal Procurement Data System.

11. Does IES have a template/preferred format for the QASP?

IES will provide a template for the Quality Assurance Surveillance Plan (QASP) with the RFP.

12. Page 7 contains a statement below, Proposed Product Service Code, concerning using the capability statements to evaluate set aside determinations, if any, for each region. It is understood that the REL Program as a whole will not be set aside. Will the Department give consideration, as they did in a REL competition several cycles past, to awarding extra points to qualified Small Businesses, under the NAICS Code(s), who bid a region as the Prime? Without such consideration, small businesses may always be relegated to a subcontractor role in the REL Program.

IES will automatically assign an “excellent” rating to the Small Business review criteria for any proposal with a prime contractor that is certified as a small business under the appropriate NAICS code(s) for this competition.

13. Page 7 of the RFI states that capability statements are limited to 7 pages. Are there any requirements around minimum font size and line spacing?

Capability statements must be seven pages double spaced, with 12-point font and standard (1 inch) margins.

14. Can the Department please clarify whether capability statements are 7 pages single or double-spaced?

See response to question #13.

15. Given the current COVID-19 crisis, and assuming its consequences through the period of this competition and into the performance of new REL contracts, can the Department elaborate on how this might affect the REL Program compared to previous cycles? Specifically, does the Department see an impact on the “rigorous” requirements and standards historically set by IES for REL research? What, if any, modifications to previously expressed standards, research design requirements or program requirements are under consideration as a result of the impact of

COVID?

IES has designed the RFI, including requirements for applied research under Task 5, with the assumption that there will be ongoing consequences due to the COVID-19 pandemic through the current period of competition and into the 2022 REL cycle. IES does not anticipate further changes to the requirements and standards for REL research or other program requirements due to the COVID-19 pandemic.

However, IES intends to continue to be as flexible as possible in response to conditions on the ground, as demonstrated through the current REL contractors’ rapid development of research-based resources in response to COVID-19.1 In the coming months, should the COVID-19 pandemic evolve in ways that necessitate changes to the REL competition or REL program requirements, IES will incorporate such changes into the RFP.

16. On page 17, the PWS states that all REL outcomes and targets must be attainable within the REL cycle’s current contract. If all outcomes and targets should be attainable by the end of the contract cycle, will the partnership be able to revise/refine outcomes during the contract?

Partnerships will be able to revise and refine logic models or equivalent, and corresponding measurement plans throughout the contract period as needed. This is discussed further under subtask 3.2 in the PWS.

17. On pages 25-26, the PWS states that long-term outcomes are in the distant future, years after the contract ends. Can you clarify which outcomes/targets must be met within the current cycle as described on page 17?

On page 17, the PWS states “all REL outcomes and their associated targets should be attainable within the REL cycle’s period of performance,” while on page 26 the PWS states “While acknowledging that partners’ long-term outcomes of interest may not be achievable within the timeline of the REL cycle, 1 See https://ies.ed.gov/ncee/edlabs/projects/covid-19/ for REL resources developed in response to the COVID-19 pandemic.

https://ies.ed.gov/ncee/edlabs/projects/covid-19/

RELs should identify targets for their projects’ long-term outcomes that are (1) consistent with the partner’s long-term outcome target but (2) attainable within the period of performance”.

Taken together, the implication is that long-term outcomes in partnership logic models, or equivalents, should be intended to be attained within the REL’s period of performance. They may represent a leading indicator or scaled-back version of partners’ longer-term goals beyond the REL contract period of performance, as described on page 26.

18. Can you describe the difference between “partners” and stakeholders if there is one? The RFI uses the terms more or less interchangeably--but occasionally hints that there might be a difference. For example, Task 3.1 p. 39 states that “for any partnering group for which more than one project across Task 4 and/or Task 5 is planned, or any group for which one Task 4 or Task project is planned that will last six months or longer, the REL shall submit a partnership proposal alongside the concept paper for the first project with the partnering group.” It later states that RELs can start with a single project then later evolve that into a partnership proposal. Does this mean that all projects serve stakeholders, but “partners” are those that meet the above criteria?

Thank you for pointing this out. The RFI intended to delineate a difference between stakeholders and partners. Partners are the subset of education stakeholders with whom RELs work directly to co-develop applied research and training, coaching, and technical support (TCTS) products and activities. As stated on page 16 of the RFI, all REL work shall be done in partnership. This is true whether a given project is part of a larger, related set of REL projects with the same set of partners as IES expects most, if not all REL work to be, or as a high-leverage, one-off project. REL projects should be focused primarily on supporting partners to achieve their intended outcomes. However, IES expects that in many cases, REL projects will also benefit education stakeholders at large. IES acknowledges that in several places, the language of the RFI should be more precise with respect to discussing partners versus stakeholders. IES is revising accordingly and will share these revisions through the RFP.

19. Can analyses of state or district data be performed by the REL or as part of a REL project under the TCTS task?

No. Data analysis in the 2022 REL cycle may only be performed under Task 5.

20. Can analyses of state or district data be performed by the REL or as part of a REL project under the TCTS task?

See response to question #19.

21. The PWS states that “training, coaching, and technical support project teams shall include content experts who have made meaningful contributions in a field relevant to the TCTS activity, either through authoring published research or practitioner guidance” (p. 45). In what capacity must the content experts be included in the project?

IES does not have specific requirements for the roles of content experts within TCTS project teams. IES expects TCTS proposals to include information on how the REL intends to utilize content expert(s) so that content expert(s) are best situated to share their expertise with partners and other project team members. Content experts may be employed by the prime contractor, a subcontractor, or be a paid consultant.

22. The PWS states that “in addition to the internal quality assurance procedures listed above, the REL shall propose an external subject matter expert (SME) for every TCTS project with a short-term goal of increasing stakeholder capacity to understand the extant research literature on a high leverage topic” (p. 45). Does the SME need to be external to the contracting organization that is awarded the REL or can the SME be an employee of the contracting organization but external to the specific project?

The SME may be internal or external to the contracting organization. However, in either case, the SME must be external to the specific project (meaning that he or she has no other role in that project beyond

SME).

23. For the newly conceived “Ask An Expert” subtask, will RELs still need to provide annotated bibliographies to the requestor?

No. Annotated bibliographies will not be required through the Ask an Expert subtask. Ask an Expert should not be considered a replacement for the Ask a REL subtask in the current REL contract cycle.2 Stakeholder requests for reference desk services similar to Ask a REL may be referred to the ERIC Helpdesk or National Library of Education.

24. Does the Department expect to establish an annual Task Order to cover various Ask An Expert services, or if not, how will Ask An Expert services be budgeted? As the need for unique expert engagement services and ad hoc meetings arises, will the total estimated cost plus fixed fee be adjusted to reflect the addition of previously unanticipated needs?

IES will not establish an annual Task Order to cover Ask an Expert services. RELs will be expected to implement and support this service as part of their annual budget.

After further consideration, IES will not require that contractors propose a set, not-to-exceed number of hours for this service as part of their Annual Plan, nor will there be any requirement to propose a set number of hours in their initial deliverable under this subtask discussed on pages 47-48 of the PWS.

Contractors will only be required to propose a budget for this subtask as part of each Annual Plan.

IES encourages RELs to make conservative assumptions for the budget and scope of work under this Subtask so that they can address key stakeholder needs. IES recommends that offerors assume 1-2 requests per jurisdiction of varying intensity when budgeting this subtask for the first 18 months of the contract. Additionally, IES expects that Ask an Expert services will not take the place of TCTS under Subtask 4.1; Ask an Expert services are intended to be less intensive than TCTS under Subtask 4.1 and be quick turnaround.

Please keep in mind that RELs may propose other REL services to support requestors in place of Ask an Expert or refer requestors to other federal service providers as needed.

25. PWS, pages 46-47, Ask an Expert, are there parameters on length of activities or IES-defined/preferred formats for products resulting from this service?

Page 46 PWS includes several types of requests for which Ask an Expert is not an appropriate vehicle to respond. This includes requests that would require more than two meetings with the requestor to fulfill.

As noted in the response to Question 24, there are no longer any restrictions on the number of hours a REL contractor may take the fulfill a request.

2 See https://ies.ed.gov/ncee/edlabs/askarel/index.asp for more information on the Ask a REL service for the current REL cycle.

https://ies.ed.gov/ncee/edlabs/askarel/index.asp

The only products that may be developed through this service are meeting materials, if needed, and a memo summarizing each Ask an Expert meeting. RELs may not produce materials typically developed under other Tasks such as applied research (Task 5) or dissemination (Task 6). Please see the response to question 24 for additional information on the scope of work under this Subtask.

26. PWS, subtask 3.1, page 40, IES states, “IES also expects the process of developing a logic model and measurement plan with partners to be brief – measured in weeks, not months – in order to maximize the time stakeholders have available to complete the work.” Does this indicate that TCTS projects cannot focus on logic model development as one step towards a longer-term research and TCTS agenda if partners indicate this as a priority?

The development of a partnership-level logic model (or equivalent) should always be a priority that helps focus the longer-term research agenda of work done in partnership between RELs and their partners. Such activities should be supported under subtask 3.1 and may not be considered a separate TCTS project.

If REL partners indicate that the development and refinement of a partnership-level logic model may require a series of conversations over a longer time for any reason, such as partners’ lack of familiarity with logic models (or equivalents), or staff turnover within a partner’s organization, it is acceptable to propose an initial logic model under subtask 3.1 and support additional meetings under subtask 3.2 to refine the initial logic model or equivalent.

REL contractors may also support the development of a logic models not at the partnership level under Subtask 4.1. For example, RELs may support partners in developing a logic model to guide the design of an intervention or to focus a partner’s research agenda outside of their work with the REL so long as the work meets the criteria of high-leverage as defined in the PWS.

27. PWS, tasks 4 and 5, pages 45 and 57, does IES expect the role of SMEs to be the same as the role of TWGs in the current contract?

Yes. The role of the SME as outlined in the PWS overlaps with the role of a Technical Working Group (TWG) in the current contract. IES will not require RELs to use Technical Working Groups in the 2022 contract cycle.

28. ED is accepting solicitation questions until February 16, 2021. How far in advance of this date does IES anticipate that a draft of the solicitation will be released based on the August 26, 2020 RFI response?

We anticipate releasing the final RFP on February 1, 2021. IES will not be releasing a draft RFP prior to the release of the final RFP on February 1, 2021.

29. Is the RFI directed towards organizations planning to prime an REL proposal?

The RFI is open to the entire vendor community and all interested parties are invited to submit a response.

30. Are organizations seeking a subcontract role on a proposal expected to submit an individual formal RFI response?

The RFI is open to the entire vendor community and all interested parties are invited to submit a response.

In addition, responding to the RFI is optional and vendors do not need to respond to submit a proposal.

31. Are organizations seeking a subcontract role expected to submit a response in combination with their prime?

Responding to the RFI is optional and vendors do not need to respond to submit a proposal.

32. Will each REL continue to have a dedicated REL website for its region? If so, will each REL be able to create new pages to adapt to user needs?

Each REL will have a website managed by an external website contractor. The REL is responsible for the content of the website, but the external website contractor will oversee the layout and design of the website.

33. Will REL staff with appropriate security clearance levels have access to the IES member site and development server?

REL staff will not be responsible for developing the website and will not need access to the IES Members Site.

34. What, if any, are the ED data security requirements that the offeror should be aware of?

Please see the “Updated Guidance on Security Clearances” document.

35. Would IES consider allowing 10-point font, single-spaced for all tables, graphics, and footnotes in the technical volume?

Yes, offerors may use 10-point font, single-spacing for all tables, graphics, and footnotes in the technical volume.

36. Are named staff other than the Director and Deputy Director required to be employees of the prime bidder?

The Director and Deputy Director are the only key staff that are required to be employees of the prime offeror. There is no requirement for all other key and named staff to be employees of the prime offeror.

37. It was our understanding that the REL program contract is required to be at least 50% firm fixed price. However, it appears that the REL 2022 contract will be cost plus fixed fee. Will there be any firm fixed price elements of the REL 2022 contract?

The REL 2022 cycle will use a cost-plus fixed fee cost structure. There will be no firm fixed price component.

38. Under Task 3A on p. 74 there is language that "the government retains the right to make changes to this subtask after award. Within one month of the contract start date the Department will assign the Practice Guide toolkit topics to each REL." Does this mean that the toolkit topic could change from what is proposed? How will the Department determine such topic changes?

There is a small chance IES will need to work with offerors and their partners to select a different WWC Practice Guide for the toolkit. IES reserves the right to allow this flexibility if a situation arises that warrants it.

39. In the technical proposal, are the key and named personnel, management plan and timelines, and organizational capabilities, experience, and resources sections included in the 150 page limit?

The descriptions of the requirements for these sections do not explicitly state that they do not count toward the page limit, yet they are not included in the guidelines for how many pages to devote to each section on page 67.

The key and named personnel, management plan and timelines, and organizational capabilities, experience, and resources sections are not included in the 150-page limit. The page limits for those sections are as follows:

- key and named personnel: 12 pages

- management plan and timelines: 5 pages

- Security Clearance Risk Levels by Position: no page limit

- organizational capabilities, experience, and resources: 3 pages

These page limits are requirements, not recommendations.

40. Please confirm that the required projects in part 2A are one Task 4 project and [one subtask 5.1 (Conduct original applied research and report findings OR 5.2 (Conduct Just-In-Time Research)].

That is, one Task 4 and one Task 5 project, selecting from subtask 5.1 and 5.2.

Part 2A in the Instructions to Offerors outlines the requirement for offerors to submit plans for two partnerships. These partnerships have slightly different requirements as it relates to concept papers:

• Partnership #1 must include one Task 4 project and one Subtask 5.1 project, for a total of two concept papers. IES has eliminated Subtask 5.2 (Just-in-Time products); therefore, the offeror may only propose a Subtask 5.1 project (see IES response to question #8).

• Partnership #2 may include two Task 4 projects OR one Task 4 project and one Subtask 5.1 project, for a total of two concept papers.

41. On p. 71 of the RFI, Part 2 of the Technical Approach, it states: “The first set of projects developed in partnership with stakeholders must include one Task 4 project and one Subtask 5.1 (Conduct original applied research and report findings) or 5.2 (Conduct Just-in-Time Research) project within the first 18 months. The second partnership may plan to execute two Task 4 projects or at least one Task 4 and one Task 5 (either Subtask 5.1 or Subtask 5.2) project within the first 18 months.” The first sentence is being interpreted to mean that the first set of projects developed in partnership with stakeholders of the first partnership must include one Task 4 project and one Subtask 5.1 (Conduct original applied research and report findings) or 5.2 (Conduct Just-in-Time Research) project within the first 18 months. Please confirm or correct our understanding.

Please see response to question #40.

42. In several places in the instructions to offerors, IES indicates, "Proposals or concept papers that have been previously reviewed by IES may not be submitted." Does this statement apply to proposals or concept papers in the same task only (e.g., proposals submitted under the current tool subtask (5.3) cannot be submitted in response to the new subtask 5.3)? For example, if an existing set of training activities (5.1) have an IES-approved proposal that have potential to be incorporated into a tool, can a vendor propose to do so?

This requirement applies to proposals or concept papers in the same task and to all REL work. RELs must propose new work that addresses the current needs of their stakeholders. Incumbent offerors cannot propose to duplicate work they have already proposed or completed under a previous REL contract.

However, incumbent offerors may build on their previous work to develop new project proposals.

43. Page 64 of the RFI lists the five states that currently comprise the Northwest Region, but the chart on page 71 indicates that there are only four jurisdictions in the Northwest. Can you please confirm that the number of jurisdictions cited on page 71 is a typo?

IES is confirming that there was a typo on page 71 in Table 1 of the Instructions to Offerors. There are five jurisdictions in the Northwest region.

44. Would IES consider providing cost estimates for each task? The RFI indicates that Tasks 3–6 may comprise up to 80% of the yearly budget. Having the cost estimate for each task would help applicants produce plans that are better aligned with the priorities of IES.

IES will not provide cost estimates for each task because the proposed work should be aligned with the priorities of the region. Task 1 and Task 2 shall comprise up to 20% of the yearly budget. Tasks 3-6 may comprise up to 80% of the yearly budget and should include a mix a of projects that meet the region’s needs.

45. Can you provide a sample timeline for the submission, review and approval process for the 5.3 concept paper, proposal, toolkit and associated 5.1 efficacy study proposal with OMB clearance that would result in a complete evaluation report by 9/2025?

No, IES will not provide a sample timeline for the submission, review, and approval process for a 5.3 toolkit and associated 5.1 efficacy study because the timeline depends on the offerors’ plans to develop and evaluate it. In developing the REL 2022 procurement materials, IES developed a timeline for toolkit development and evaluation, and has determined that both activities are feasible within the 5-year contract. However, IES expects offerors to work with their partners and develop plans for toolkit development and evaluation that may differ from the sample timeline IES created because the offeror has incorporated accommodations for their partners or other innovations. Offerors’ timelines might and should look different across proposal submissions.

To clarify, concept papers are not required for the REL toolkit or the toolkit evaluation. For these two projects, the successful offeror will begin the review process with the proposals submitted with their REL contract proposal. Post-award, the successful offeror’s toolkit proposal will be reviewed by the Subject Matter Expert and then by IES. The toolkit proposal will not be reviewed by RPR because the WWC practice guide is an IES peer-reviewed publication.

The successful offeror may resubmit a revised evaluation proposal as soon as they are ready to do so, along with any accompanying OMB clearance package that is required. The evaluation proposal and OMB package may be reviewed while the contractor’s toolkit is in development, in the review process, or in the pilot and usability testing phase. Offerors should refer to the revised version of Appendix E (see response to Question #8) for the review timeline for research proposals and reports.

46. The PWS specifies designing a toolkit based on a WWC practice guide that aligns to specific stakeholder need. Does this mean that different RELs could end up developing different toolkits on the same practice guide?

IES has decided to assign each region one WWC Practice Guide instead of providing three options per region. Now that each region has one option, RELs will not develop toolkits on the same Practice Guide.

See the table below for WWC Practice Guide assignments.

* IES anticipates that the forthcoming WWC Practice Guide Assisting Students Struggling with Mathematics:

Intervention in Grades K-6 will be published prior to the Request for Proposals (RFP) for the Regional Educational Lab Program. In the event that this Practice Guide is not published prior to the RFP, IES has provided an alternate option (Assisting Students Struggling with Mathematics: Response to Intervention for Elementary and Middle Schools) from which offerors for REL Northeast & Islands may base a toolkit. If the Assisting Students Struggling with Mathematics toolkit is published prior to the RFP as planned, offerors in the Northeast & Islands region may not use the alternate option.

47. Can IES provide an example timeline of activities that include toolkit development that is feasible within a 5-year window and includes all IES and OMB approvals? We note that OMB approvals are likely required during development to collect data for the formative evaluation that "is a component of the toolkit development phase under Subtask 5.3" and will be required during the efficacy and implementation evaluation. We note that IES grant programs, which do not require OMB approval or the various reviews required in the REL program, do not include development plus implementation plus efficacy studies in a 5-year window. EIR development grants include these expectations. However, these grants do not include IES or OMB review and often require no-cost extensions to complete.

Please see the response to question #45.

48. In a couple of places in the PWS, TCTS activities are mentioned as being able to occur after an applied research project is complete. However, it has been our experience that research reports generally come out quite late in the contract and we are not able to release findings to a limited stakeholder group until at least one round of RPR review that doesn’t result in significant concerns.

REL Region WWC Practice Guide Assignment

Appalachia Teaching Math to Young Children

Central Teaching Strategies for Improving Algebra Knowledge in Middle and High School Students

Mid-Atlantic Teaching Elementary School Students to Be Effective Writers

Midwest Developing Effective Fractions Instruction for Kindergarten through 8th Grade

Northeast & Islands Assisting Students Struggling with Mathematics: Intervention in Grades K-6* OR Assisting Students Struggling with Mathematics: Response to Intervention (RtI) for Elementary and Middle Schools

Northwest Using Technology to Support Postsecondary Student Learning

Pacific Teaching Secondary Students to Write Effectively

Southeast Assisting Students Struggling with Reading: Response to Intervention (RtI) and Multi-Tier Intervention in the Primary Grades

West Improving Reading Comprehension in Kindergarten Through 3rd Grade https://ies.ed.gov/ncee/wwc/PracticeGuide/18 https://ies.ed.gov/ncee/wwc/PracticeGuide/20 https://ies.ed.gov/ncee/wwc/PracticeGuide/17 https://ies.ed.gov/ncee/wwc/PracticeGuide/15 https://ies.ed.gov/ncee/wwc/PracticeGuide/2 https://ies.ed.gov/ncee/wwc/PracticeGuide/2 https://ies.ed.gov/ncee/wwc/PracticeGuide/2 https://ies.ed.gov/ncee/wwc/PracticeGuide/25 https://ies.ed.gov/ncee/wwc/PracticeGuide/22 https://ies.ed.gov/ncee/wwc/PracticeGuide/3 https://ies.ed.gov/ncee/wwc/PracticeGuide/3 https://ies.ed.gov/ncee/wwc/PracticeGuide/14

Are RELs required to propose TCTS activities at the conclusion of any of the applied research projects?

RELs should propose a TCTS project after the release of an applied research project if the TCTS project is necessary to help partners meet their medium-term outcomes. If a TCTS project is necessary to support stakeholders’ use of the findings, RELs should plan projects that allow enough time to include the TCTS project. At minimum, RELs should plan research projects that will be released with enough time to conduct a briefing with appropriate stakeholders to support their use of the findings, which is a requirement of applied research projects under Subtask 5.1.

49. PWS, subtask 1.10, page 32, How many people, and who, does IES expect to participate in the two-day site visit from the REL evaluation in years 2 and 3?

We anticipate that three REL staff will participate in the two-day site visit and will include both the Director and Deputy Director as well as one additional person to be determined in year one.

50. PWS, page 30, governing board, What is the rationale behind having the Governing Board approve new members? This has the potential to limit the diversity of viewpoints in the group.

The process of approving new Governing Board members is a multi-stage process which seeks to 1) value the role of the Governing Board as an independent entity that helps advance the mission of the REL program and 2) ensure diversity of expertise and experience. The REL proposes each new Governing Board member to IES and provides a justification explaining how the potential Governing Board member meets the Education Sciences Reform Act of 2002 (ESRA) requirements and is well situated to further the work of the REL. After IES reviews the justification, the new member can be approved by the Governing Board.

51. The PWS states that “the composition of the REL Governing Board shall represent the interests of all states and other appropriate constituencies in the region” (p. 29). Please clarify what it means to “represent the interests of states and other appropriate constituencies.” How are these concepts operationalized/defined?

Please refer to Section 174(h) of ESRA for a list of stakeholders who must be invited to serve on the REL Governing Board. IES does not define “appropriate constituencies,” however, all proposed members must be well situated to further the work of the REL, and therefore may include stakeholders such as education researchers, teachers, and individuals representing the interests of learners in economically disadvantaged areas, both rural and urban, and other underserved learners.

52. PWS, subtask 2.1, pages 34-35, Can IES provide information on the year in which they expect RELs to present cross-REL activities to program offices from their leadership topic, for budgeting purposes?

This subtask will be exercised at the discretion of IES. IES does not anticipate presentations to occur within the first 18 months of the contract.

53. In reference to collaboration with Comprehensive Centers described on page 36, the PWS states that RELs and CCs must establish regularly occurring meetings. Can you please clarify how often meetings will be required?

Meetings shall occur at least quarterly.

54. On page 19, the RFP states "each REL shall develop at least one toolkit that supports scaling of WWC Practice Guide recommendations." On page 19, the RFP also states that, for the development of the toolkit, each REL is to "Identify a WWC Practice Guide that aligns to a specific stakeholder need." However, the footnote on page 19 and the text on page 100 indicate that each REL must choose between one of 3 selected practice guides as the basis for their toolkit. If a different high leverage challenge is identified in the region, can a different Practice Guide be proposed?

No. Each REL must develop a toolkit based on the Practice Guide assigned to the region.

55. Subtask 5.3, page 51 (and elsewhere in the PWS), When developing tools to implement scientifically valid research from WWC practice guides and other IES investments, to what extent will RELs carry out evidence reviews to update the evidence base? Some of the practice guides are quite old (e.g., Assisting Students Struggling with Reading: Response to Intervention (RtI) and Multi-Tier Intervention in the Primary Grades was published in 2009) and even some of the newer guides (e.g., published in 2018) will be out of date by the time the contract starts in 2022.

Carrying out evidence reviews under subtask 5.3 is not permitted as it is considered outside the scope of the REL program.

56. Does IES plan to expand its dissemination channels beyond twitter, YouTube and email (i.e., IES Newsflashes)?

No, not at this time.

57. PWS, page 11, Is IES’ definition of evidence-based consistent with ESSA, including all ESSA evidence levels?

Tiers 1, 2 and 3 of the Every Student Succeeds Act (ESSA) are generally the evidence levels to which the PWS is referring when the term “evidence-based” is used. However, IES expects the RELs to look for, share and work with the highest tier of evidence that is available. In addition, if only evidence at the Tier 4 level is available, then for some projects, this will be acceptable.

58. PWS page 27, long-term outcomes, does IES consider the “early career outcomes” to apply to teachers as well as other types of careers (e.g., broader outcomes following postsecondary education and training programs)?

Yes. Examples of “improved early career outcomes” for non-teachers are improved employment, earnings, and career-ladder or career-advancement. Examples of early career outcomes for teachers are teacher retention, measures of teacher effectiveness and measures of teacher career-ladder advancement where such ladders exist.