PRESTO Live Industry Day_Audio Transcript 4-21-2022.pdf

Procurement of Research Evaluation and Statistics Task Orders (PRESTO) Live Industry Day

– Audio Transcript (4/21/2022)

Slide 0 Cover Slide:

[Speaker: Alean Miller]

Thank you for joining us for the Procurement of Research Evaluation and Statistics Task Orders, called PRESTO, industry day. This industry day is part of a series of industry engagement activities that IES is conducting as part of market research. To stay current on PRESTO and future industry engagement activities, please continue to visit SAM.gov – the Notice ID is provided on-screen [Notice ID: IESE220353AP].

Please also bear in mind the PRESTO draft SOW which can be accessed on SAM.gov, remains pending the conclusion of ALL market research activities.

Slide 1 Agenda:

We appreciate you joining us today as we cover the following topics:

• Opening remarks to formally kickoff today’s industry day event;

• Background on IES and the PRESTO vehicle;

• More information on the PRESTO scope categories, including examples of the type of work that might be supported in each;

• A review of questions received by industry via the web form provided on SAM.gov; and

• Finally, an overview of the award schedule and upcoming industry engagement activities.

Today’s content is intended to build on the information previously provided in a pre-recorded industry day presentation that is accessible on SAM.gov. While IES intends to provide some examples of potential work in each scope category, the list is not intended to be exhaustive or representative of every type of work that might be supported by PRESTO.

Similarly, IES intends to address some common questions submitted by industry partners, but the Government does so at their sole discretion and may not address every question submitted thus far.

Slide 2 Welcome:

With that said, thank you all again for joining us. For the purposes of today’s industry day presentation. All participants have been muted and the meeting chat will remain closed for the duration of today’s event. As such, the Q&A facilitation will address only those questions previously submitted by the April 15th deadline via the Industry Day Q&A web form, posted on SAM.gov – there will be NO live Q&A throughout the course of today’s presentation. We greatly appreciate all those who previously submitted questions via the Industry Day Q&A web form which is posted on SAM.gov.

Additionally, the Industry Day Q&A web form will be re-opened for additional questions which may be incorporated in the next Small Business Industry Day event which is scheduled for April 26 – please visit SAM.gov to register for the small business event if you are interested and have not already.

Now, I would like to present today’s speakers. My name is Alean Miller and I am both the Contracting Officer Representative, or COR, on PRESTO, as well as a Senior Contracts Manager at IES. I am joined today by Elise Christopher, Study Director of Longitudinal Surveys, Stephen Provasnik, Deputy Commissioner of NCES, and Endrias Leuleseged, Contracting Officer.

Slide 3 Background:

[Speaker: Alean Miller]

Thank you, Calvin. The Institute of Education Sciences, or IES, is the statistics, research, and evaluation arm of the U.S. Department of Education. IES is independent and non-partisan, with a mission to provide scientific evidence on which to ground education practice and policy and to share this information in formats that are useful and accessible to educators, parents, policymakers, researchers, and the public. For more information about IES, please visit the site shown on the screen [https://www.ies.ed.gov/].

IES is preparing to release the PRESTO contract vehicle. PRESTO shall be a multiple award, Indefinite Delivery/Indefinite Quantity (ID/IQ) contract vehicle with an initial five-year period of performance and optional award up to five additional years.

PRESTO is being issued to meet the following goals:

• Recompete and expand previous ESSIN contract requirements to support IES’ mission,

• Provide a vehicle that supports IES, across all its Programs and Centers, as well as the larger Department of Education as applicable, and

• Provide access to a larger pool of qualified vendors and increased competition.

At this time, I will hand the presentation over to Stephen Provasnik, who will provide a brief overview of the PRESTO scope categories before beginning to walk through some examples of work in each category.

Slide 4 Scope Categories:

[Speaker: Stephen Provasnik]

Thanks, Alean, and thank you to all who have joined us for information on the PRESTO contract vehicle today.

PRESTO is made up of five distinct scope categories with associated NAICS codes assigned to each category. For your reference, the proposed NAICS codes can be found in the appendix of this presentation.

This slide provides an overview of the five categories, which are (1) Education Sciences Support Activities, (2) Assessment Development Support, (3) Communications and Outreach, (4) Recruitment, and (5) IES Operational Support.

Note that vendors may be awarded on more than one category. Additionally, task orders will be issued with a primary scope category; however, a nominal amount of scope may be included from another scope category.

The next slide is intended to highlight where small business has been taken into consideration, starting with the acquisition strategy and PRESTO contract vehicle design.

Slide 5 Scope Categories (Cont.):

Based on the results of our market research, IES anticipates that Categories 1 through 4 will contain both a “restricted” and “unrestricted” pool in each. The requirements are the same in both pools, but there will be separate thresholds for evaluation at the ID/IQ level and the restricted pool is limited to small business offerors only.

Category 5 will be a small business set-aside, accessible only to small business vendors.

Task orders will be issued subsequent to the ID/IQ vehicle award, and additional market research will be required for each task order request to determine whether the work can be set-aside for small business vendors in the restricted pool.

Please note, today’s presentation does NOT include details related to the evaluation criteria for the vehicle awards– this type of information and more instructions to Offerors will be provided in Section L of the Request for Proposal or RFP.

Slide 6 Scope Categories (Cont.):

The updated scope categories and their requirements (and sub-requirements) can be seen on this slide. Scope categories and the draft SOW remain subject to change throughout the course of PRESTO market research; however, the Government anticipates (as I explained in the previous slide) having both a restricted and unrestricted pool in Categories 1 through 4 and reserving Category 5 as a small business set-aside.

Industry partners already familiar with the draft SOW may note that some of the initial Category 5 requirements have shifted or been re-distributed across Categories 1-4. To re-iterate, these changes are tentative as of today and are not reflected in the draft SOW currently available on SAM.gov.

The next set of slides provides an overview of each scope category. We will present some examples for each category in order to spotlight the type of potential work that could be supported by vendors awarded task orders through PRESTO. I want to stress that these are only illustrative examples, but we have provided them to benefit industry partners who previously requested further clarification of the scope category requirements.

Slide 7 Category 1:

Category 1, “Education Sciences and Support Activities,” scope category 1 is comprised of the work or activities required to conduct surveys and evaluations that collect data and produce data products.

Category 1 is the largest of the PRESTO scope categories, and we intend to provide examples of work that align with the highlighted requirements on the following slides.

These examples were provided by IES subject matter experts across its Programs and Centers, and, while not an exhaustive list, they are intended to give you an idea for the type of work PRESTO is currently scoped to support in Category 1.

Slide 8 1.1.3 Data Processing, Management, and Data Creation:

Requirement 1.1.3, “Data Processing, Management, and Data Creation,” includes expert support managing and converting data gathered during a study or in an evaluation collection process, into organized data in fully documented data files, ready for secondary analyses by the government and the public.

An example of this type of work includes the data monitoring of collection activities in the National Center for Education Statistics, or NCES. NCES has a need to monitor the data collection activities of its various national studies.

Additionally, there is a need to oversee the transfer of data between the data collection systems, data editing systems, and data center.

This type of work requires the contractor’s understanding and familiarity with statistical software, like SAS; databases, like SQL; and data analytics. Such activities require the contractor to prepare and maintain code with proper comments, archiving procedures, and other quality management processes.

Slide 9 1.1.4 Data Management and Information System Management:

Requirement 1.1.4, “Data Management and Information System Management” includes, but is not limited to, data processing, method recommendation, and data testing. To demonstrate this work, here is an example from the Integrated Postsecondary Education Data System, or IPEDS, and its work in regularly updating and archiving data in the Data Center.

This type of work requires further knowledge of data structures, SAS, data tools and databases. A contractor might need to support the reformatting of data files and the creation of metadata tables for reports, variables, legislative information, and data collection information.

Slide 10 1.2.1 Internal Expert Reviewing Activities:

Requirement 1.2.1, “Internal Expert Reviewing Activities,” includes the provision of expert technical support for consultations on and review of a wide range of technical materials that go into the development and operations of surveys, assessments, and evaluations.

An example of this type of work is provided in the review of the Early Childhood Longitudinal Study, or ECLS, national kindergarten and first-grade data collection instruments. In this case, the instrument content and format will be largely based on the instruments used in the field test, and data collection instruments will be developed outside of the contractor’s work. A professional review of data collection instruments might include:

• Cross-checking included items against those fielded in other ECLS collections,

• Evaluating the effectiveness of the proposed instrumentation for collecting the intended data,

• Determination of whether the instruments can be completed in the anticipated timeframes for completion, and

• Assessing how seamlessly instruments interface with other instruments and databases central.

Slide 11 1.2.6 Program-Based and Methodology Innovations:

Requirement 1.2.6, “Program-Based and Methodology Innovations,” includes the development of innovative approaches, technologies, and procedures related to general education science support.

In this example, the contractor would assist National Center for Education Evaluation, or NCEE, with the development of an innovative guide for questionnaire and survey items for the purposes of improving evaluations of policy implementation. This guide would draw on best practices for item development, but specifically as applied toward understanding the extent to which state or local officials employ key levers to achieve the objectives of federal legislation. The guide could include suggested procedures for working from high-level research questions to outlining survey topics and then survey items, illustrative examples that use those procedures, and any checklists or tools that help users follow the procedures.

Slide 12 1.2.8 Support for External/Internal Evaluation Activities:

Requirement 1.2.8, “Support for External/Internal Evaluation Activities,” this requirement includes the analytics support for all external and internal evaluation activities, as well as research and development, to include evidence reviews, evaluation feasibility studies, design studies, and evaluation preparation.

In this example, the contractor would assist National Center for Education Evaluation, or NCEE, for an evidence review to answer a question about the effectiveness of a particular program, practice, instructional approach, or policy.

We refer to this here as an “intervention.”

The contractor would be required to undertake the following steps in order to produce a What Works Clearinghouse (WWC) Intervention Report (IR) that is based on the evidence review, which would require:

• Undertake a systematic, search for the causal research on a particular intervention,

• Review all studies that are identified through the search against the latest WWC standards,

• Use the latest WWC procedures to conduct a fixed-effects meta-analysis of the studies that meet WWC standards to estimate the impact of that intervention on outcomes of interest,

• Draft a WWC IR based on the meta-analysis, and

• Revise the IR based on IES review and peer review and produce a final IR for publication.

Slide 13 1.3.3 Rapid Response Surveys:

Requirement 1.3.3, “Rapid Response Surveys,” includes support to prepare and conduct rapid response surveys, from the point of identifying and defining a data need, to completing a complete draft of a first look report and dataset.

Examples of this type of work might be time sensitive, where the Government requires support from a contractor to prepare and conduct a rapid response survey in only a matter of months. The contractor would provide support from the point of identifying a data need to a complete draft of a first look report and data https://ies.ed.gov/ncee/wwc/ set. In general, the surveys should take approximately 3 months or less, excluding government clearance time requirements.

Slide 14 Category 2:

Category 2, Assessment Development Support, this scope category is comprised of the development of assessment components for sample survey studies and large-scale assessments.

Once again, we plan to walk through examples of work that align with the requirements you see highlighted here. These examples were provided by IES subject matter experts across our Programs and Centers. While not an exhaustive list, they are intended to give you an idea for the type of work PRESTO is scoped to support in Category 2.

Slide 15 2.3.1 Item Development: Non-Cognitive Items:

Requirement 2.3.1, “Item Development: Non-Cognitive Items,” this requirement includes the development, identification, documentation of sources, and evaluation of non-cognitive items to be used in non-assessment data collections.

An example of this type of work might include reviewing recommendations from a content review panel (e.g., for content and data collection approaches, non-cognitive items that exist for the topic appropriate for the expected respondent type, and data collection mode), finding or developing non-cognitive items, documenting item sources, including information about the properties of the non-cognitive item and justification for its selection for the government study, and developing items for a variety of respondents (e.g., students, teachers, and parents) who represent diverse respondent groups.

Slide 16 2.4.1 Psychometric Analysis: Cognitive/Non-Cognitive Items:

Requirement 2.4.1, “Psychometric Analysis: Cognitive/Non-Cognitive Items,” this requirement includes the evaluation of the effectiveness of cognitive/non-cognitive survey items and their interoperability with each other against baseline expectations.

An example of this type of work might pertain to a psychometric report which the Government requires support to develop. The psychometric report could address whether the psychometric objectives of the assessment battery were “not met,” “met,” or “exceeded,” and could include an in-depth description of the types of scores reported and included in the data file. A document outline might be provided by the Government with minimum technical reporting sections as shown on screen.

Slide 17 2.4.2 Psychometric Analysis: Process Data:

Requirement 2.4.2, “Psychometric Analysis: Process Data,” this requirement includes the evaluation of process data, metadata, and paradata.

This type of work might require research using information contained in the log files collected by the computer system while students take assessments or tests on computers. This work must augment students' scores and provide additional information on students' interactions with the assessment and testing environment.

Slide 18 2.9 Cognitive Labs and Focus Groups:

Requirement 2.9, “Cognitive Labs and Focus Groups,” this requirement includes the provision of support in the piloting or testing of new survey and assessment items, especially open-ended or “constructed response” cognitive items, as well as new data collection formats or modes.

An example of this type of work would be cognitive laboratory studies for item development, wherein a contractor must design and implement a cognitive laboratory research study to support development of examples like the National Teacher and Principal Survey (NTPS), Teacher Follow-up Survey (TFS) and Principal Follow-up Survey (PFS) items and/or usability testing. The cognitive laboratory research shall be directed towards an understanding of:

1. The extent to which the items are understood by respondents and how their answers might be interpreted,

2. The validity of the items, and/or

3. The extent to which the formatting and flow of the items is usable and understood by respondents.

The research may focus on items that have not been fielded before within the context of a larger questionnaire or may focus on modifications to previously fielded items. This research will seek to identify any areas of ambiguity and confusion on the part of respondents.

Those are the examples from Category 2. Now, I will hand the presentation over to Elise Christopher who will walk us through an overview and some examples for Categories 3 through 5, Elise.

Slide 19 Category 3:

[Speaker: Elise Christopher]

Thank you, Stephen. In Category 3, IES requires vendors with experience in communications and public relations for the purpose of supporting the IES Communications Management Office (CMO) in the areas of communications planning and management, and external and internal communications. Vendors will be asked to provide public relations consultation, communication strategic planning, product promotion, and campaign development to reach target audience groups.

For Communications Planning and Management (Requirement 3.1), IES requires support for the day-to-day functions of a new Communications Management Office (CMO). This work may include, but is not limited to:

• Audience research and targeting

• Brand development and analysis

• Communications tracking

• Drafting and updating CMO policies procedures, and guidelines as needed

For External and Internal Communications (Requirement 3.2), IES requires support to proactively identify ways to promote IES products through communications campaigns, as well as to track the success of those campaigns for future communications planning. This work may include, but is not limited to:

• Public and media relations support

• Providing editorial feedback on IES products

• Multimedia production

Slide 20 Category 4:

Category 4, Recruitment, is comprised of expert support to recruit educational entities and groups traditionally underrepresented in education research.

Once again, we plan to walkthrough examples of work that align with the requirements you see highlighted here. These examples were provided by IES subject matter experts across our Programs and Centers. While not an exhaustive list, they are intended to give you an idea for the type of work PRESTO is scoped to support in Category 4.

Slide 21 4.1 Recruitment for Education Sciences Activities:

Requirement 4.1, “Recruitment for Education Sciences Activities,” includes assistance in the identification and recruitment of participants for IES-sponsored programs, surveys, and other education data collection activities. Recruitment targets include postsecondary institutions, professional organizations, schools, State and Local education agencies, and other public or private enterprises to identify and recruit study participants to respond to surveys, participate in evaluations, and participate in other data collection activities concerning topics related to educational settings, as requested by individual IES Centers and divisions.

An example of this type of work has been provided for the National Indian Education Study or NIES. NIES support services are designed to meet participation challenges so that selected schools participate in the study. Since 2007, all Bureau of Indian Education (BIE) schools serving students in grades 4 and/or 8 have been included in the NIES sample. The cooperation of schools and students must be obtained following the timeline set by the Sampling & Data Collection contractor.

The timely communication to all selected schools and their education agencies is critical to the success of the study. Focused recruitment of BIE schools for 2024 does not begin until 2023. However, the contractor would be expected to share information related to the study in the years prior to the recruitment period with a variety of American Indian / Alaska Native constituencies (including BIE schools) to help maximize awareness of the study and to better facilitate 2024 study participation leading up to recruitment period.

Slide 22 4.2 State Coordinators Recruitment Center:

Requirement 4.2, “State Coordinators Recruitment Center,” includes support to maintain and manage a national recruitment center to work with applicable stakeholders at the state and local level, as well as private organizations.

An example of this type of work has been provided for the National Recruitment Center. This support entails maintaining and managing a national recruitment center to work in every state with schools, state and local education agencies, and other private organizations. Recruitment targets are study participants to respond to research surveys, participate in program evaluations, and participate in other data collection activities.

Slide 23 4.4 DEIA Initiative Support:

Requirement 4.4, Diversity, Equity, Inclusion and Accessibility, or DEIA, includes support for DEIA initiatives, to include training and professional development, strategic planning, data collection and analysis, grants management support, administrative and logistical support, and outreach and communications support.

IES’s ability to conduct and support high-quality research is dependent on our ability to attract, train, partner with, and support talented researchers, statisticians, and evaluators from all backgrounds. IES needs contractor support to conduct outreach to diverse communities of researchers. Potential activities include identifying potential communities of researchers who have not been historically funded by IES, conducting need-sensing activities with these communities, developing outreach plans, conducting outreach events, and developing outreach and collateral materials (print, multi-media, and/or virtual).

Slide 24 Category 5:

Category 5, IES Operational Support, is comprised of general mission support to enable ongoing functions across IES.

As a reminder, the Government anticipates Category 5 being exclusive to small business. Once again, we plan to walkthrough examples of work that align with the requirements you see highlighted here. These examples were provided by IES subject matter experts across our Programs and Centers. While not an exhaustive list, they are intended to give you an idea for the type of work PRESTO is scoped to support in Category 5.

Slide 25 5.1.2 Business Case Quality Management Support:

Requirement 5.1.2, “Business Case Quality Management Support,” includes technical project management of specified IES IT investment planning, processes, and performance reporting (e.g., the Sample Surveys Division Information Technology Investment Management (ITIM) Quality Assurance/Quality Control Support contract) to ensure that IES IT investments are well-monitored, accurate, and that they experience continuous improvement.

This work might also include the following activities:

• Gathering and compiling information from project CORs

• Drafting documents, budgets and/or reports for use in the Department's IT investment management system, FOLIO; and

• Working with IES project managers to ensure overall accurate management of IT spending and reporting.

Slide 26 5.2.1 Event Planning:

Requirement 5.2.1, “Event Planning,” includes the provision of full logistical elements to plan, support, promote, and implement government-sponsored meetings, conferences, and other engagements as applicable.

An example of this type of work has been provided for the Center-wide Teacher Panel or CTP Event Support, wherein contractor support is needed to assist the National Center for Education Statistics (NCES) to maintain and support the CTP. Typical event support services may include, but is not limited to:

• Establishing and maintaining contact with qualified panel members

• Conducting background research on topics of relevance to CTP activities

• Developing materials to include meeting agendas, PowerPoint slide decks, talking points, brief surveys, and data collection tools for use at CTP meetings

• Scheduling and leading CTP virtual or in-person meetings

This concludes the scope category examples walkthrough. As a reminder, scope category requirements may continue to shift as the Government continues to conduct market research activities, and examples provided here today are only intended to demonstrate one instance or the type of work that might be supported under PRESTO. We hope this provides some insight to our industry partners, and I will turn the presentation over now to our Contracting Officer, Endrias, who will present the PRESTO Q&A.

Slide 27 Questions & Answers:

[Speaker: Endrias Leuleseged]

Good morning, my name is Endrias Leuleseged, and I am the Contracting Officer for this procurement. We appreciate those industry partners who have already contributed to PRESTO market research efforts, either through responding to the Sources Sought Notice or submitting a question in advance of today’s industry day.

As previously noted, not every question submitted via the web form on SAM.gov will be addressed, and it remains at the sole discretion of the Government to further solicit and respond to industry. However, the Government anticipates re-opening the industry Q&A web form posted on SAM.gov to collect additional questions in advance of the upcoming Small Business Industry Day on April 26.

Slide 28 Questions & Answers (Solicitation & Award):

Questions have been organized by topic (Solicitation & Award and Small Business).

Starting with Solicitation & Award, the Government received numerous questions related to the logistics of proposal submission, as well as the possibility for contractors to enter into teaming arrangements. Questions and Answers on the screen will not be read verbatim, however are provided for your reference.

While Section L of the PRESTO RFP will provide further instructions to offerors, Questions 1-3 highlight the fact that Government anticipates requiring separate technical proposals for each scope category. For example, Offerors may choose to bid on Categories 1, 2, and 5; thus, they should submit 3 separate technical proposals for each scope category.

Additionally, IES received some questions related to the application of restricted pools or small business set-aside categories. In sum, the Government anticipates Categories 1-4 will be comprised of a restricted and unrestricted pools with the restricted pool being open to small businesses only. Scope category 5 has been declared a small business set-aside.

Slide 29 Questions & Answers (Solicitation & Award):

With regards to teaming, the Government does encourage teaming by both large and small businesses to strengthen the organization’s capabilities.

IES also received several questions related to task orders. The Government does not anticipate releasing any task orders until the ID/IQ vehicle has been awarded.

Once the IDIQ is awarded, task orders will be issued to the most appropriate category. Vendors not included in the category the task order is issued through will not be able to bid on the work as a prime contractor, however, can serve as a subcontractor to primes in that category.

Finally, in response to Question #6, the Government does anticipate releasing PRESTO as a full and open ID/IQ.

Slide 30 Questions & Answers (Solicitation & Award):

Question 7 asks the Government to provide more information on the award of task orders to a primary scope category. Vendors awarded under the primary scope category will be eligible to bid on and support work in that scope category. As previously stated, a nominal amount of scope from another scope category may be included within a task order.

Again, I want to emphasis that tasks orders will NOT be awarded during the competition of the PRESTO ID/IQ.

Slide 31 Questions & Answers (Small Business):

The remaining questions are related to small business.

In summary, all small businesses are eligible to compete on the vehicle. The Government anticipates maintaining a restricted pool for small business vendors in scope Categories 1-4, and scope Category 5 has been declared a small business set-aside. Only small businesses awarded in the primary scope category are eligible to bid on the task order issued under that category as a prime contractor.

Small businesses within a selected category have the opportunity to compete against large businesses when a requirement is not set-aside for small businesses and issued to the unrestricted pool.

Slide 32 Important Dates:

The activities outlined on the screen reflect the key elements of IES’ industry engagement plan leading up to an anticipated RFP release in Summer 2022. IES previously released a sources sought, which closed on March 24th. Please note that you can still be considered for the RFP if you did not respond to the sources sought notice.

Our next industry day will be on April 26th and content is tailored to small business vendors. It will also include a facilitated question and answer segment to address industry questions about the vehicle and draft Statement of Work.

As a reminder, the PRESTO draft SOW which can be accessed on SAM.gov, remains pending the conclusion of ALL market research activities, and questions from industry must be submitted via the web form provided on SAM.gov in order for the Government to consider addressing and including them in industry day content.

The Department’s responses to questions received and addressed will be posted on SAM.gov in the near future along with this PowerPoint. In addition, a copy of the company names who registered for this industry day will be posted as well.

If you have any further questions when the MS Forms option is no longer available on SAM.gov, please direct those to me. Thank you for your time today and IES looks forward to your continued engagement on this opportunity.