PRESTO Statement of Work.pdf

Version Date: July 21, 2022

U.S. Department of Education (ED)

Institute of Education Sciences (IES)

Procurement of Research Evaluation and Statistics Task

Orders (PRESTO)

Section C - Scope of Work

Overview

IES intends to re-compete the Education Statistics Service Institute Network (ESSIN) contract by issuing a vehicle that encompasses additional requirements that supports the entire IES mission and that is accessible to the entire Department of Education (ED) as applicable. This new vehicle has been titled Procurement of Research Evaluation and Statistics Task Orders

(PRESTO), and shall be a multiple award, Indefinite Delivery/Indefinite Quantity (ID/IQ) contract vehicle.

This contract vehicle supports IES in its role as the independent education research, evaluation, and statistics arm of ED, while providing a diversified and qualified pool of vendors to help IES and ED meet mission critical needs.

Background

The goal of IES is to provide national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study, in order to provide parents, educators, students, researchers, policymakers, and the general public with reliable information about (A) the condition and progress of education in the United States, from early childhood through postsecondary education; (B) educational practices that support learning and improve academic achievement and access to educational opportunities for all students; and (C) the effectiveness of federal and other education programs.

The Education Sciences Reform Act (ESRA, Title I of P.L. 107-279) established IES as an independent research arm of ED. Title I, Part A, of ESRA establishes IES as a research institute within ED intended to provide “national leadership in expanding fundamental knowledge and understanding of education from early childhood through postsecondary study.” IES is authorized to carry out these objectives by compiling statistics, conducting research and evaluations, and disseminating information.

IES consists of four national education centers and an Office of the Director:

• National Center for Education Research (NCER) which sponsors research leading to the accumulation of knowledge and understanding of education.

• National Center for Education Statistics (NCES) which collects and analyzes education information and statistics in a manner that meets the highest methodological standards.

• National Center for Education Evaluation and Regional Assistance (NCEE) which supports evaluation, technical assistance, development, and dissemination activities.

• National Center for Special Education Research (NCSER) which sponsors research to expand knowledge and understanding of the needs of infants, toddlers, and children with disabilities.

• Office of the Director, which carries out specific duties, such as proposing research priorities and establishing procedures for technical and peer review. This includes the

Office of Science, which oversees the scientific peer review processes for IES grant applications and reports

Scope of Work

The scope of PRESTO comprises five distinct scope categories with associated NAICS codes assigned to each category. The scope of this vehicle is intended to be broad enough to capture the needs of IES throughout the entirety of the period of performance1. The scope of work associated with each of these task areas may change based on changes in applicable laws and regulations. Alterations to practices within each task area because of a change in law or regulation is within scope. Appendix A provides examples of work that falls within the scope of this ID/IQ. Appendix A is not an exhaustive example, and other work may fall within the scope of the task areas described below that is not otherwise listed or described in Appendix A.

The scope categories of PRESTO are as follows:

1. Education Sciences Support Activities

2. Assessment Development Support

3. Communications and Outreach

4. Recruitment

5. IES Operational Support

1 For the purposes of complying with the Competition in Contracting Act (CICA) parties responding to this requirement are notified that the scope for each category is incredibly broad and encompasses the universe of work that falls under these scope areas or that is substantially related to each scope area.

Moreover, as scientific practices evolve over time, performance in each scope area may shift based on changing practices for each area. Any incremental or evolutionary shift in practice for each area is within the scope of work as defined by this document.

Task orders will be awarded within a primary scope category2. Task Orders may be issued with secondary scope areas in one or more other categories.

Category 1: Education Sciences Support Activities

IES requires support to conduct surveys and evaluations that collect data and produce data products to support the mission of IES. Any effort that collects data from human subjects will require Office of Management and Budget (OMB) and/or Institutional Review Board (IRB) clearance packages.

Specifications and examples are available here:

https://pra.digital.gov/clearance-process/, https://www.reginfo.gov/public/do/PRAOMBHistory?ombControlNumber=1850-0944, https://www.reginfo.gov/public/do/PRAViewICR?ref_nbr=202104-1850-001.

The activities defined in the following sections are reflective of the work required to support IES’ mission goals.

1.1 General Education Sciences Support

1.1.1 Sampling

Activities such as sample designs and drawing samples representative of target parameters.

The support provided must ensure that explanations of sampling frames, selection criteria, goals and related metrics, and all other applicable details are included in all sampling activities.

1.1.2 Data Collection

Various data collection activities, including all method development and management of the process as well as data security and respondent privacy and confidentiality.

2 Primary scope area represents the majority of cost/price associated with each task area though this may not amount to more than 50% of cost/price if more than one scope area is applicable.

1.1.3 Data Processing, Management, and Data Creation

Support managing and converting data gathered during a study or in an evaluation collection process into organized data in fully documented data files ready for secondary analyses by the

Government, other contractors, the public, or any other third-party as required at the order level as applicable.

1.1.4 Data Management and Information System Management

Support, to include a broad range of data management and system management activities that include but are not limited to, data processing, method recommendation, data testing, and any other related activities.

1.1.5 Weighting and Variance Estimation

Development and review of weights and variances, to include providing recommendations and completing applicable documentation. Weighting shall account for all industry standard information, including, but not limited to, sample performance with factors, differential nonresponse across subpopulations that might introduce bias, and adjustments to known population totals.

1.1.6 Statistics and Data Analysis

Support across the full range of statistical and data analysis, from complex multi-faceted approaches to specialized techniques to support various IES mission needs.

1.1.7 Geospatial Activities

Support for geospatial activities to address statutory data requirements; manage data assets;

design analytic tools; integrate additional data collections; and support inter-and intra-agency production and analytic needs.

1.2 National Statistical, Evaluation, and Research Activities

1.2.1 Expert Help and Expert Reviewing Activities

Provision of expert technical support for consultations on and review of a wide range of technical materials that go into the development and operations of surveys, assessments, and evaluations. Provision of support and assistance, to include access to a wide array of senior statisticians, senior survey methodologists, senior technical experts, and senior subject matter experts with expertise across the full range of IES education sciences work. This support shall provision IES with the expert help it needs to execute its mission goals.

1.2.2 National Technical Assistance

Technical assistance services to external IES partners to meet applicable IES research, statistical, and/or evaluation standards, such as the NCES Statistical Standards, What Works

Clearinghouse (WWC) Standards, and Standards for Excellence in Education Research, and any other relevant standards that IES produces in future years.

1.2.3 Data Dissemination and Training Support

Preparation of training materials to support the release of data files and facilitate the use of IES data by the Government and the public or other relevant, defined third parties.

1.2.4 Congressional Allocation Program Support

Expert support in the calculation of Federal funds for allocation across education grant programs funded by the Federal Government. such as, but not limited to, Title 1. The support required includes though is not exclusively limited to assembling data needed under the formula allocation legislation, conducting data processing and analyses to prepare and verify assembled data, calculating annual formula allocation distributions, and preparing documentation to explain allocations.

1.2.5 Applied and Basic Research & Development (R&D) Studies

Conducting research and development to validate, test, and improve all aspects of education sciences activities.

1.2.6 Program-Based and Methodology Innovations

Development of innovative approaches, technologies, and procedures related to general education science support.

1.2.7 Report and Publication Production

Content generation and drafting support to aide in the development, production, digitalization, and dissemination of all applicable IES reports (e.g., first look reports, stats-in-brief reports, datapoints, analytical reports, technical reports, and manuals, and other reports required for the duration of the ID/IQ); NCES annual reports (e.g., the Digest of Education Statistics, the

Condition of Education, Projections of Education Statistics) and special topical reports (e.g., report focused on rural education or children with disabilities), literature reviews, conference papers, journal articles, and evaluation, research, and grant reports. The nature, content, format, and forum of reports may evolve over time and within the scope of this task.

1.2.8 Support for External/Internal Evaluation Activities

Analysis support for all external and internal evaluation activities, as well as research and development, to include evidence reviews, evaluation feasibility studies, design studies, and evaluation preparation.

1.2.9 IES Technical and Quality Reviews

Review support for draft deliverables and final products developed by IES or other entities to ensure that they are complete, thorough, accurate, neutral, non-biased, of technical high quality, easily understood, and meet NCES’s Statistical Standards, tabular guidelines, as well as the

IES Style Guide.

1.2.10 Training and Technical Assistance

Support in the development and delivery of training materials and methods to aid in the use of

IES products.

1.3 Sample Survey Activity Support

1.3.1 Sample Survey Management

Management of cross-sectional sample survey work across the entire country including, at times, Puerto Rico and outlying territories.

1.3.2 Longitudinal Studies Panel Management

Longitudinal panel management over long periods of time to ensure that data collection contractors can readily contact longitudinal study panel members between data collection operations to maintain participation in the longitudinal studies.

1.3.3 Rapid Response Surveys

Support to prepare and conduct rapid response surveys, from the point of identifying and defining a data need, to completing a complete draft of a first look report and dataset.

1.3.4 International Studies Activities

Support for the ED’s participation in international studies and for collaboration with a variety of international organizations and partners working in the field of Education.

Category 2: Assessment Development Support

IES requires support to develop assessment components for its sample survey studies and to develop and conduct large-scale assessments. The activities defined in the following sections are reflective of the work required to support IES’ mission goals.

2.1 Assessment Management

Conducting large-scale assessment studies and coordination across the entire country, including, at times, Puerto Rico and outlying territories.

2.2 Framework Development and Comparisons

Development of analytical and evaluative frameworks to define the scope and content of measurement targets or data collection instruments.

2.3 Item Development

2.3.1 Non-cognitive Items

Development, identification, documentation of sources, and evaluation of non-cognitive items to be used in data collections.

2.3.2 Cognitive Items

Provision of cognitive item writing expertise, as well as content knowledge expertise support, in the core subjects of mathematics, reading, and science, and in other subjects (e.g., civics, computer and information literacy, creativity, economics, financial literacy, technology and applications of STEM, social studies, writing, or others to be identified at the order level and not otherwise listed here).

2.3.3 Process Data Items

Support for designing “process data” measures and variables that can be incorporated into the design of an instrument, assessment or questionnaire, the metadata measures to describe captured process data, and the paradata measures to capture information regarding data collection operations.

2.3.4 Item Scoring and Coding

Scoring and coding of survey, assessment, and evaluation items through manual and automated processes3.

2.4 Psychometric Analysis

2.4.1 Cognitive/Non-Cognitive Items

Evaluating the effectiveness of cognitive/non-cognitive survey items and their interoperability with each other against baseline expectations.

2.4.2 Process Data

Evaluating process data, meta-data, and paradata.

2.5 Survey Feasibility Assessment

Evaluating the feasibility of any aspect of a survey or study, to include supporting framework and item development, questionnaires, instrument testing, field test settings, and data collection operation support (e.g., to determine whether planned procedures work in live data collection conditions), plans and procedures (e.g., recruiting strategies for full-scale data collections; for processes to enter schools, bring students into common assessment areas, provide respondents with access to instruments, and access and respond to on-line instruments in different browsers or on different hand-held devices).

2.6 Instrument Design

Design and the development of survey and assessment instruments for cross-sectional and longitudinal panel studies, including instruments with non-cognitive, cognitive, and process or paradata/metadata items.

3 The use, development and/or implementation of Artificial Intelligence mechanisms to support this task is within scope.

2.6.1 Questionnaire Development

Developing questionnaires and ensuring there is a logical flow of content, clear notation of expected skip patterns, possible internal logic checks if the questionnaires are to be computer assisted for respondents or interviewers, and instructions and assistance for respondents and interviewers.

2.6.2 Assessment Development

Assessment development for cross-sectional and longitudinal panel studies.

2.7 Cognitive Labs and Focus Groups

Provision of support in the piloting or testing of new survey and assessment items, especially open-ended or “constructed response” cognitive items, as well as new data collection formats or modes.

Category 3: Communications and Outreach

IES requires support for communications activities to include strategic communications oversight, external communications support, and internal communications support. The activities defined in the following sections are reflective of the work required to support IES’ mission goals.

3.1 Communications Planning and Management

3.1.1 Communications Planning

Drafting and updating Communications Management Office (CMO) policies, procedures, and guidelines as needed. Work may also include proactively identifying ways to promote IES products through multi-channel communications strategies (for example, web content, blogs, social media, email, and other media) as well as using audience and user analytics and research to inform and pilot communications strategies.

3.1.2 Communications Management

Conducting day-to-day functions of a new Communications Management Office (CMO), tracking IES communications in a centralized manner, and supporting strategic communications initiatives and events. Work may also include brand research and development, brand focus groups and workshops, providing branding recommendations, drafting brand guidelines, creating templates, and performing audits of communications content for brand compliance.

3.2 External and Internal Communications Content

3.2.1 Content Development and Editorial Support

Creating, reviewing, editing, publishing, and monitoring the impact of external and internal communications content. All applicable content must be Section 508 compliant, and this work also includes Section 508 compliance reviews on existing content.

3.2.2 Public and Media Relations

Executing public and media relations campaigns, including support for press outreach.

3.2.3 Graphic Design

Formatting, editing, and providing graphics for all applicable IES products. All graphic design provided must be in alignment with IES’ communications strategy and guidelines. This work includes designing digital products, including Section 508 compliance.

3.2.4 Social Media

Developing social media content, social media campaigns, and the application of social media analytics to measure success and track progress for the CMO, including newsletters.

Category 4: Recruitment

IES requires support to recruit educational entities and groups traditionally underrepresented in education research on a national and international scale. The activities defined in the following sections are reflective of the work required to support IES’ mission goals.

4.1 Recruitment for Education Sciences Activities

Assistance in the identification and recruitment of participants for IES-sponsored programs, surveys, and other education data collection activities. Recruitment targets include postsecondary institutions, professional organizations, schools, state and local education agencies, and other public or private enterprises to identify and recruit study participants to respond to surveys, participate in evaluations, and participate in other data collection activities concerning topics related to educational settings, as requested by individual IES Centers and divisions.

4.2 State Coordinators Recruitment Center

Support to establish, maintain, and manage a national recruitment center to work with applicable stakeholders at the state and local level, as well as private organizations.

4.3 Centralized Contact List

Support in developing and maintaining a centralized contact list of schools for IES outreach purposes.

4.4 Diversity, Equity, Inclusion, and Accessibility (DEIA) Initiatives

Support for DEIA initiatives, to include training and professional development, strategic planning, data collection and analysis, grants management support, administrative and logistical support, and outreach and communications support.

Category 5: IES Operational Support

IES requires general mission support to enable ongoing functions across the IES. The activities defined in the following sections are reflective of the work required to support IES’ mission goals.

5.1 Project Management Support

5.1.1 Project Program Integration and Coordination

Coordinating work for multiple aspects of research project work (e.g., surveys, assessments, and evaluation data collection) when multiple vendors are developing separate components of the project.

5.1.2 Business Case Quality Management Support

Technical project management of specified IES IT investment planning, processes, and performance reporting (e.g., the Sample Surveys Division Information Technology Investment

Management (ITIM) Quality Assurance/Quality Control Support contract) to ensure that IES IT investments are well-monitored, accurate, and that they experience continuous improvement.

5.1.3 Institutional Monitoring and Modernization Activities

Provision of institutional monitoring to understand how IES work is being conducted, and to implement approaches to improve and modernize IES operations.

5.1.4 Compliance Support

Ensuring compliance with federally legislated or administratively determined mandates (e.g., Evidence Act, Federal Statistical Directive 4, IES public access policy) and the provision of compliance tracking support.

5.2 Logistics Support

5.2.1 Event Planning

This includes the provision of full logistical elements to plan, support, promote, and implement

Government-sponsored meetings, conferences, and other engagements as applicable.

5.2.2 Help Desk Support

Support to respond to requests for information about IES studies, data, and products from the public, other ED staff, or federal agencies. This includes, but is not limited to, providing information about IES products via e-mail, telephone, or regular mail; and, on an as-needed basis, providing special tabulations.

Other Direct Costs, Order-Level Materials, and Information

Technology

Other Direct Costs and other Order Level Materials are considered within the scope of this requirement if they are provided in support of work associated with any of the scope categories.

In addition to Other Direct Costs or other Order Level Materials, work that requires Information Technology components or services is within scope as long as it is directly related to work associated with one or more scope categories identified. Pricing for Other Direct Costs, Order Level Materials, and any IT goods is not priced at the IDIQ level and may be negotiated at the order level as necessary.

Appendix: Examples for PRESTO

Category 1: Education Sciences Support Activities

IES requires support to conduct surveys and evaluations that collect data and produce data products to support the mission of IES. Any effort that collects data from human subjects will require Office of Management and Budget (OMB) and/or Institutional Review Board (IRB) clearance packages.

1.1 General Education Sciences Support

1.1.1 Sampling

Examples of this work: Different types of sampling include, but are not limited to, county, state, or national samples, simple random sampling, complex stratified sampling, and targeted sampling to support small area estimation or other special applications, as well as sampling from Address Based Frames, the Common Core of Data, the Integrated Postsecondary

Education Data System, and others.

1.1.2 Data Collection

This work includes, but is not limited to:

• Supporting evaluations by focus groups

• Conducting interviews

• Administering assessments and surveys

• Creating and maintaining time logs

• Extracting and coding administrative records (e.g., school and college transcripts, state assessments, program records and data files maintained by other agencies)

• Developing projections of education statistics

• Coding program records (e.g., grant applications, peer reviews of grant applications and IES reports, annual plans, performance reporting)

Examples of this work: Large-scale field sample survey data collection work, small-scale sample survey data collection work, other types of data collection (e.g., administrative data collection work), hiring and training of data collectors, and sample management.

1.1.3 Data Processing, Management, and Data Creation

This work includes, but is not limited to:

• Checking and cleaning raw data from both cross-sectional and longitudinal panel studies (including editing and imputation)

• Conducting a disclosure risk analysis and proposing swapping and data masking techniques to reduce the risk of respondent identification

• Evaluating and documenting response bias in the data

• Developing and applying bias corrections

• Conducting operations to impute missing data (when appropriate and if requested by the Government)

• Preparing a complete description of the data collection methodology, evaluation, and data processing (e.g., documentation of response bias, bias corrections, and missing data imputation)

• Preparing information, documentation, and instructions on how to read the data into widely used statistical packages and how to appropriately use the data file

• Processing files for distribution through IES online analysis tools (e.g., DataLab, ELSi, NAEP and International Data Explorer)

1.1.4 Data Management and Information System Management

This work includes, but is not limited to:

• Entering data from web-based collections

• Uploading data to information systems

• Generating data validation reports, composite or derived variables, descriptive analysis reports, web tables, and other products

• Supporting data migration

• Managing data assets, to include data files, web tables, methodology reports, and file structure changes

• Documenting database structure and content, program coding, and relationship of code modules and function entry points

• Monitoring data extractions, migrations of databases, editing and imputed data files

• Reviewing quality of data and data assets against applicable IES standards

• Developing and implementing an information system service management plan using industry standard metrics for IT systems supporting data collection

• Developing and implementing IT systems to support data collection, to include but not limited to a Configuration Management System (CMS)

• Operating and maintaining IT systems for data collection in accordance with all Federal Information Security Act (FISMA) standards and provide support for all certification and accreditation (C&A) activities to ensure the Authority to Operate

Examples of this work: Data processing and management, providing recommendations of methods, documentation, and data testing.

1.1.5 Weighting and Variance Estimation

This work includes, but is not limited to the development of:

• Weighting plans that must discuss why each step of an overall weighting process is applied

• Actual weights appropriate for the sampling approach

• Replicate weights appropriate to sample designs and collection experience to estimate variances

• Documentation to detail how to apply the weights appropriately to different types of analyses

The Government typically prefers to use replicate weighting approaches to estimate variances for sample-based studies.

Examples of this work: Developing weighting and variance estimates, providing recommendations for methods, and completing documentation.

1.1.6 Statistics and Data Analysis

This work includes, but is not limited to:

• Supporting evaluations and research activities that require analyses with public-or restricted-use files maintained by IES or by other federal offices (e.g., Federal Student Aid administrative records) to produce policy analyses for evaluations

• Supporting assessment that requires sophisticated data analysis with plausible values to detect changes in achievement trends

• Supporting international data analysis that requires the use of multiple population weights to make comparisons between countries

• Supporting longitudinal data analysis that requires panel weights to appropriately test hypotheses

Examples of this work: Different forms of standard analytical techniques, including but not limited to developing small area/sample size estimation and reporting techniques, forecasting models (e.g., econometric time series models, exponential smoothing, modeled projections), geo-spatial analytical and data management techniques (including GIS), organizing and creating new data sets from publicly available data, and analyzing such data.

1.1.7 Geospatial Activities

This work includes, but is not limited to:

• Developing and maintaining geospatial data web services and usage metrics

• Designing and maintaining georeferenced analytic and access tools

• Administrating organization-level licenses, tools, architecture, and operational platforms to support geospatial activities

• Maintaining and integrating large spatial and geodemographic databases and data collections

• Providing geospatial data access through federal and public data portals and through IES online tools (e.g., American Community Survey – Education Department (ACS-ED) Maps, ACS-Tables, Open Data)

• Supporting questions from researchers, program administrators, and the public related to educational geography

Examples of this work: Managing and integrating organization-level geospatial and geodemographic data assets; design and administrate geospatial IT applications and architecture; developing and disseminating data as web services through public portals; and supporting spatial analytic projects within and between organizations.

1.2 National Statistical, Evaluation, and Research Activities

1.2.1 Expert Help and Expert Reviewing Activities

This work includes but is not limited to:

• Reviewing and consulting on plans and proposals (e.g., sampling plans, data collection plans, weighting plans)

• Reviewing and consulting on operations and the execution of plans and proposals (e.g., sampling; data collection processes; data editing, imputations, and other data file development specifications; weighting specifications intermediate data files)

• Reviewing specifications prepared by others and providing quality control review of intermediate and final data files

• Reviewing and consulting on frameworks, item development, and assessment reporting

• Convening panels of senior experts to deliberate and provide recommendations (e.g., technical advice, standards and guidance, frameworks; questionnaire and assessment item selection; and peer review of products, reports, grants, and evaluations)

• Providing technical experts who consult with IES on specific problems

• Responding to higher profile requests from Commissioners of IES Centers (NCES, NCEE, NCER and NCSER), the Deputy Director for Science, and the Director of IES for quick turnaround (ranging from 1 to 5 business days)

• Informing development of survey instruments (such as indices, scales, social and emotional measures)

• Providing special populations expertise and content specialists with expertise that can include but is not limited to core academics (such as math, science, reading); STEM specialties (such as technology, engineering, computer literacy); and non-core subjects (such as fine arts, civics, creative writing)

• Preparing working papers, evaluative reports, and comparative reports

• Working with non-NCES datasets

• Responding to information requests from partnering organizations

• Preparing timely and accurate indicators for the annual America’s Children report

• Understanding the organizational structures and procedures of partnering organizations

Examples of this work: Provision of expert technical support, providing expert technical help, developing specific statistical and education research questions, and supporting the development and management of large-scale assessments, high-profile statistical and research and developing evaluation questions, supporting IES partnering activities with federal, state, and local agencies; private national organizations and businesses; and university statistical and research programs.

1.2.2 National Technical Assistance

Examples of this work: Supporting IES partners to meet IES research, statistical, or evaluation standards.

1.2.3 Data Dissemination and Training Support

This work includes, but is not limited to:

• Supporting the release of IES datafiles (including geospatial web services), any supporting materials, and appropriate technical documentation

• Developing and/or maintaining online data analysis and dissemination tools (e.g., as the NCES Online Codebook and DataLab)

• Preparing training materials to support the use of IES datafiles, including developing and producing online data set training materials (e.g., NCES Distance Learning and Data Training program)

• Developing in-person and online training for researchers to use IES datasets in appropriate ways (e.g., applying weights for data with complex sample designs), especially IES grantees

• Making IES data resources more accessible (e.g., 508 compliant dashboards).

• Expanding access to and use of IES data products, including target outreach to underserved communities.

Examples of this work: Facilitating the release of datafiles and developing, supporting, and providing training in the appropriate use of datasets.

1.2.4 Congressional Allocation Program Support

This work includes, but is not limited to:

• Assembling data needed under the formula allocation legislation

• Conducting data processing and analyses to prepare and verify assembled data

• Calculating annual formula allocation distributions

• Preparing documentation to explain allocations

Examples of this work: Working with formula allocations and their calculations.

1.2.5 Applied and Basic Research & Development (R&D) Studies

This work includes, but is not limited to:

• Survey operations (e.g., survey design through data collection and data processing)

• Information and products generated by IES through data tools, reports, and other dissemination processes

• Assessment survey methodologies (e.g., Item Response Theory (IRT), conditioning models, adaptive approaches, AI-scoring, AI-item generation)

• Linking assessments (e.g., NAEP and an international assessment)

• IES research and research standards (e.g., IES research and evaluation standards, WWC standards for effectiveness studies)

• Improved methods for study recruitment

• Modeling (e.g., demographic projections, time series, small area estimation)

Examples of this work: Conducting research and development activities to validate, explore, and improve any operational aspects of studies, including but not limited to data collections, methods for item and instrument development, sampling strategies, data collection strategies, weighting approaches, and imputation processes.

1.2.6 Program-Based and Methodology Innovations

• Introducing modern technologies, techniques, methodologies (e.g., improved methods for item and instrument development, sampling strategies, data collection strategies, weighting approaches, imputation processes, IRT)

• Instituting program and operational procedures (e.g., software, apps, quality control processes, AI applications)

• Introducing new survey, assessment, and evaluation designs

Examples of this work: Innovating and developing modern approaches for all aspects of general education sciences.

1.2.7 Report and Publication Production

This work includes, but is not limited to:

• Identifying authors with a depth of research expertise in a methodological or topic area to review and analyze research results

• Reviewing relevant literature and relevant data sources

• Developing an acceptable report design, plan (including analysis), and project timeline

• Generating content and drafting indicators, report, digest, or other publication type in line with this plan and project timeline

• Supporting the authors to ensure materials meet all current IES standards (e.g., Statistical Standards, tabular guidelines, and Style Guide) and are properly formatted from initial draft versions

• Responding to various rounds of comments and requested edits, which will require the preparation of multiple drafts versions to respond to comments from IES staff and reviewers

• Producing a final version of the report that meets all current IES standards and requirements

Examples of this work: Developing and producing reports and publications in scientific topic areas related to the IES mission (e.g., education research, sociological and psychological research, statistical research) and provide examples, making clear the timeframe within which the work was done, the amount of revision and/or number of drafts needed prior to approval to release the report, the type of analyses, the type of report or publication, and the audience for which the report was prepared.

1.2.8 Support for External/Internal Evaluation Activities

This work includes, but is not limited to:

• Determining if a proposed evaluation has evidence to support the study (e.g., through an evidence review), is widely used or is of interest to stakeholders, and if it can be implemented in the field, including how and whether experts are available to provide support for implementation

• Exploring how to encourage participation in an evaluation

• Supporting modest-scale data collection (e.g., interviews or focus groups with grantees, state/district/school officials, or potential provider/trainers)

• Developing a report detailing options for IES to consider

• Drafting instruments

• Providing options for the unit of random assignment, sampling and sample size outcomes, data sources, and/or analytical methods

• Developing options for the scope of or priorities for data collection instruments

• Developing a report detailing options for IES to consider

• Drafting instruments

• Setting up, facilitating, and running a competition for IES to select a provider to support implementation of the program, practice, or policy being evaluated

• Conducting or coordinating the development of data collection instruments

• Conducting or coordinating outreach and recruitment of sites to participate in evaluations and their associated data collections

• Performing or coordinating the random assignment of participants

Examples of this work: Conducting evidence reviews, implementing evaluation feasibility studies, conducting design studies, facilitating evaluation preparations and supporting evaluations of education programs, practices, and policies.

1.2.9 IES Technical and Quality Reviews

The range of such reviews and recommendations includes, but is not limited to:

• Variables, analyses, statistical tests, and supporting documentation for data products and professional writing

• Statistical displays (e.g., labels, tables, charts, graphics, interactive and online products)

• Argumentation (e.g., claims, hypotheses, evidence, inferences, conclusions)

• Textual content (e.g., statements, descriptions, narratives, content organization/structure, summaries)

• Product release and dissemination materials

Examples of this work: Providing thorough review of highly technical research products (e.g., reports, evaluations, statistical tables, data files, data documentation), providing examples and indicating past timeframes for (and timeliness of turning around) reviews, recommendations, and final quality assurance.

1.2.10 Training and Technical Assistance

This work includes, but is not limited to:

• Updating requirements for computer hardware

• Updating requirements for software compatibility

• Updating protocols for communications and file sharing

• Updating conventions for file structure and data definitions (which allow for ready access and sharing of data with users)

• Developing training and technical assistance (e.g., bootcamps and workshops on education research, evaluations, and/or statistics) for communities of researchers and institutions historically not funded by IES.

Examples of this work: Developing materials and providing training for general and technical audiences, as well as experience relevant to implementing cyber security protocols in a cloud and on-premises environment, database management and support, and public facing web design and support.

1.3 Sample Survey Activity Support

1.3.1 Sample Survey Management

This work includes, but is not limited to:

• Managing data collection for large national sample surveys administered by NCES.

Examples of this work: National Household Education Survey – NHES; National Teacher and

Principal Survey - NTPS

1.3.2 Longitudinal Studies Management

This work includes, but is not limited to:

• Managing data collection for large national longitudinal studies administered by IES.

Examples of this work: Early Childhood Longitudinal Study – ECLS; High School Longitudinal

Study of 2009 - HSLS:09; Beginning Postsecondary Students Longitudinal Study – BPS;

National Longitudinal Transition Study-2 (NLTS2) (ed.gov).

https://ies.ed.gov/ncser/projects/nlts2/

1.3.3 Rapid Response Surveys

This work includes expert support and expertise in preparing and conducting rapid response surveys within a matter of a few months. From the point of identifying and defining a “data need” to completing a complete draft of a first look report and dataset, surveys require less than 3 months execution, excluding Government clearance time requirements.

Examples of this work: Rapid response survey management; Fast Response Survey System at

NCES.

1.3.4 International Studies Activities

This work includes, but is not limited to:

• Attending international study working meetings and conferences

• Reviewing and commenting on international study documentation or international organization’s documents (e.g., mandates, minutes, program and work budgets, consultation paperwork, proposals, and conference summaries)

• Preparing working papers, and evaluative and comparative reports

• Working with international large-scale assessment (ILSA) datasets

• Conforming to international and national statistical standards and study guidelines/rules

• Responding to international study center information requests

• Preparing timely and accurate responses to international organization reporting and survey forms (e.g., UN and OECD surveys, Education at a Glance), that require transformations of NCES data and special analyses of unpublished data

• Understanding the organizational structures and procedures of the OECD and IEA

Examples of this work: Specific international organizations and large-scale international studies.

Category 2: Assessment Development Support

IES requires support to develop assessment components for its sample survey studies and to develop and conduct large-scale assessments. The activities defined in the following sections are reflective of the work required to support IES’ mission goals.

2.1 Assessment Management

This work includes, but is not limited to:

• Managing data collection for large national assessment studies administered by NCES.

https://nces.ed.gov/surveys/frss/

Examples of this work: Assessment development management.

2.2 Framework Development and Comparisons

This work includes, but is not limited to:

• Reviewing any relevant previous frameworks and content review panel recommendations for scope and content

• Documenting framework sources, including information about the framework’s structure or organization, its topics and subtopics, and justification for the inclusion or exclusion of content

• Comparing frameworks by content specialists

Frameworks need to be vetted by experts in the field to evaluate them for bias or omissions and may be circulated for public comment regarding their thoroughness and objectivity by the public.

Examples of this work: Developing and implementing different types of frameworks defining measurement targets and instruments.

2.3 Item Development

2.3.1 Non-cognitive Items

This work includes, but is not limited to:

• Reviewing recommendations from a content review panel (e.g., for content and data collection approaches, non-cognitive items that exist for the topic appropriate for the expected respondent type, and data collection mode)

• Finding or developing non-cognitive items

• Documenting item sources, including information about the properties of the non-cognitive item and justification for its selection for the Government study

• Developing items for a variety of respondents (e.g., students, teachers, and parents) who represent diverse respondent groups

• Providing a comparison of items by content specialists

If appropriate non-cognitive items do not exist, the Government will require the capacity to develop those needed items. New non-cognitive items need to be tested through cognitive laboratory testing and field testing to evaluate if they are capturing expected information and are understood by respondents as intended. Note that if non-cognitive items are copyrighted, it is necessary to identify who owns the rights to the items and assure agreements are in place to use the items for a Government study.

Examples of this work: Non-cognitive item development, to include, but not limited to:

• Cross-sectional studies

• Longitudinal panel studies

• Matrix-structured questionnaires

• Multi-path-dependent questionnaires

• Evidence-based development

2.3.2 Cognitive Items

This work includes, but is not limited to, developing and documenting:

• Multiple choice, open-ended or constructed response, and scenario-based items and item sets

• Paper-based, digital, interactive, online, and app-delivered items and item sets

• Problem-solving and experimental items and item sets

• Adaptive items and item sets (including automatically scored items in a multi-state design)

• High and low-end performance items and item sets

• Contexts and response mode (e.g., school/academic, work, authentic or “real world”)

• AI-generated items and item sets

Examples of this work: Cognitive item development, by subject domain, item type, adaptivity, covering the range of ability from low to high-end performance, context and response modes, use of current research-based approaches, and use of AI.

2.3.3 Process Data Items

Examples of this work: Process data, metadata, and paradata design and development.

2.3.4 Item Scoring and Coding

Examples of this work: Scoring and coding, including manual and automated processes.

2.4 Psychometric Analysis

2.4.1 Cognitive/Non-Cognitive Items

This work includes, but is not limited to:

• Analyzing and documenting how sets of items that are part of scales or other multi-item constructs are expected to behave and the rationale for that behavior

• Testing and analyzing the performance statistics of items to determine whether they work together as expected

• Preparing comprehensive documentation of the tests and analyses conducted, the reasons for the tests, results of the tests, and recommendations based on test results

Experience with cognitive/non-cognitive item analyses for cross-sectional studies should be distinguished from experience with longitudinal panel study items.

Examples of this work: Psychometric analyses of cognitive/non-cognitive items, including the range of tests and analyses they have conducted or supported.

2.4.2 Process Data

This work includes, but is not limited to:

• Evaluating forensic and operational applications (e.g., evaluating and improving timing, item functioning, optimal selection of items)

• Using process data, meta-data, and paradata as an input in the scoring process or as a performance measure

• Conducting research (e.g., understanding problem solving, thinking processes, and student behaviors)

Examples of this work: All types of process data analyses, including the range of applications they have supported.

2.5 Survey Feasibility Assessment

This work includes, but is not limited to, supporting:

• Frameworks, items and item development, questionnaires

• Instrument testing in cognitive labs

• Field test settings in multiple languages

• Full data collection operations (e.g., to determine whether planned procedures work in live data collection conditions), plans and procedures (e.g., recruiting strategies for full-scale data collections; for processes to enter schools, bring students into common assessment areas, provide respondents with access to instruments, and access and respond to on-line instruments in different browsers or on different hand-held devices)

Examples of this work: Evaluating the feasibility of such aspects of a survey or study.

2.6 Instrument Design

2.6.1 Questionnaire Development

This work includes, but is not limited to:

• Planning out the questionnaire’s structure appropriately (e.g., for the targeted population, for successive panels in a longitudinal study, for a paper-based or computer format)

• Assuring the items included and questionnaire structure result in measurement that allows for the analysis of change over time for longitudinal and repeated cross-sectional studies

• Providing documentation of the need for every item the questionnaire, especially those that ask for personally identifying information

• Using commercially available software that aligns with the proper layout and programing

• Testing in cognitive labs and field trials to evaluate whether respondents can complete the questionnaire instruments in the expected time, to determine if skip patterns and logic checks are working as expected, and to determine if any items or concepts are proving difficult to respondents

• Modification and finalization of formats after testing and field trials

Examples of this work: Questionnaire development, including, but not limited to, simple, path-dependent, multi-part, matrix, and adaptive questionnaires; cross-sectional and longitudinal study questionnaires; and paper-and-pencil, digital, interviewer-assisted, and any computer assisted formats.

2.6.2 Assessment Development

Examples of this work: Assessment development, including, but not limited to, simple, path-dependent, matrix, and adaptive instruments; and paper-and-pencil, digital, and other formats.

2.7 Cognitive Labs and Focus Groups

Examples of this work: Conducting cognitive testing for surveys or data collections within education industry.

Category 3: Communications and Outreach

IES requires support for communications activities to include strategic communications oversight, external communications support, and internal communications support. The activities defined in the following sections are reflective of the work required to support IES’ mission goals.

3.1 Communications Planning and Management

3.1.1 Communications Planning

This work includes, but is not limited to:

• Assessing communications current state and developing recommendations for improvements, specifically around channels, brand, and audience

• Developing communications office…