Attachment A - NAEP Design and Coordination Team SOO_V2.pdf

i

Attachment A

Statement of Objectives (SOO)

NAEP Design and Coordination Team Task 1: Program and Operations Coordination

Task 2: Sampling and Weighting

Task 3: Design, Analysis, and Reporting

National Assessment of Educational Progress

(NAEP)

2024-2029 ii

Table of Contents

Exhibit A- NAEP Coalition Components…………………………...………………...………10

Exhibit B- NAEP Structure: A Coalition of Teams.………….....………...…………....…….11

Exhibit C- NAEP Contracts not Included in the Coalition……...…………………………...12

Exhibit D- NAEP Schedule of Assessments…………....….…………………………………..13

1.0 Task 1 – Program and Operations Coordination (POC)….………..……………..……..14

1.1 Support NCES Strategic Planning for NAEP……………………………………15

Table 1.1 R&D Studies Across the NAEP Program….………………….18

1.2 Provide Program and Operations Coordination Across Multiple Contractors and

NCES………………………………………………………………………

1.3 Coordinate Development of Information Collection Request (ICR) for Office of

Management and Budget Approval………………………………….…………...25

1.11 General Activities…………………………………………………………….….26

Appendix 1.11.1……………...…………….………………….…………49

2.0 Task 2 - Sampling and Weighting (SW)…………………..…………….………………...55

2.1 Design and Select Samples…………………………………………………...….56

2.2 Generate Sampling Weights…………………………….……………...………..59

2.3 Targeted Special Studies…………………………………………………………60

2.11 General Activities………………………………………………………………..61

Appendix 2.11.1………………………………………………………….83

3.0 Task 3 - Design, Analysis, and Reporting (DAR)...…………..…………………………...89

3.1 Design………………………………………...………………….………………90

3.2 Psychometric and Statistical Analysis…………………….…..………...…..…...98

3.3 Reporting………………………………………………………………………..107

3.4 Research and Development…………………………………………….……….115

3.11 General Activities…...………………………………………………………….116

Appendix 3.11.1………………………………………………………..139

Procurement Overview The U.S. Department of Education intends to enter into ten, 5-year contracts with qualified entities to establish a Coalition of contractors Teams to carry out the design and coordination, assessment content, data collection, program support, web and reporting of one of its flagship programs, the National Assessment of Educational Progress (NAEP). NAEP is the nation’s largest, longest running and most highly regarded nationally representative assessment of student achievement in the United States. The program has maintained trend lines measuring the nation’s student achievement for more than 50 years on its Long-Term Trend (LTT) assessments in mathematics and reading and more than 30 years on its Main NAEP assessments, which include results for the nation, states, Puerto Rico, and the Department of Defense Education Activity (DODEA), and more than 20 large urban districts that have joined the Trial Urban District Assessment (TUDA) program over the years. It is imperative that the program maintain its core role of reporting educational progress over time, while, at the same time, taking advantage of unprecedented changes in education, in education assessment, and in the learning sciences, as well as rapid advances in technology to innovate, improve efficiency, and otherwise continue to lead the field of large-scale educational assessment.

As the Federal statistical agency charged with implementation of NAEP, the National Center for Education Statistics (NCES) has made a commitment to lead the field of large-scale, technology-based assessments and has identified four goals for the activities defined in this procurement.

They are:

Maintain NAEP’s high standards for quality and its position as the gold-standard in large-scale assessment, including:

o Maintenance of NAEP’s treasured 30-year trend lines in mathematics and reading at grades 4 and 8;

o The highest quality assessment and survey items in large-scale assessment;

o Smooth, trouble-free experiences for the states, districts, schools, educators, students, and other stakeholders participating in NAEP studies;

and o Accessible, informative, timely, relevant, objective, credible, error-free data for the wide range of stakeholders consuming NAEP data.

Improve cost-efficiency of processes across NAEP so that NAEP continues to be the best value for money in large-scale assessment.

Develop methods for constant innovation in all aspects of NAEP to improve

NAEP’s utility, the experience of NAEP for participants and data users, and the program’s cost-efficiency.

Create the future in large-scale assessment through far-reaching research and development and a deep understanding of developments in education and technology and where NAEP can make the best contribution to the nation’s understanding of educational progress.

There are numerous environmental factors that will influence NAEP during this period, many of which will be referenced throughout several Statements of Objectives (SOO). Examples include:

ongoing transition from a paper-and-pencil to technology-based assessment, the emerging interest in international comparisons for states and districts, advances in technology, including artificial intelligence and other developments that could broaden the capabilities of NAEP or improve efficiencies, advances in test development and new item types, development of measures of preparedness for post-secondary education and work, adoption of new content standards across most states that may or may not align with NAEP content frameworks, inclusion of more contextual information in reporting, the integration of NAEP and other NCES surveys and the overall impact on operationalization of the assessment, growth of virtual schools and out-of-school and technology-based learning by students, and potential budget reductions throughout the education and assessment enterprises.

NCES envisions this procurement period both as vital to maintaining a quality assessment and as a continuation of its transition to the future.

Purpose of NAEP

The purpose of NAEP is to serve as a national yardstick of student achievement. NAEP informs the public about what American students know and can do in various subject areas, including civics, mathematics, reading, science, and U.S. history. The assessment results are representative of the nation, regions of the country, and where appropriate, states and selected urban districts at grades 4, 8 and 12. NAEP Long-Term Trend assessment results are representative of the nation and regions of the country for students at ages 9, 13, and 17.

Administration

The Commissioner of Education Statistics (Commissioner) in NCES develops, administers, and reports NAEP while the National Assessment Governing Board (Governing Board), appointed by the Secretary of Education but independent of the Department, sets policy for NAEP.

Main NAEP assessments are made up of the following operational components:

National assessments include mathematics, reading, science, writing, civics, and U.S. history. These assessments follow the content frameworks developed by the Governing Board and use the latest advances in assessment research. Most subjects are assessed at grades 4, 8, and 12—although not all grades are assessed each time.

Students in both public and private schools are assessed.

State assessments are administered in four subjects (mathematics, reading, science, and writing). Only public schools are included in state-level assessments.

The Trial Urban District Assessment (TUDA) is a multiyear study of the feasibility of a district-level NAEP. Twenty-seven districts currently participate in

TUDA.

The NAEP High School Transcript Study (HSTS) is conducted in high schools selected for NAEP mathematics and science assessments. Results are reported primarily at the national level.

The National Indian Education Survey (NIES) is conducted in schools with

American Indian/Alaska Native students. Results are reported at the national level and for states with large samples of AI/AN students. (OPTION)

While Main NAEP comprises the bulk of the program, NAEP is a system of assessments that seeks to provide a comprehensive measure of educational progress in the country. Other major components are the long-term trend assessments, special studies, and studies for insuring the validity of NAEP items. They are as follows:

Long-Term Trend assessments (LTT) are given at the national level only, and are administered in a different manner from that used for main NAEP assessments.

LTT reports results in mathematics and reading that represent trends since the 1970s. LTT assessments are currently administered every four years.

Special studies are periodically conducted to answer a variety of questions. Examples of past special study topics include:

Inclusion and accommodations in NAEP, Math Accessibility Study, NAEP-Trends in International Math and Science Study Linking Study, Knowledge and Skills Appropriate study, Socioeconomic Status, and

Various technology-based assessment studies.

Pilot studies are conducted on all items prior to their use in NAEP assessments. All items are reviewed and go through several improvement activities. Cognitive labs and small-scale tryouts are conducted as needed on new and complex item types.

The NAEP Coalition Concept All NAEP assessments have critical path processes that are conducted by individual contractors as part of the overall team or Coalition of contractors; however, they are dependent upon one another for completion. Broadly, these processes are as follows:

Overall assessment design, Assessment and survey content development, Application development, including assessment and administration platforms for administering assessments, and other tools and applications that support the

NAEP program, Sampling and weighting, Data collection, Scoring, scaling, estimation, and analyses, Reporting and dissemination, and

Research and development across all aspects of the work.

Further, all the processes include the following:

state involvement to ensure participation of schools and the quality of the sampling and data collection processes, the use of the technology in various administrative functions and in the administration of digitally-based assessments, and planning and coordination to ensure that the various processes are aligned and coordinated with one another, and the schedule is maintained for timely reporting and dissemination of results and documentation.

Current Operating Practices NAEP instruments generally have a common block/booklet structure. Booklets consist of two separately timed 30-minute cognitive blocks followed by separately timed background questions.

NAEP subjects using the common block structure and assessed in the same testing period are spiraled together and administered in the same session.

Reading and mathematics assessments have the highest priority of all assessments for support and scheduling with item development occurring at least three years before an operational assessment. These items are pilot tested up to two years in advance as part of an operational assessment.

NAEP operates on a general principle that pilot tested items are not changed prior to use in an operational assessment. Pilot tested items that are of high quality but not used in the upcoming operational assessment are banked for future use. Pilot test data are not publicly released nor used to provide performance estimates.

Main NAEP assessments are administered during a six-week window beginning the last week of January and concluding the first week of March. In extreme cases, such as a natural disaster or major snowstorm, an additional week or two in March may be used for make-up assessments.

Assessments within states and large urban districts are spread across the entire assessment window.

States and districts are required to participate in grade 4 and 8 reading and mathematics assessments every two years (i.e., mandatory assessments). All other assessments are voluntary for states.

NAEP samples are geographically representative of the reporting jurisdictions and subgroups.

Schools and students are chosen with a measurable probability of selection.

In a state-level assessment, the national sample is composed of the aggregate of state samples.

However, when a state elects not to have the state-representative samples needed to report at the state level, some students in that state may be included to ensure representativeness of the national sample.

Long-Term Trend assessments are administered: in the fall to 13-year-olds, in the winter to 9-year-olds, and in the spring to 17-year-olds.

Testing accommodations are provided to students with disabilities and English learners to remove barriers to their participation if the constructs tested are not altered by the accommodations. For example, the reading assessment cannot be read aloud to students or translated to another language, as the NAEP reading framework defines the construct to include reading printed text in English. Data released from the assessments must report on students with a disability (SD) and English learners (EL) as a percentage of all students who completed the assessment. Within those two populations, the percent excluded, the percent tested with accommodations, and the percent tested without accommodations are reported.

The Governing Board plans and executes the initial release events for assessments after the Commissioner has officially released the results. The current primary means for releasing the initial 4th and 8th grade reading and mathematics results is a summary report intended for a public audience. National and state reading and mathematics reports at grades 4 and 8 are required to be available for release within 6 months after the completion of data collection. All other NAEP initial data releases must be available within 12 months after the completion of data collection, or on a different release schedule provided by the Governing Board. Releases include development of electronic materials and websites for public consumption. NAEP continues its move to more electronic materials and to greater use of the Web and social media for release activities.

Reports and data products from assessments on the Governing Board schedule where data are being released for the first time are identified as “initial release” and are differentiated from other NCES reports and products in the review process. “Initial release” products are reviewed by the Assessment Division (AD), at the NCES center-wide level for signoff by the Chief Statistician, by the NCES Commissioner, and by the Director of the Institute of Education Sciences (IES).

Other products, in addition to the reviews by the Assessment Division, the Chief Statistician, and the NCES Commissioner, undergo a peer review (either internal or external) organized by the

IES.

To the extent possible, NAEP designs, samples and analyses are conducted to maintain trend over time.

Aspects of the NAEP design may change during this procurement period (2024-2029). Inclusion policy and an improved measure of socioeconomic status are some of the practices that are under review for possible changes. Another possible change includes increasing the precision of measurement particularly at the lower and upper ends of the student ability distribution. In addition, as outlined below, there are several challenges that will need to be addressed during the procurement period. Consequently, there will be the need for flexible approaches to design modifications as NAEP’s role continues to evolve either through changes in legislation, Governing Board and Department of Education policies, technology, or the fields of educational measurement, education, or schooling.

Current Operating Cycle

The project consists of preparing for and conducting parallel assessments in different stages1.

Special studies and pilot testing are integrated into the operational assessments, as needed. All Coalition contractors are always actively involved during the calendar year and activities are often completed on tight timelines. The following paragraphs provide a sampling of the scheduling complexities.

The contractors responsible for the design of the assessment must develop a preliminary design up to 4 years in advance of the assessment so that all other contractors can prepare. The sampling contractor, for example, must begin the sample selection process in January the year before the assessment so states can be notified in May for the subsequent school year.

The item development contractor will be writing, reviewing and pilot testing items for four to six content areas across grades 4, 8 and 12 each year. Items are reviewed by NCES, a standing committee of content experts, the Governing Board, and the states throughout the year.

For the data collection contractor, the peak period is January to March, when substantial numbers of additional people are employed to collect data from students. But other times of the year, staff will be involved in developing training for data collectors, conducting the high school transcript studies, and working with private schools.

The reporting contractor must meet a tight six-month or twelve-month deadline for all initial report releases. In addition, the contractor analyzes and drafts other reports and conducts other studies or analyses throughout the year.

The contractor for materials and distribution begins their process of compiling materials in the late spring for distribution during the following winter based on sample schools identified in early May and sample students identified in the October – November period. The contractor is responsible for 24-hour turn-around of additional materials during the data collection in the January – March period.

1 When there is a new framework being created by the Governing Board, the cycle takes five or more years.

The contractor supporting the 50 state and 27 district coordinators for their various activities throughout the sampling, data collection, and reporting processes maintain weekly contact via electronic media, conduct three (3) to six

(6) in-person workshops each year, and provide orientation to NAEP for the 20- 25 percent of new coordinators each year.

The web contractor supports the release of The Nation's Report Card, online technical documentation and other key updates to the public websites regarding NAEP and assessment resources. This contractor also hosts the system on which the assessment platform lives and transmits data and is responsible for developing and maintaining several internal applications and tools that support the NAEP program.

The platform development contractor implements, supports, maintains, and enhances a NAEP digitally-based assessment (DBA) Platform and supporting technology systems. NAEP’s DBA Platform comprises the full end-to-end technical framework, or environment, that delivers the NAEP assessment to students.

Challenges NAEP’s primary purpose is to provide a fair and accurate measure of student achievement in various content areas and to report trends over time. At the same time, NAEP must be responsive to environmental and policy changes and to the emerging needs of national and state policymakers. NCES and the Coalition contractors must not only administer required assessments based on an established schedule and with designated populations but also face the competing priority of responding to changes, some of which cannot be anticipated.

For example, innovations in better and cheaper technology and their integration into the classroom and out-of-school learning experiences of the country’s students dictate that developing an assessment for both national and state assessments must be forward-thinking.

Creative development must consider cutting-edge technology, equity and fairness for students, ease of use, timeliness of administration, and flexibility of the jurisdiction environment.

NAEP will be challenged to identify the type of infrastructure that will be necessary to maintain a standard, unbiased administration while maintaining high levels of item security, and low operational burden and cost. The test administration software must allow seamless transition from assessing any of the NAEP digitally-based assessments to allowing these digitally-based items to function in the NAEP Questions Tool.

In addition to delivering assessments via appropriate technology, it will become increasingly more important that the application of technology to all parts of the NAEP system is updated to keep pace with the latest industry advances in a pragmatic and cost-effective manner. Many of NAEP’s administrative processes can be paperless.

Whenever commercial-off-the-shelf (COTS) software is appropriate and available, it should be given first consideration in the evaluation of solutions. In addition, some custom systems may also be developed. All these systems will have to be maintained and enhanced so that NAEP can easily incorporate new innovations in hardware and software with minimal impact on operations and/or budget.

Other challenges that will have to be addressed during this procurement period include the following:

International comparisons. There is a growing demand by national, state, and local policy makers and the public to know how their schools compare not only to the nation and states but to other countries. There is a recognition that our students will be competing with students from this country, as well as students worldwide. NAEP, the only state-representative assessment, is in an excellent position to serve as the link with international assessments to provide state and district estimates for comparison with other countries.

Preparedness for Post-Secondary Education and Work. A different type of challenge arises from the call for the use of the 12th grade NAEP reading and mathematics assessments in a way that captures and measures preparedness for participation in post-secondary education, the military, and/or the workplace.

Universal Design. NAEP assessments should be developed to maximize accessibility for all students to the test items.

Adaptive testing. Adapting which test blocks are administered to an examinee holds the promise of providing better measurement of students across the full range of student performance.

Changing state content standards. States content standards are continuously changing in a post Common Core State Standards initiatives environment. States have continued to modify standards and assessments during a time of changing accountability measures.

Trend. Critical to NAEP’s mission of reporting on the progress of education is maintaining trend. However, to remain relevant, NAEP must respond to changing environments. NAEP routinely faces this challenge through bridge studies that provide statistical evidence for continuing trend lines. However, with the major shifts in what and how NAEP assesses student performance, there may be other more efficient and cost-effective methods that can be used.

Procurement Principles

There are six primary principles the Coalition contracts shall adhere to in this procurement:

NAEP data must be accurate, of high quality and on time.

NAEP must continue to lead the field of large-scale assessment, especially in meeting the upcoming challenges.

NAEP must adapt to rapidly changing environmental factors.

NAEP must be designed in a way that maintains reporting trends over time.

NAEP must improve efficiencies in both costs to the government and burden on participants.

NAEP must ensure that the diversity of a workforce that characterizes the full range of NAEP stakeholders is represented at every level of NAEP operations.

To do so NAEP must make sure that every attempt possible is made to incorporate professional staff who represent that diversity and that each of the NAEP contractors increase the participation of small and minority-owned businesses within each contract.

Procurement Components

As seen in the following NAEP Coalition Components chart (Exhibit A), the new NAEP Coalition Procurements will consist of three functional teams, each functional team will contain their own team of pre-formed teaming vendors, and two stand-alone contracts which form ten critical path components with each component a separate contract award. Three of the Ten functional tasks are described in Tasks 1 through 3 in this Statement of Objectives (SOO), with each task corresponding to a separate contract.

The components are as shown in Exhibit A.

Exhibit A: NAEP Coalition Components

Task 1. Program and Operations Coordination

Supports overall strategic planning and design for NAEP;

ensures coordination among Coalition contractors; maintains data for tracking program progress; and provides logistical support for complaints and substantive comments.

Task 2. Sampling and Weighting

Selects samples; prepares sampling weights.

Task 3. Design, Analysis, and Reporting

Designs all pilot and field tests, operational assessments, and special studies; analyzes data ensuring reporting of valid results;

proposes and prepares psychometric and statistical analyses compatible with previous NAEP methodologies; specifies data needed to meet the goals for reporting; and prepares reports.

Task 4. Assessment Content Development

Develops cognitive items, scoring rubrics and survey questions;

assists in the training of scorers; conducts cognitive interviews/small-scale pilots of items, rubrics, and survey questions; translates items and survey questions; and conducts item reviews

Task 5. Scoring Human and AI-scoring of student responses.

Task 6. Platform Development

Develops NAEP assessment delivery platform utilizing a state-of-the-art, age-appropriate user interface (UI) and overall user experience (UX) that is consistent, intuitive, and accessible across all subjects and grade levels

Task 7. NAEP Support and Service Center

Provides support, training, and resources to state and TUDA coordinators to ensure the accurate and timely sampling, administration and reporting of NAEP in each state and TUDA district.

Tak 8. Data Collection

Administers assessments and collects data for pilot and field tests, operational assessments, and special studies; ships completed assessment materials to the scoring sites; conductions High School Transcript Study.)

Task 9. Program Support Management

Manage the NAEP program in alignment with Project Management (PM) best practices to ensure proper scheduling, quality control, risk management and communication across contractors and handoffs.

Task 10. Web Development, Operations, and Maintenance Support

Develops, implements, and supports public and private web applications and services; identifies and recommends emerging technologies; monitors compliance with all NCES web cybersecurity requirements; and ensures timely and accurate release of web products.

As seen in Exhibit B, the Coalition includes three sets of contracts considered as teams within the Coalition and to be competed as teams and two contracts that will be competed as standalone under separate RFPs. This is in contrast with prior competitions for NAEP contracts in which potential bidders formed teams to bid on the whole set of contracts. Separating the larger Coalition into smaller teams and standalone contracts for the competition is intended to make it easier for potential bidders to put together competitive bids and provide the government the best and lowest cost proposals.

Once the competition for contracts has been completed, all winning contractors must work together as a Coalition to ensure the successful implementation of NAEP. However, the three teams of contracts will be competed as teams because the hand offs between contractors within each team are so critical that vendors must develop relationships and plans before the competition to help ensure success on the critical hand offs within teams.

Exhibit B. The NAEP Structure: A Coalition of Teams

From left to right in Exhibit B above, the teams are Assessment Content, Design and Coordination, and Data Collection.

Design and Coordination: This team includes separate contracts for Program and Operations Coordination, Design, Analysis, and Reporting, and Sampling and Weighting. This is the core team for supporting the overall design of NAEP and coordination among contractors across the Coalition and requires real-time, close coordination across team contractors.

Assessment Content: This team includes separate contracts for Assessment Content Development, Scoring, and NAEP Platform Development that require close collaboration to ensure the successful and efficient development of the instruments for undertaking NAEP, including the scoring of student responses.

Data Collection: This team includes separate contracts for Data Collection and the NAEP Support & Service Center—two contracts that must work hand-in-glove for successful administration of NAEP in the field.

The Program Support Management and Web Development, Operations, and Maintenance Support contracts (shown in gray) will provide key support for the Coalition and are involved with key hand offs within the Coalition but may not require partnership with other vendors pre-bidding to ensure successful participation in the Coalition post-bidding.

Each of the three team sets of contracts will be bid as its own team, with contractors working together to submit a unified single group of proposals that will be accepted or rejected in their entirety. Any organization interested in making an offer on one or more components within a team must seek out other organizations that are recognized for complementary expertise to collaborate on a team proposal. The other two contracts will be standalone. NCES will award ten contracts, one for each component, with more than one contract being awarded to a single organization as appropriate. The contractors of all ten contracts will be required to develop agreements to ensure successful collaboration as an overall coalition.

Please note that other, non-Coalition contracts also support NAEP. See Exhibit C for a list of non-Coalition contracts. Additional collaboration between Coalition and non-Coalition contracts—not just within the Coalition contracts—is vital to the successful implementation of

NAEP.

Exhibit C: NAEP Contracts not Included in the Coalition

Publications, Outreach and Dissemination Support (POODS)

Procurement of Research Evaluation and Statistics Task Orders (PRESTO)

NAEP Logistics NAEP-State Technical and

Operations Cooperative National Indian Education Study

(NIES) Support Services Contact

1. Overview of the Work

Independently, and not as an agent of the United States Government, the Contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below. The work to be covered under this SOO begins with the award of the contracts in 2024 with completion of the 2024 assessments and ends in 2029 with development work for the 2030 assessments and beyond. Each assessment cycle includes field/pilot tests, and the operational assessments.

In November 2023, the Governing Board set the current schedule of assessments as presented in Exhibit D (see https://www.nagb.gov/naep/assessment-schedule.html).

Exhibit D: NAEP Schedule of Assessments

Year Subject National Grades Assessed

State Grades Assessed

TUDA Grades Assessed

2024 Reading Mathematics Science Transcript Studies

4, 8, 12 4, 8, 12

4, 8

2025 Long-term Trend ~

2026 READING

MATHEMATICS

Civics U.S. History

4, 8 4, 8

4, 8 4, 8

2028 Reading

Mathematics

SCIENCE

Transcript Studies

4, 8, 12 4, 8, 12 4, 8

4, 8, 12 4, 8, 12

4, 8 4, 8

2029 Long-term Trend ~ 2030 Reading

Mathematics

CIVICS

U.S. HISTORY

4, 8 4, 8 8, 12 8, 12

4, 8 4, 8

2032 Reading

Mathematics Science

WRITING

Transcript Studies

4, 8, 12 4, 8, 12 4, 8, 12 4, 8, 12

4, 8, 12 4, 8, 12

2033 Long-term Trend ~

Notes:

(1) Grades tested are 4, 8, and 12 unless otherwise indicated, except that long-term trend assessments sample students at ages 9, 13, and 17 and are conducted in reading and mathematics.

(2) Subjects in BOLD ALL CAPS indicate the years in which a new framework is planned for implementation.

Assessments to be Implemented

The assessment schedule (Exhibit D) provides a list of the assessments to be administered during this contract period, as well as future assessments for which development may be required in the contract period. However, please note that the schedule is subject to change. For example, the schedule may change to place NAEP assessments back on an “odd-year” schedule, as it was before the pandemic caused rescheduling of the 2021 NAEP assessments to 2022. If this occurs, assessments planned for fall 2025 and beyond are likely to be moved out one year. Any changes in assessment schedule will be reflected in updates to the NAGB assessment schedule webpage (https://www.nagb.gov/naep/assessment-schedule.html).

The flow of work on a NAEP assessment begins long before the operational assessment year and continues afterwards. Assessment content development begins several years before an assessment is fielded operationally; items are piloted typically two years before fielding operationally; and most operational innovations undergo proof-of-concept and field test stages in the years before operational implementation. Analysis and sampling plans must be developed well in advance of assessments and data collection operations, including contacting states, districts, and schools, as well as recruiting and training field staff will occur well before assessments go into the field. Reporting and dissemination, including the preparation of data files and documentation to be shared with researchers, as well as in-depth analysis by the NAEP program, extends well beyond the initial release of an assessment’s results. Further details on the scope and timing of work are included in the SOO tasks for each contract.

About the Statement of Objectives

The next part of the SOO is organized into three tasks, Task 1, Program Operations and Coordination (POC), Task 2, Sampling and Weighing (SW) and Task 3 Design, Analysis, and Reporting (DAR). While each contract will have specific responsibilities, there is a set of more general activities and associated tasks that are common to all the contracts described at the end of each task. Each general activity and the associated tasks are relevant to each contract and shall be included in each contract offer. The work below includes expectations for the NSSC and DC team. This team will work with the other teams and Coalition members to ensure a successful implementation of the NAEP program.

1.0 Task 1 - Program and Operations Coordination (POC)

The National Assessment of Educational Progress (NAEP) program is a complex suite of assessments carried out in an extensive, interconnected, and ongoing operation. The NAEP program includes operational assessments, pilot assessments and studies, other research & development intended to inform the National Center for Education Statistics (NCES) about the efficacy of a given process, and future planning, within a given year.

Scope

The Program and Operations Coordination (POC) contractor serves as the study director and provides key roles in supporting NCES in day-to-day program and operations coordination of NAEP, research and development, strategic planning, and the design and development of

NAEP.

The POC contractor supports NCES in coordinating the core activities of the NAEP program across multiple contractors, as well as providing intellectual support for strategic development of the program. In this way, the POC contractor serves as the NAEP programs key organizing force in support of NCES’s direction of the program.

As the contractor responsible for effecting the smooth integration of the various components of the NAEP program across multiple NAEP contractors and providing NCES key support for the strategic direction of NAEP, the POC contractor shall support NCES NAEP goals to:

Ensure NAEP’s high standards for quality and its position as the gold-standard in large-scale assessment, including:

Maintenance of NAEP’s treasured 30-year trend lines in mathematics and reading at grades 4 and 8;

The highest quality assessment and survey items in large-scale assessment;

Smooth, trouble-free experiences for the states, districts, schools, educators, students, and other stakeholders participating in NAEP studies;

Accessible, informative, timely, relevant, objective, credible, error-free data for the wide range of stakeholders consuming NAEP data;

Improve cost-efficiency of processes across NAEP so that NAEP continues to be the best value for money in large-scale assessment;

Develop methods for constant innovation in all aspects of NAEP to improve NAEP’s utility, the experience of NAEP for participants and data users, and the program’s cost-efficiency; and

Create the future in large-scale assessment through far-reaching research and development and a deep understanding of developments in education and technology and where NAEP can make the best contribution to the nation’s understanding of educational progress.

Program and Operations Coordination Tasks

1.1 Support NCES Strategic Planning for NAEP

1.2 Provide Program and Operations Coordination across multiple contractors and NCES

1.3 Coordinate development of information collection requests (ICR) for

OMB approval 1.11General Activities

These tasks are divided into subtasks and provided in further detail below.

1.1 Support NCES strategic planning for NAEP

The contractor shall support NCES in developing the NAEP Strategic Plan to cover a five-year period beginning at the start contract award and shall also provide revisions or updates to the strategic plan, as necessary. To help develop the plan, the contractor shall develop procedures for scanning and monitoring NAEP's environment for changes in the large-scale assessment industry at the state, district, and international levels. Such procedures shall include identifying opportunities and threats that support or detract from efforts to carry out assessments as planned, to advance NAEP as the leader in large scale assessment, and to promote quality in assessment practices. The contractor shall also help identify and interpret trends and changes in the education environment, assist NCES in implementing procedures for forecasting changes and trends, and identify strategies to effectively manage the impact of such changes on NAEP stakeholders and contractors, through outreach and communication aimed at facilitating an understanding and acceptance of change. These actions should be the basis for creating a new plan, and for providing recommendations to update the plan.

They shall focus on high-level issues, such as changing state and international policies, rapidly emerging technological changes, educational inequity, declining participation in assessments at some grade levels or with some subjects, and the challenge of interpreting performance trends.

1.1.1 Support NAEP Strategic Plan document

The contractor shall support NCES in developing and maintaining the NAEP Strategic Plan document. The initial Plan document shall cover a five-year period from the start of the contract.

Over time, the Plan document shall continue to look forward over a five-year horizon. The contractor shall develop a first draft within the first month of the start of the contract and implement a review/revise/resubmit process to produce a final initial plan within three months of the start of the contract. The contractor shall develop and implement a process for revision of the Plan document across the contract cycle, as well as tracking the program’s progress against the Plan. The Plan shall be a document that NCES can use effectively to communicate its vision of NAEP in the short and long-term; ensure that the vision is consistent with the program’s goals and developments in education, technology, and in large-scale assessment; plan activities to support development of the program toward this vision; and track the program’s progress. The process for development and maintenance of the Plan document must ensure that the plan continually serves these purposes for NCES.

1.1.2 Support the NAEP Executive Steering Committee

The contractor shall establish and maintain the NAEP Executive Steering Committee (Committee), comprising of Corporate Officers from all NAEP functional teams, the NCES Commissioner, NAEP Associate Commissioner, NCES Director of Strategic Partnerships, NCES Chief Psychometrician, and Assessment Division (AD) managers. The Committee’s purpose is to support development, maintenance, and tracking of progress against the Strategic Plan Document; report to NCES on the status of the implementation of NAEP work under the contracts; and provide other tactical and strategic support to NCES in the direction of the NAEP program.

The contractor shall schedule and convene the Committee meetings. As needed, the Committee shall meet up to five times annually. At the start of the contract and the start of each calendar year thereafter, the contractor will set tentative dates and times and draft preliminary agendas for the upcoming Committee meetings. Dates/times and agenda items may change as events arise throughout the year; however, the initial draft agendas and timing of the meetings shall reflect projected needs for the year as reflected in the Strategic Plan Document and NAEP assessment schedule.

The contractor shall also convene additional ad-hoc expert panels to meet with the Committee, as the need arises, up to four times per year. In past years NAEP assembled ad-hoc panels of experts from various relevant disciplines and industries beyond the large-scale assessment community to provide input for NAEP’s strategic planning. To ensure a diversity of perspectives on as broad a range of pertinent issues as possible, the composition of any panels of experts convened for such purposes shall vary according to need.

The contractor shall document all meetings (regular Committee meetings as well ad-hoc panels), including deliberations and resultant recommendations, and deliver them to the Contract Officer Representative (COR), and the Committee within two weeks of the meeting. The contractor shall monitor progress on action items identified during the meetings and make recommendations to NCES on approaches to operationalizing resulting recommendations of the Committee and ad-hoc panels. Other NAEP meetings, consisting of NAEP, and other NAEP program staff, will be described in more detail in tasktask 1.2.2 (Coordinate planning and review meetings among NAEP contractors and NCES). The contractor shall report the status of action items and implementation of recommendations as part of their weekly communications with the COR.

1.1.3 Coordinate NAEP Research and Development

NAEP Research and Development (R&D) drives the NAEP program forward by identifying new and better ways to measure and report on student achievement and expanding what NAEP is capable of, enabling NAEP to retain its position as the gold standard for assessment and continue to serve as a model for other large-scale assessments. R&D efforts have in the past focused on validating and improving assessment practices, anticipating, and responding to external policy-related influences on the scope and nature of the assessments, leveraging other data sources to expand what NAEP can report about student achievement, and developing and refining psychometric and statistical procedures.

NAEP’s R&D is conducted by various contractors both inside and outside of the NAEP program and expert groups that focus on different aspects of NAEP. Table 1.1 shows the types of NAEP R&D studies that have historically been conducted and the contract(s) or activity under which they have been conducted during the current contract period (though not all activities are conducted under the current contracts). A list of published NAEP studies is available on the NCES website (https://nces.ed.gov/pubsearch/).

Table 1.1 R&D Studies Across the NAEP Program

Contract/Activity Purpose and Approach

Special Studies and Proof-of-Concept Studies

Studies that require data collection from moderate to large samples and involve the participation of some or all the NAEP contractors.

Such studies might focus on subject areas and measure student achievement, study a methodological improvement or be operational in nature, such as those supporting NAEP modernization efforts.

Assessment Content Development (ACD) contract

Targeted studies conducted under the ACD contract, designed to inform, and advance cognitive ACD for new frameworks or survey questionnaire development, or to address specific issues in NAEP item pools.

NAEP Platform Development (PD) contract

Usability studies conducted under the NAEP PD contract designed to support development and implementation of next-Gen platform features, tools, and interactive components. These are typically conducted with small sample sizes using cognitive interview approaches.

Survey Assessment Innovation Lab (SAIL)

Innovative research studies carried out within the NAEP teams but with support from external researchers as appropriate. These are intended to keep NAEP at the forefront of innovation and best practice, but at the same time grounded in the operational realities of

NAEP.

NAEP Validity Studies (NVS) Panel

The NVS Panel (under a non-Functional Teams contract) recommends and conducts studies aimed at maintaining and developing high technical quality of NAEP assessments. Topics emerge through discussion between the panel and NCES. Panel members are nationally-recognized psychometricians and experts in NAEP subject areas. The panel’s work can be found at https://www.air.org/project/naep-validity-studies-nvs-panel

Education Statistics Services Institute Network (ESSIN)

ESSIN (non-NAEP teams) researchers produce ad-hoc research or papers to address specific needs of the NAEP program to advance NAEP methodology. For example, recent research has included exploring linkages between NAEP and external datasets and developing a new measure of student socio-economic status.

Design and Analysis Committee (DAC), under the Design, Analysis and Reporting (DAR) contract

DAC reviews and proposes methodologies and analytical procedures for NAEP studies and operational data collections. DAC members, as individuals, or within groups, identify and explore issues pertinent to the design of NAEP. DAC members are national authorities in methodological areas critical to the design and implementation of NAEP. DAC special studies are typically focused on statistical and psychometric methodology and issues.

These many efforts need to be coordinated to ensure that they are, individually and collectively, aligned with the NAEP Strategic Plan and NCES’s priorities, in a way that NAEP’s R&D portfolio is comprehensive and addresses areas of interest and significance but, also R&D overall is moving forward to advance the NAEP program. This coordination is also important for ensuring that outcomes of R&D are used by the NAEP program and communicated to NAEP stakeholders.

The POC contractor shall provide support to NCES in developing and maintaining a coherent R&D program to ensure that NAEP R&D efforts address priority areas, that special studies and research projects are conducted in coordination with other NAEP activities, as appropriate, and R&D outcomes are communicated to stakeholders both internal and external to the NAEP program and incorporated into NAEP processes when appropriate.

1.1.3.1 Develop, monitor, and maintain a comprehensive R&D agenda

The contractor shall support NCES in developing, monitoring, and maintaining a coherent and comprehensive research agenda for a rolling five-year horizon that aligns with the NAEP Strategic Plan and is consistent with NAEP operational planning. The research agenda shall identify R&D to support innovations and improvements to NAEP methodologies and processes.

Development of the research agenda shall take into consideration the foci of current and recent R&D conducted by NAEP functional teams, on-functional teams partners and expert groups, identifying gaps and opportunities for new R&D efforts to address unmet needs and emerging issues given the state of the field. The R&D agenda shall inform decisions about NAEP R&D commitments. Working with an NCES inhouse R&D advisory committee, the contractor shall make recommendations for creating the required balance across various research issues and needs by taking a broad view of NAEP’s goals and advising on the formulation of a research agenda that will best serve NAEP.

NAEP R&D Landscape Review. To support establishing the R&D agenda, the contractor shall review and summarize current and recent NAEP R&D activities conducted across the functional and non-functional team’s contracts and external panels, with input from contractors as needed.

This review will provide a comprehensive view of NAEP’s R&D efforts to date, highlight alignment of R&D with NCES goals, identify any duplications of effort, and identify gaps and potential new directions considering the NAEP Strategic Plan Document. It will also take into consideration the state of the field and emerging technologies and innovations outside of NAEP (for example, artificial intelligence).

Develop and Maintain R&D Agenda. The Executive Steering Committee will consider the R&D summary report and identify priorities for an R&D agenda that aligns with the NAEP Strategic Plan Document and builds on current efforts and expands R&D to address new innovations and directions for NAEP. The contractor shall prepare a R&D agenda, that outlines the objectives, studies, and a timeline of the R&D. The research agenda shall be reviewed continuously and updated on an annual basis considering NAEP R&D conducted to date, developments in the field, and emerging needs of the NAEP program.