Attachment A - NAEP Assessment Content Team (SOO).pdf

i

Attachment A

Statement of Objectives (SOO)

NAEP Assessment Content Development Team

Task 4: Assessment Content Development

Task 5: Scoring

Task 6: NAEP Platform Development

National Assessment of Educational Progress

(NAEP)

2024-2029 ii

Table of Contents

Exhibit A- NAEP Coalition Components…………………………………………………….xi

Exhibit B- NAEP Structure: A Coalition of Teams………………………………..………xiii

Exhibit C- NAEP Contracts not Included in the Coalition…………………………..……xiv

Exhibit D- NAEP Schedule of Assessments…………………………………………………xv

4.0 Task 4 - Assessment Content Development………………………………………………1

4.1 Manage Assessment Content Development…………………………………………3

4.2 Develop and Maintain Content for the Cognitive Assessment………………………4

4.3 Develop and Maintain Content for the Survey Questionnaires…………………..…17

4.4 Prepare for the Assessment Administration…………………………………………26

4.5 Convene External Reviews………………………………………………………….30

4.6 Conduct Research to Improve Assessment Development…………………………..34

4.7 Support IT Business Process……………………………………………………..…36

4.11 General Activities………………………………………………………………...…38

5.0 Task 5- Scoring……………………………………………………………………………68

5.1 Human Scoring and Monitoring…………………………………………………….70

5.2 Automated Scoring and Monitoring…………………………………………...……74

5.3 Conduct Targeted Special Studies…………………………………………………..77

5.11 General Activities…………………………………………………………………...78

6.0 Task 6- NAEP Platform Development…………………………………………………. 107

6.1 Implement, Maintain and Enhance the DBA Platform and

Supporting Systems…………………………………………………………………110

6.2 Ensure Quality of the Platform and Supporting Systems………………………...…123

6.3 Operate and Support the Platform and Supporting Systems………………………..127

6.4 Conduct Studies and R&D for the Platform and Supporting Systems…………...…135

6.5 Convene Platform Development Steering Meetings………………………………..138

6.11 General Activities………………………………………………………………...…139

iii

Procurement Overview

The U.S. Department of Education intends to enter into ten, 5-year contracts with qualified entities to establish a Coalition of contractors Teams to carry out the design and coordination, assessment content, data collection, program support, web and reporting of one of its flagship programs, the National Assessment of Educational Progress (NAEP). NAEP is the nation’s largest, longest running and most highly regarded nationally representative assessment of student achievement in the United States. The program has maintained trend lines measuring the nation’s student achievement for more than 50 years on its Long-Term Trend (LTT) assessments in mathematics and reading and more than 30 years on its Main NAEP assessments, which include results for the nation, states, Puerto Rico, and the Department of Defense Education Activity

(DODEA), and more than 20 large urban districts that have joined the Trial Urban District

Assessment (TUDA) program over the years. It is imperative that the program maintain its core role of reporting educational progress over time, while, at the same time, taking advantage of unprecedented changes in education, in education assessment, and in the learning sciences, as well as rapid advances in technology to innovate, improve efficiency, and otherwise continue to lead the field of large-scale educational assessment.

As the Federal statistical agency charged with implementation of NAEP, the National Center for

Education Statistics (NCES) has made a commitment to lead the field of large-scale, technology-based assessments and has identified four goals for the activities defined in this procurement.

They are:

• Maintain NAEP’s high standards for quality and its position as the gold-standard in large-scale assessment, including:

o Maintenance of NAEP’s treasured 30-year trend lines in mathematics and reading at grades 4 and 8;

o The highest quality assessment and survey items in large-scale assessment;

o Smooth, trouble-free experiences for the states, districts, schools, educators, students, and other stakeholders participating in NAEP studies;

and o Accessible, informative, timely, relevant, objective, credible, error-free data for the wide range of stakeholders consuming NAEP data.

• Improve cost-efficiency of processes across NAEP so that NAEP continues to be the best value for money in large-scale assessment.

• Develop methods for constant innovation in all aspects of NAEP to improve

NAEP’s utility, the experience of NAEP for participants and data users, and the program’s cost-efficiency.

• Create the future in large-scale assessment through far-reaching research and development and a deep understanding of developments in education and technology and where NAEP can make the best contribution to the nation’s understanding of educational progress.

There are numerous environmental factors that will influence NAEP during this period, many of which will be referenced throughout several Statements of Objectives (SOO). Examples include:

iv

• ongoing transition from a paper-and-pencil to technology-based assessment,

• the emerging interest in international comparisons for states and districts,

• advances in technology, including artificial intelligence and other developments that could broaden the capabilities of NAEP or improve efficiencies,

• advances in test development and new item types,

• development of measures of preparedness for post-secondary education and work,

• adoption of new content standards across most states that may or may not align with NAEP content frameworks,

• inclusion of more contextual information in reporting,

• the integration of NAEP and other NCES surveys and the overall impact on operationalization of the assessment,

• growth of virtual schools and out-of-school and technology-based learning by students, and

• potential budget reductions throughout the education and assessment enterprises.

NCES envisions this procurement period both as vital to maintaining a quality assessment and as a continuation of its transition to the future.

Purpose of NAEP

The purpose of NAEP is to serve as a national yardstick of student achievement. NAEP informs the public about what American students know and can do in various subject areas, including civics, mathematics, reading, science, and U.S. history. The assessment results are representative of the nation, regions of the country, and where appropriate, states and selected urban districts at grades 4, 8 and 12. NAEP Long-Term Trend assessment results are representative of the nation and regions of the country for students at ages 9, 13, and 17.

Administration

The Commissioner of Education Statistics (Commissioner) in NCES develops, administers, and reports NAEP while the National Assessment Governing Board (Governing Board), appointed by the Secretary of Education but independent of the Department, sets policy for NAEP.

Main NAEP assessments are made up of the following operational components:

• National assessments include mathematics, reading, science, writing, civics, and

U.S. history. These assessments follow the content frameworks developed by the

Governing Board and use the latest advances in assessment research. Most subjects v are assessed at grades 4, 8, and 12—although not all grades are assessed each time.

Students in both public and private schools are assessed.

• State assessments are administered in four subjects (mathematics, reading, science, and writing). Only public schools are included in state-level assessments.

• The Trial Urban District Assessment (TUDA) is a multiyear study of the feasibility of a district-level NAEP. Twenty-seven districts currently participate in

TUDA.

• The NAEP High School Transcript Study (HSTS) is conducted in high schools selected for NAEP mathematics and science assessments. Results are reported primarily at the national level.

• The National Indian Education Survey (NIES) is conducted in schools with

American Indian/Alaska Native students. Results are reported at the national level and for states with large samples of AI/AN students. (OPTION)

While Main NAEP comprises the bulk of the program, NAEP is a system of assessments that seeks to provide a comprehensive measure of educational progress in the country. Other major components are the long-term trend assessments, special studies, and studies for insuring the validity of NAEP items. They are as follows:

Long-Term Trend assessments (LTT) are given at the national level only, and are administered in a different manner from that used for main NAEP assessments.

LTT reports results in mathematics and reading that represent trends since the

1970s. LTT assessments are currently administered every four years.

Special studies are periodically conducted to answer a variety of questions. Examples of past special study topics include:

• Inclusion and accommodations in NAEP,

• Math Accessibility Study,

• NAEP-Trends in International Math and Science Study Linking Study,

• Knowledge and Skills Appropriate study,

• Socioeconomic Status, and

• Various technology-based assessment studies.

Pilot studies are conducted on all items prior to their use in NAEP assessments. All items are reviewed and go through several improvement activities. Cognitive labs and small-scale tryouts are conducted as needed on new and complex item types.

vi

The NAEP Coalition Concept

All NAEP assessments have critical path processes that are conducted by individual contractors as part of the overall team or Coalition of contractors; however, they are dependent upon one another for completion. Broadly, these processes are as follows:

• Overall assessment design,

• Assessment and survey content development,

• Application development, including assessment and administration platforms for administering assessments, and other tools and applications that support the

NAEP program,

• Sampling and weighting,

• Data collection,

• Scoring, scaling, estimation, and analyses,

• Reporting and dissemination, and

• Research and development across all aspects of the work.

Further, all the processes include the following:

• state involvement to ensure participation of schools and the quality of the sampling and data collection processes,

• the use of the technology in various administrative functions and in the administration of digitally-based assessments, and

• planning and coordination to ensure that the various processes are aligned and coordinated with one another, and the schedule is maintained for timely reporting and dissemination of results and documentation.

Current Operating Practices

NAEP instruments generally have a common block/booklet structure. Booklets consist of two separately timed 30-minute cognitive blocks followed by separately timed background questions.

NAEP subjects using the common block structure and assessed in the same testing period are spiraled together and administered in the same session.

Reading and mathematics assessments have the highest priority of all assessments for support and scheduling with item development occurring at least three years before an operational vii assessment. These items are pilot tested up to two years in advance as part of an operational assessment.

NAEP operates on a general principle that pilot tested items are not changed prior to use in an operational assessment. Pilot tested items that are of high quality but not used in the upcoming operational assessment are banked for future use. Pilot test data are not publicly released nor used to provide performance estimates.

Main NAEP assessments are administered during a six-week window beginning the last week of

January and concluding the first week of March. In extreme cases, such as a natural disaster or major snowstorm, an additional week or two in March may be used for make-up assessments.

Assessments within states and large urban districts are spread across the entire assessment window.

States and districts are required to participate in grade 4 and 8 reading and mathematics assessments every two years (i.e., mandatory assessments). All other assessments are voluntary for states.

NAEP samples are geographically representative of the reporting jurisdictions and subgroups.

Schools and students are chosen with a measurable probability of selection.

In a state-level assessment, the national sample is composed of the aggregate of state samples.

However, when a state elects not to have the state-representative samples needed to report at the state level, some students in that state may be included to ensure representativeness of the national sample.

Long-Term Trend assessments are administered: in the fall to 13-year-olds, in the winter to 9-year-olds, and in the spring to 17-year-olds.

Testing accommodations are provided to students with disabilities and English learners to remove barriers to their participation if the constructs tested are not altered by the accommodations. For example, the reading assessment cannot be read aloud to students or translated to another language, as the NAEP reading framework defines the construct to include reading printed text in English. Data released from the assessments must report on students with a disability (SD) and English learners (EL) as a percentage of all students who completed the assessment. Within those two populations, the percent excluded, the percent tested with accommodations, and the percent tested without accommodations are reported.

The Governing Board plans and executes the initial release events for assessments after the

Commissioner has officially released the results. The current primary means for releasing the initial 4th and 8th grade reading and mathematics results is a summary report intended for a public audience. National and state reading and mathematics reports at grades 4 and 8 are required to be available for release within 6 months after the completion of data collection. All other NAEP initial data releases must be available within 12 months after the completion of data collection, or on a different release schedule provided by the Governing Board. Releases include development of electronic materials and websites for public consumption. NAEP continues its move to more electronic materials and to greater use of the Web and social media for release activities.

viii

Reports and data products from assessments on the Governing Board schedule where data are being released for the first time are identified as “initial release” and are differentiated from other

NCES reports and products in the review process. “Initial release” products are reviewed by the

Assessment Division (AD), at the NCES center-wide level for signoff by the Chief Statistician, by the NCES Commissioner, and by the Director of the Institute of Education Sciences (IES).

Other products, in addition to the reviews by the Assessment Division, the Chief Statistician, and the NCES Commissioner, undergo a peer review (either internal or external) organized by the

IES.

To the extent possible, NAEP designs, samples and analyses are conducted to maintain trend over time.

Aspects of the NAEP design may change during this procurement period (2024-2029). Inclusion policy and an improved measure of socioeconomic status are some of the practices that are under review for possible changes. Another possible change includes increasing the precision of measurement particularly at the lower and upper ends of the student ability distribution. In addition, as outlined below, there are several challenges that will need to be addressed during the procurement period. Consequently, there will be the need for flexible approaches to design modifications as NAEP’s role continues to evolve either through changes in legislation, Governing Board and Department of Education policies, technology, or the fields of educational measurement, education, or schooling.

Current Operating Cycle

The project consists of preparing for and conducting parallel assessments in different stages1.

Special studies and pilot testing are integrated into the operational assessments, as needed. All

Coalition contractors are always actively involved during the calendar year and activities are often completed on tight timelines. The following paragraphs provide a sampling of the scheduling complexities.

• The contractors responsible for the design of the assessment must develop a preliminary design up to four years in advance of the assessment so that all other contractors can prepare. The sampling contractor, for example, must begin the sample selection process in January the year before the assessment so states can be notified in May for the subsequent school year.

• The item development contractor will be writing, reviewing and pilot testing items for four to six content areas across grades 4, 8 and 12 each year. Items are reviewed by NCES, a standing committee of content experts, the Governing

Board, and the states throughout the year.

• For the data collection contractor, the peak period is January to March, when substantial numbers of additional people are employed to collect data from students. But other times of the year, staff will be involved in developing training

1 When there is a new framework being created by the Governing Board, the cycle takes five or more years.

ix for data collectors, conducting the high school transcript studies, and working with private schools.

• The reporting contractor must meet a tight six-month or twelve-month deadline for all initial report releases. In addition, the contractor analyzes and drafts other reports and conducts other studies or analyses throughout the year.

• The contractor for materials and distribution begins their process of compiling materials in the late spring for distribution during the following winter based on sample schools identified in early May and sample students identified in the

October – November period. The contractor is responsible for 24-hour turn-around of additional materials during the data collection in the January – March period.

• The contractor supporting the 50 state and 27 district coordinators for their various activities throughout the sampling, data collection, and reporting processes maintain weekly contact via electronic media, conduct three (3) to six

(6) in-person workshops each year, and provide orientation to NAEP for the 20-

25 percent of new coordinators each year.

• The web contractor supports the release of The Nation's Report Card, online technical documentation and other key updates to the public websites regarding

NAEP and assessment resources. This contractor also hosts the system on which the assessment platform lives and transmits data and is responsible for developing and maintaining several internal applications and tools that support the NAEP program.

• The NAEP Platform Development contractor implements, supports, maintains, and enhances a NAEP digitally-based assessment (DBA) Platform and supporting technology systems. NAEP’s DBA Platform comprises the full end-to-end technical framework, or environment, that delivers the NAEP assessment to students.

Challenges

NAEP’s primary purpose is to provide a fair and accurate measure of student achievement in various content areas and to report trends over time. At the same time, NAEP must be responsive to environmental and policy changes and to the emerging needs of national and state policymakers. NCES and the Coalition contractors must not only administer required assessments based on an established schedule and with designated populations but also face the competing priority of responding to changes, some of which cannot be anticipated.

For example, innovations in better and cheaper technology and their integration into the classroom and out-of-school learning experiences of the country’s students dictate that developing an assessment for both national and state assessments must be forward-thinking.

Creative development must consider cutting-edge technology, equity and fairness for students, ease of use, timeliness of administration, and flexibility of the jurisdiction environment.

x

NAEP will be challenged to identify the type of infrastructure that will be necessary to maintain a standard, unbiased administration while maintaining high levels of item security, and low operational burden and cost. The test administration software must allow seamless transition from assessing any of the NAEP digitally-based assessments to allowing these digitally-based items to function in the NAEP Questions Tool.

In addition to delivering assessments via appropriate technology, it will become increasingly more important that the application of technology to all parts of the NAEP system is updated to keep pace with the latest industry advances in a pragmatic and cost-effective manner. Many of

NAEP’s administrative processes can be paperless.

Whenever commercial-off-the-shelf (COTS) software is appropriate and available, it should be given strong consideration in the evaluation of solutions. In addition, some custom systems may also be developed. All these systems will have to be maintained and enhanced so that NAEP can easily incorporate new innovations in hardware and software with minimal impact on operations and/or budget.

Other challenges that will have to be addressed during this procurement period include the following:

International comparisons. There is a growing demand by national, state, and local policy makers and the public to know how their schools compare not only to the nation and states but to other countries. There is a recognition that our students will be competing with students from this country, as well as students worldwide. NAEP, the only state-representative assessment, is in an excellent position to serve as the link with international assessments to provide state and district estimates for comparison with other countries.

Preparedness for Post-Secondary Education and Work. A different type of challenge arises from the call for the use of the 12th grade NAEP reading and mathematics assessments in a way that captures and measures preparedness for participation in post-secondary education, the military, and/or the workplace.

Universal Design. NAEP assessments should be developed to maximize accessibility for all students to the test items.

Adaptive testing. Adapting which test blocks are administered to an examinee holds the promise of providing better measurement of students across the full range of student performance.

Changing state content standards. States content standards are continuously changing in a post

Common Core State Standards initiatives environment. States have continued to modify standards and assessments during a time of changing accountability measures.

Trend. Critical to NAEP’s mission of reporting on the progress of education is maintaining trend. However, to remain relevant, NAEP must respond to changing environments. NAEP routinely faces this challenge through bridge studies that provide statistical evidence for continuing trend lines. However, with the major shifts in what and how NAEP assesses student performance, there may be other more efficient and cost-effective methods that can be used.

xi

Procurement Principles

There are six primary principles the Coalition contracts shall adhere to in this procurement:

1. NAEP data must be accurate, of high quality and on time.

2. NAEP must continue to lead the -field of large-scale assessment, especially in meeting the upcoming challenges.

3. NAEP must adapt to rapidly-changing environmental factors.

4. NAEP must be designed in a way that maintains reporting trends over time.

5. NAEP must improve efficiencies in both costs and burden on participants.

6. NAEP must ensure that the diversity of a workforce that characterizes the full range of NAEP stakeholders is represented at every level of NAEP operations.

To do so NAEP must make sure that every attempt possible is made to incorporate professional staff who represent that diversity and that each of the

NAEP contractors increase the participation of small and minority-owned businesses within each contract.

Procurement Components

As seen in the following organizational chart (Exhibit A), the new NAEP Coalition

Procurements will consist of ten critical path components with each component a separate contract award. Ten of the components are described in tasks 1 through 10 in this SOO, with each task corresponding to a separate contract. Task 8.3 describes activities common to all contractors and that shall be included in each offeror’s proposal.

The components are as shown in Exhibit A.

Exhibit A: NAEP Coalition Components

1. Program and

Operations

Coordination

Supports overall strategic planning and design for NAEP;

ensures coordination among Coalition contractors; maintains data for tracking program progress; and provides logistical support for complaints and substantive comments.

2. Sampling and

Weighting

Selects samples; prepares sampling weights.

3. Design, Analysis and Reporting

Designs all pilot and field tests, operational assessments, and special studies; analyzes data ensuring reporting of valid results;

proposes and prepares psychometric and statistical analyses compatible with previous NAEP methodologies; specifies data needed to meet the goals for reporting; and prepares reports.

xii

4. Assessment

Content

Development

Develops cognitive items, scoring rubrics and survey questions;

assists in the training of scorers; conducts cognitive interviews/small-scale pilots of items, rubrics, and survey questions; translates items and survey questions; and conducts item reviews

5. Scoring Human and AI-scoring of student responses.

6. NAEP Platform

Development

Develops NAEP assessment delivery platform utilizing a state-of-the-art, age-appropriate user interface (UI) and overall user experience (UX) that is consistent, intuitive, and accessible across all subjects and grade levels

7. NAEP Support and Service Center

Provides support, training, and resources to state and TUDA coordinators to ensure the accurate and timely sampling, administration and reporting of NAEP in each state and TUDA district.

8. Data Collection Administers assessments and collects data for pilot and field tests, operational assessments, and special studies; ships completed assessment materials to the scoring sites; conductions

High School Transcript Study.)

9. Program Support

Management

Manage the NAEP program in alignment with Project

Management (PM) best practices to ensure proper scheduling, quality control, risk management and communication across contractors and handoffs.

10. Web

Development, Operations, and

Maintenance

Supports

Develops, implements, and supports public and private web applications and services; identifies and recommends emerging technologies; monitors compliance with all NCES web cybersecurity requirements; and ensures timely and accurate release of web products.

As seen in Exhibit B, the Coalition includes three sets of contracts considered as teams within the Coalition and to be competed as teams and two contracts that will be competed as standalone under separate RFPs. This is in contrast with prior competitions for NAEP contracts in which potential bidders formed teams to bid on the whole set of contracts. Separating the larger

Coalition into smaller teams and standalone contracts for the competition is intended to make it easier for potential bidders to put together competitive bids and provide the government the best and lowest cost proposals.

Once the competition for contracts has been completed, all winning contractors must work together as a Coalition to ensure the successful implementation of NAEP. However, the three teams of contracts will be competed as teams because the hand offs between contractors within each team are so critical that vendors must develop relationships and plans before the competition to help ensure success on the critical hand offs within teams.

xiii

Exhibit B. The NAEP Coalition

From left to right in Exhibit B below, the teams are Assessment Content, Design and

Coordination, and Data Collection.

• Assessment Content: This team includes separate contracts for Assessment Content

Development, Scoring, and NAEP Platform Development that require close collaboration to ensure the successful and efficient development of the instruments for undertaking

NAEP, including the scoring of student responses.

• Design and Coordination: This team includes separate contracts for Program and

Operations Coordination, Design, Analysis, and Reporting, and Sampling and Weighting.

This is the core team for supporting the overall design of NAEP and coordination among contractors across the Coalition and requires real-time, close coordination across team contractors.

• Data Collection: This team includes separate contracts for Data Collection and the

NAEP Support & Service Center—two contracts that must work hand-in-glove for successful administration of NAEP in the field.

The Program Support Management and Web Development, Operations, and Maintenance

Support contracts (shown in gray) will provide key support for the Coalition and are involved with key hand offs within the Coalition but may not require partnership with other vendors pre-bidding to ensure successful participation in the Coalition post-bidding.

Each of the three team sets of contracts will be bid as its own team, with contractors working together to submit a unified group of proposals that will be accepted or rejected in their entirety.

Any organization interested in making an offer on one or more components within a team must xiv seek out other organizations that are recognized for complementary expertise to collaborate on a team proposal. The other two contracts will be standalone. NCES will award ten contracts, one for each component, with more than one contract being awarded to a single organization as appropriate. The contractors of all ten contracts will be required to develop agreements to ensure successful collaboration as an overall coalition.

Please note that other, non-Coalition contracts also support NAEP. See Exhibit C for a list of non-Coalition contracts. Additional collaboration between Coalition and non-Coalition contracts—not just within the Coalition contracts—is vital to the successful implementation of

NAEP.

Exhibit C: NAEP Contracts not Included in the Coalition

• Publications, Outreach and

Dissemination Support (POODS)

• Procurement of Research

Evaluation and Statistics Task

Orders (PRESTO)

• NAEP Logistics

• NAEP-State Technical and

Operations Cooperative

• National Indian Education Study

(NIES) Support Services Contact

1. Overview of the Work

Independently, and not as an agent of the United States Government, the Contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below. The work to be covered under this SOO begins with the award of the contracts in 2024 with completion of the 2024 assessments and ends in 2029 with development work for the 2030 assessments and beyond. Each assessment cycle includes field/pilot tests, and the operational assessments.

In November 2023, the Governing Board set the current schedule of assessments as presented in

Exhibit D (see https://www.nagb.gov/naep/assessment-schedule.html).

https://www.nagb.gov/naep/assessment-schedule.html xv

Exhibit D: NAEP Schedule of Assessments

Year Subject

National

Grades

Assessed

State Grades

Assessed

TUDA Grades

Assessed

2024 Reading

Mathematics

Science

Transcript Studies

4, 8, 12

4, 8, 12

4, 8

2025 Long-term Trend ~

2026 READING

MATHEMATICS

Civics

U.S. History

4, 8

4, 8

4, 8

4, 8

2028 Reading

Mathematics

SCIENCE

Transcript Studies

4, 8, 12

4, 8, 12

4, 8

4, 8, 12

4, 8, 12

4, 8

4, 8

2029 Long-term Trend ~

2030 Reading

Mathematics

CIVICS

U.S. HISTORY

4, 8

4, 8

8, 12

8, 12

4, 8

4, 8

2032 Reading

Mathematics

Science

WRITING

Transcript Studies

4, 8, 12

4, 8, 12

4, 8, 12

4, 8, 12

4, 8, 12

4, 8, 12

2033 Long-term Trend ~

Notes:

(1) Grades tested are 4, 8, and 12 unless otherwise indicated, except that long-term trend assessments sample students at ages 9, 13, and 17 and are conducted in reading and mathematics.

(2) Subjects in BOLD ALL CAPS indicate the years in which a new framework is planned for implementation.

xvi

Assessments to be Implemented

The assessment schedule (Exhibit D) provides a list of the assessments to be administered during this contract period, as well as future assessments for which development may be required in the contract period. However, please note that the schedule is subject to change. For example, the schedule may change to place NAEP assessments back on an “odd-year” schedule, as it was before the pandemic caused postponement of the 2021 NAEP assessments to 2022. If this occurs, assessments planned for fall 2025 and beyond are likely to be moved out one year. Any changes in assessment schedule will be reflected in updates to the NAGB assessment schedule webpage

(https://www.nagb.gov/naep/assessment-schedule.html).

The flow of work on a NAEP assessment begins long before the operational assessment year and continues afterwards. Assessment content development begins several years before an assessment is fielded operationally; items are piloted typically two years before fielding operationally; and most operational innovations undergo proof-of-concept and field test stages in the years before operational implementation. Analysis and sampling plans must be developed well in advance of assessments and data collection operations, including contacting states, districts, and schools, as well as recruiting and training field staff will occur well before assessments go into the field. Reporting and dissemination, including the preparation of data files and documentation to be shared with researchers, as well as in-depth analysis by the NAEP program, extends well beyond the initial release of an assessment’s results. Further details on the scope and timing of work are included in the SOO tasks for each contract.

About the Statement of Objectives

The next part of the SOO is organized into three tasks, Task 4. Assessment Content

Development (ACD), Task 5. Scoring and Task 6. NAEP Platform Development (NPD). While each contract will have specific responsibilities, there is a set of more general activities and associated tasks that are common to all the contracts described at the end of each task. Each general activity and the associated tasks are relevant to each contract and shall be included in each contract offer. The work below includes expectations for the ACD, Scoring and NPD team.

This team will work with the other teams and Coalition members to ensure a successful implementation of the NAEP program.

https://www.nagb.gov/naep/assessment-schedule.html

4.0 Task 4 - Assessment Content Development (ACD)

The purpose of this contract is to develop content for NAEP’s Digitally-Based Assessments

(DBA) and Paper-Based Assessments (PBA2), including development and maintenance of the cognitive assessments, survey questionnaires, and supporting content required to successfully administer NAEP (e.g., student directions, ancillary materials). The assessment content development contractor shall:

• Implement efficient, effective, and systematic processes for the development and review of new cognitive content (including scoring rubrics), survey questions, and supporting assessment materials (e.g., ancillary documents) that align with NAEP’s goals for online and device agnostic delivery and NAEP framework requirements.

• Develop and maintain pools of content that seamlessly operate within NAEP’s assessment and questionnaire delivery platforms provided by the NAEP Platform

Development (NPD) contractor (including the platform’s digital universal design elements, tools, and accommodations), support measurement and reporting across the full range of student performance, and are fair, accessible, valid, and engaging to students.

• Develop and maintain survey questionnaires that support the reporting and understanding of student achievement, and are efficient, engaging, fair, and accessible to respondents.

• Configure cognitive content and survey questions into blocks, including supported accommodations blocks, according to the assessment design provided by the Design, Analysis and Reporting (DAR) contractor.

• Review and certify content and supporting ancillary materials ahead of each administration.

Overview

NCES has a commitment to ensuring all digital and paper-based content developed for the NAEP assessments is accessible, innovative, and engaging, while efficiently, validly, and reliably measuring the full range of student performance and aligning to NAEP frameworks. This includes – but is not limited to – the following goals for the Assessment Content Development

(ACD) contract:

• Integrate considerations for diversity, equity, inclusivity, and accessibility within all assessment development processes and procedures and ensure that NAEP content is free from bias.

• Utilize mechanisms and approaches to efficiently develop robust pools of items and scoring rubrics (e.g., automated item generation, multiple item versions with varying difficulty) while also maintaining fidelity to validity and reliability.

2 All NAEP assessments except for the following have transition to DBA: Grades 4 and 12 Civics and U.S. History;

Ages 9, 13 and 17 Long-Term Trend.

• Advance the development of survey questions that support diagnostic reporting and understanding of student achievement, and achievement trends over time.

• Support the NAEP Platform Development (NPD) contractor to create efficiencies and technical solutions for the development and implementation of content for NAEP’s digital assessment and questionnaire platforms.

• Implement approaches for development of content that measure the full range of student performance to improve the capacity of NAEP to report performance for all students, and particularly for improving measurement at the lower end of the performance scale.

• Propose technical approaches, solutions, and innovations that advance automation and other efficiencies in assessment development processes (e.g., automated item and content generation, translation).

• Examine assessment data (pilot and operational) to enhance the quality of the cognitive content and survey questions.

Scope

The following tasks represent the scope of this contract. The contractor must perform the tasks for each operational assessment, pilot test, and special study (e.g., National Indian Education

Study (NIES)) to be conducted in the 2024-2029 period of performance (see Exhibit X for

Assessment Development Activities). To accomplish these tasks, the ACD contractor will need to work closely with all NAEP contractors, and particularly with Design, Analysis and Reporting

(DAR), Scoring, NAEP Platform Development (NPD), and NAEP Data Collection (NDC).

Exhibit D, NAEP Schedule of Assessments on page 13 in the Overview task, indicates the assessments that are to be considered in this procurement. Note that the schedule is subject to change. The assessments to be conducted during this contract period will be based on frameworks established by the National Assessment Governing Board (NAGB).

Assessment Content Development Tasks

4.1 Manage Assessment Content Development

4.2 Develop and Maintain Content for the Cognitive Assessments

4.3 Develop and Maintain Content for the Survey Questionnaires

4.4 Prepare for the Assessment Administration

4.5 Convene External Reviews

4.6 Conduct Research to Improve Assessment Development

4.7 Support IT Business Processes

4.11 General Activities

4.1 Manage Assessment Content Development

The NAEP assessment development cycle includes the following phases: (1) assessment design and development planning, (2) content development, including iterative reviews by a wide range of experts (e.g., practitioners), pretesting in front of target audiences, and platform implementation reviews, (3) block and booklet assembly, (4) pilot administration preparation, including in-system platform milestone reviews and certification, (5) piloting and post-piloting activities, including scoring and analyses, (6) selection of items for the operational assessment and block and booklet assembly, (7) operational administration preparation (i.e., in-system platform milestone reviews and certification), and (8) operational administration and post-operational activities, including scoring, analyses, and item release. The development cycle ends with a systematic review of the item pool to identify the item development needs for the next development cycle. Collaboration and coordination with other NAEP contractors are essential to ensuring the success of the NAEP program. Furthermore, research and development are integrated throughout each phase of the assessment development cycle. For example, moving towards multi-stage adaptive testing may require additional targeted development and changes in the block assembly approach.

Under this task, the contractor shall manage all assessment content development activities as part of the NAEP assessment development cycle, across the various tasks within this contract and in collaboration other NAEP contractors. The contractor shall develop and monitor detailed processes and procedures for implementing NAEP’s assessment development cycle that ensures that all NAEP assessment content is valid, reliable, accessible, and equitable, and meets best practices for assessment development. The contractor shall also establish and maintain, with approval from NCES, development and review protocols to ensure that all assessment development content (e.g., items, questions, indices, tasks, passages, tools) are constructed to reflect and enhance NCES’s program goals for diversity, equity, inclusivity, and accessibility. Of critical importance to this task is the management of contract deliverables. Therefore, contract management personnel shall be responsible for the tracking and monitoring of contract deliverables to ensure they are completed in accordance with the project schedules and meet quality standards.

The contractor shall develop a Project Management Plan that provides details on how the end-to-end assessment content development activities will be executed, monitored, and controlled. The plan shall document robust processes, protocols, and staffing to develop, implement, coordinate, and monitor all content development activities in support of this contract. The plan shall also include an overall assessment content development schedule that integrates key milestones and hand-offs from various assessment content project schedules and other NAEP contractor schedule dependencies.

Cybersecurity is a major concern for NCES and the Government, and externally hosted software and systems that include confidential information (e.g., test items and booklets, embargoed data) and personally identifiable information (PII) must be granted an Authority to Operate (ATO).

The NAEP system has an ATO managed by the Web Development, Operations, and

Maintenance Supports (WDOMS) contractor that emphasizes and enforces the importance of security for all NAEP content and data. To ensure the security of the NAEP content, it is expected and preferred that NAEP content developed using external IT software shall live within the ATO boundary of the NAEP system.

4.2 Develop and Maintain Content for the Cognitive Assessments

Under this task, the contractor shall develop and maintain cognitive content for each pilot test, operational assessment, and special study administration for the period of performance. More information on the NAEP cognitive instrument can be found at this link. Development of cognitive content shall include developing items, tasks, and scoring rubrics. The contractor shall adhere to the highest standards in assessment development practices and implement efficient, effective, and standardized development processes across all NAEP assessments. The contractor shall ensure that NAEP cognitive assessment content are:

• Aligned with the NAEP frameworks and assessment item specifications,

• Valid measures of subject-specific knowledge and skills that span the range of student performance,

• Equitable and free from bias against any group of students,

• Grade appropriate,

• Linguistically accessible to all students,

• Consistent with the principles of universal design and comply with Section 508 requirements,

• Compatible with digital platform and delivery requirements, i.e., online and device agnostic,

• Statistically sound, i.e., meet commonly accepted item statistical standards (e.g., difficulty, discrimination, and differential item functioning (DIF)),

• Engage students to do their best and allow students the opportunity to demonstrate their skills and knowledge in the construct of interest, including students with disabilities

(SDs) and English learners (ELs),

• State of the art and follow best practices in education technology for digital content development.

The development and maintenance of assessment content shall be conducted in close collaboration with the DAR and NPD contractors. New development shall be purposeful and targeted to fulfill the reporting and design objectives for each assessment identified by the DAR contractor. The ACD contractor shall collaborate with the DAR contractor to determine:

• The number and types of items/tasks needed to fulfill measurement objectives and reporting goals established by the DAR contractor.

https://nces.ed.gov/nationsreportcard/tdw/instruments/cog.aspx

• The number and types of items/tasks needed to achieve sound measurement at each

NAEP achievement level and across the student performance distribution.

• The number of piloted items/tasks in the item pool that are available for use.

• The number of items/tasks that require substantive revisions after pilot testing (such items must be re-piloted to verify their psychometric properties before being used in an operational assessment).

• The number of and which items to release after each operational assessment.

New development shall employ an efficient development process that utilizes technological advances in content development, including authoring tools (e.g., content authoring systems, benchmark components) to ensure new content, whether developed internally or externally, can be implemented efficiently and effectively into the digital assessment platform. The ACD contractor shall collaborate with the NPD contractor on the following:

• Development and execution of end-to-end assessment development processes for authoring, implementing, and reviewing content within the digital assessment platform.

• Development of content and platform specifications and new features to support NAEP assessment content and NAEP’s goals for device agnostic assessment, including updating

NAEP style guides as needed.

• Development of externally developed content, if applicable (e.g., passages, multi-media stimulus, scenario-based tasks).

• Development of digital assessment resources (e.g., universal design elements, digital tools, interactive item components, simulations).

• Confirmation of the accuracy and functionality of the content in the NAEP platform, following implementation of content by the NPD contractor.

• Development of a schedule to support content development and platform implementation across all stages of development (e.g., external reviews, pretesting) and multiple assessment cycles.

NCES’s goals for NAEP digital cognitive assessment development are to leverage technological approaches to develop and administer content that aligns with NCES’s standards and NAEP

Frameworks, and maintains NAEP as the gold standard for large-scale assessments efficiently and effectively. The contractor shall employ innovative approaches that reflect technological advances in assessment development to achieve these goals.

The following subtasks are included as part of cognitive assessment development.

4.2.1 Develop the Framework Implementation Plan

The National Assessment Governing Board periodically reviews existing NAEP frameworks to determine whether updates are required. If a substantial update has been made to a NAEP framework or a new framework has been created, the contractor shall conduct necessary steps and procedures to implement the framework for the relevant content area assessment (e.g., review and code existing item pool to the new framework). As part of this task, the contractor shall also develop a Framework Implementation Plan. The Framework Implementation Plan serves as a comprehensive road map for carrying out the relevant activities for assessment design, development, and implementation, and record of the actions and procedures necessary to ensure that the successful implementation of NAEP cognitive assessments and survey questionnaires are aligned with new and updated frameworks.

The Framework Implementation Plan shall include, at a minimum, the following components:

• A review of the requirements from the framework that can be implemented within

NAEP’s existing design and technology.

• Identification of new or updated requirements in the framework that require additional research to ensure correct implementation and a plan for researching these requirements.

• Alternative methods for achieving the requirements in the framework if existing NAEP design and technology do not allow for the attainment of these goals.

• An analysis of the trend item pool (including both cognitive and survey questionnaires) to determine the amount of trend content that can be carried forward under the new or updated framework and the resulting gaps in the item pool to accomplish the framework’s requirements.

• A roadmap across multiple development cycles for implementing the framework requirements.

The ACD contractor shall develop the Framework Implementation Plan in collaboration with other relevant NAEP contractors. The contractor should assume a Framework Implementation

Plan will be required for each new or updated framework shown on the NAEP assessment schedule through the period 4 years after the end of the contract period.

4.2.2 Develop the Cognitive Assessment Content Development Plan

Typically, the ACD contractor initiates development planning at least two years prior to the piloting of new items.