Cost structure review SOO draft 092822.pdf

NAEP Cost Structure Review Statement of Objectives 1

DRAFT STATEMENT OF OBJECTIVES (SOO)

Review of Cost Structures for Administration of the National Assessment of Educational Progress

August 24, 2022

I. Purpose

The purpose of this procurement is a thorough, independent expert review of the cost structures for the administration of the National Assessment of Educational Progress (NAEP), including all work processes and their impact on cost, from study design through dissemination of data. Results of the cost structure review will be used to re-design NAEP processes to improve cost efficiency, while maintaining NAEP’s high standards for quality, timeliness, and the accurate and reliable measurement of trends.

Results of the review will be used also to inform decisions about the structuring of the array of contracts through which NCES implements NAEP. Currently, NCES uses a 9-contract Alliance of vendors working together as a team on work that is highly interdependent, as well as 5 standalone contracts, and more than

50 contracts with states and other jurisdictions to fund the work of NAEP State Coordinators embedded in state departments of education. Over the years, work has been pulled out of the Alliance contracts and procured through smaller independent contracts to increase small business engagement, improve competition, and grow the number of vendors with capabilities in NAEP work. NCES seeks to continue to make strides in these areas without increasing risk to the success of the program.

NCES is seeking a vendor with demonstrated experience and expertise in the following:

• Conducting cost structure analysis for large-scale assessments, particularly group level assessments in addition to state assessments or large-scale formative and summative assessments focused on individual scores,

• Applying and interpreting psychometrics procedures for large-scale assessments,

• Analyzing and advising on matrix assessment sampling and data collection methodologies, digital assessment platforms, assessment scoring, data visualization and reporting, and those features that distinguish NAEP from other assessments, such as the need for maintenance of measurement of trends.

NAEP Cost Structure Review Statement of Objectives 2

II. Overview of NAEP

Legal Context

NAEP, often called The Nation’s Report Card, is a congressionally mandated project of the National

Center for Education Statistics (NCES), located within the U.S. Department of Education. Legal requirements for governance and administration of NAEP are described in U.S. Code, Title 20, Chapter

76, Subchapter III - National Assessment of Educational Progress. The law defines the purpose of the

NAEP as “to improve the effectiveness of our nation's schools by making objective information about student performance in selected learning areas available to policymakers at the national, state and local levels. To enhance its utility, such information should be both representative and comparable, and should be maintained in a manner that ensures the privacy of individual students and their families.”

The National Assessment Governing Board (NAGB), an independent, bipartisan organization made up of governors, state school superintendents, teachers, researchers, and representatives of the general public, sets policy for NAEP (see 20 U.S. Code § 9621). NAGB provides direction on the content of NAEP, including developing assessment frameworks for each subject assessed, as well as setting the assessment calendar, which schedules when assessments are administered.

NCES, one of thirteen principal Federal statistical agencies, is responsible for the implementation of

NAEP, including design and development of assessment and survey instruments, sampling for the administration of NAEP, recruitment and data collection, processing, scoring, and analyzing data, and reporting results. The NCES Commissioner has contracting authority for the implementation of NAEP

(see 20 U.S. Code § 9622).

Key Aspects of NAEP

NAEP is the largest nationally representative and continuing assessment of what students in public and private schools in the United States know and can do in various subjects. Since 1969, NAEP has been a common measure of student achievement across the country in mathematics, reading, science, and many other subjects. Depending on the assessment, NAEP report cards provide national, state, and some district-level results, as well as results for different demographic groups. NAEP data are also used in special studies conducted by NCES. These have included comparisons of proficiency standards across state assessments; insights from high school transcripts, including courses taken and credits earned; and in-depth looks at how different demographic groups perform across different types of schools.

Policymakers, researchers, and educators use NAEP results to inform educational improvements across the nation, within states and TUDA districts, and for various student groups. Parents, media, and the general public use NAEP results to monitor educational progress in their communities and compare https://www.law.cornell.edu/uscode/text/20/chapter-76/subchapter-III https://www.law.cornell.edu/uscode/text/20/chapter-76/subchapter-III https://www.law.cornell.edu/uscode/text/20/9621 https://www.law.cornell.edu/uscode/text/20/9622

NAEP Cost Structure Review Statement of Objectives 3 performance with other regions of the country. NAEP also provides states with a benchmark to target important efforts that raise the bar for student achievement and ensure that students have equal opportunities to succeed.

High Interest in Maintaining NAEP’s Trend Measures

A critical aspect of NAEP is the reporting of trends in student performance over time. NAEP’s long-term trend assessments in mathematics and reading for students ages 9, 13, and 17 have provided unbroken trends in measurement of progress in student achievement for the nation since the early 1970s. The mathematics and reading assessments in grades 4 and 8 have provided unbroken trends for the nation and many states since the early 1990s. These trend measures of educational progress are valued extremely highly by NAEP stakeholders and require the highest standards in assessment design, administration, psychometrics, and sampling and other statistics.

Need for Low Burden to States, Schools, Students

Although NAEP has components that are legally mandated for states, school and student participation is voluntary. Moreover, NAEP assessments are low stakes for students as NAEP scores are always reported as group level averages and never generated for individual students. The low stakes nature of the assessment means that even if the students participate in the assessment, they might not be motivated to show their best effort if they perceive the assessment to be too burdensome. Scores are not reported at school level either and although average scores are reported at state level, there are no consequences attached to how high or low those average scores are. For these reasons, the assessments must be designed so that they are not burdensome to students and schools, not costly to states, and will allow the

Program to maintain trend.

NAEP Contracts

NAEP is implemented largely through a set of nine contracts called the “NAEP Alliance,” to reflect the interconnected nature of the work of the contracts. The Alliance contracts cover activities that are highly interdependent and have a significant risk for failure if contractors do not cooperate effectively. The current Alliance contracts range from support for NCES in planning and coordinating NAEP work to design and development to the administration of the assessments, data collection, data analysis and reporting results (see table 1). More information about the scope of work of the current Alliance contracts and how the work interrelates across contracts is in the SAM.gov solicitation notice.

Other work is contracted through several independent, non-Alliance contracts, including contracts for communications and outreach, logistics, secondary analysis and reporting, the funding of the NAEP state coordinators, and some quality assurance work. These independent non-Alliance contracts have been https://sam.gov/opp/4abb678b2b573e778af80498c57a3492/view

NAEP Cost Structure Review Statement of Objectives 4 pulled out of the alliance contracts to increase small business, improve access to a greater array of vendors, and enhance quality control.

Mission of the Cost Structure

Review

The mission of the cost structure review is a thorough, independent review of the processes used to implement NAEP, how the processes are structured in the

NAEP Alliance contracts, and how the processes and their structures influence cost, quality, and timeliness.

Results of the cost structure review will be used to re-design NAEP processes, to improve cost efficiency, and to re-think the structuring of the array of NAEP contracts to improve competition and increase small business engagement in NAEP, while maintaining NAEP’s high standards for quality and timeliness, and the maintenance of trend. The major product of the cost structure review will be a report (1) identifying key aspects of NAEP’s implementation processes and cost structure, (2) describing NAEP’s processes and cost structure in comparison with other, similar largescale assessments, and (3) recommending changes to NAEP’s processes and contract structure and describing any tradeoffs of cost efficiency for quality and timeliness implied by the recommended changes (4) recommending contract structures that would improve competition

Table 1. Current NAEP Alliance Contracts

Contract Title Scope of Work

Planning and Coordination

Coordination; data for tracking progress; logistical support for complaints and substantive comments

Program Support Management

Scheduling, quality control, risk management and communication across contractors and handoffs

Design, Analysis and Reporting

Design of studies; analysis;

specification of data needed to meet reporting goals; reports

Item Development Cognitive items, scoring rubrics and survey questions; assistance in training of scorers; cognitive interviews/small-scale pilots;

translation

Sampling and Data Collection

Sampling; sampling weights;

administration of assessments; data collection; transcript studies

Support and Service Center

Support, training, and resources to state and TUDA coordinators

Materials, Distribution, Processing and Scoring

Assessment and auxiliary materials;

distribution of assessment booklets and materials; scoring training materials; scoring

Web/Technology Development, Operations, and Maintenance

Internet-related applications and services; web and other computer-based products and services

NAEP Platform Development

Assessment delivery platform

NAEP Cost Structure Review Statement of Objectives 5

III. Period and Place of Performance Period of performance is four months from the start of the award of the contract. Work must be performed in secure environments, because of the sensitivity of source information the contractor will be reviewing.

IV. Background Resources Key background information on NAEP is available at the websites listed in table 2.

Table 2. NAEP-Related Websites

NAEP legal context 20 U.S. Code Subchapter III – National Assessment of

Educational Progress

About NAGB www.nagb.gov

About NCES NCES.ed.gov

NAEP websites NCES NAEP site

Shared NAGB/NCES NAEP site

Solicitation for current NAEP Alliance contracts

Solicitation notice at SAM.gov

Recent National Academies of Science, Engineering, and Medicine reports

A Pragmatic Future for NAEP: Containing Costs and

Updating Technologies (2022)

A Vision and Roadmap for Education Statistics (2022)

V. Performance Objectives:

• Review of work processes to identify key aspects of the cost structure

• Comparative analysis of current NAEP and other large scale assessment cost structures

• Evidence-based recommendations for re-designing NAEP processes to improve cost efficiency, while maintaining high quality, timeliness, and the accurate and reliable measurement of trends

• Evidence-based recommendations for structuring the array of alliance and independent contracts for improved efficiency and competition without increasing risk of the success of the program

VI. Operating Constraints

Some detailed cost data may be proprietary and potentially not accessible for the vendor’s analysis.

https://www.law.cornell.edu/uscode/text/20/chapter-76/subchapter-III https://www.law.cornell.edu/uscode/text/20/chapter-76/subchapter-III https://www.nagb.gov/ https://nces.ed.gov/ https://nces.ed.gov/nationsreportcard/about/future_of_naep.aspx https://www.nationsreportcard.gov/highlights/science/2019/ https://sam.gov/opp/4abb678b2b573e778af80498c57a3492/view https://nap.nationalacademies.org/catalog/26427/a-pragmatic-future-for-naep-containing-costs-and-updating-technologies https://nap.nationalacademies.org/catalog/26427/a-pragmatic-future-for-naep-containing-costs-and-updating-technologies https://nap.nationalacademies.org/catalog/26392/a-vision-and-roadmap-for-education-statistics

Mission of the Cost Structure Review