Attachment D - Design and Coordination Team SOOs (1-3).docx

Tasks 1, 2, & 3 (Team Contracts)

Statement of Objectives Design and Coordination Team Program and Operations Coordination Sampling and Weighting Design, Analysis and Reporting

National Assessment of Educational Progress

(NAEP)

2024-2029

1. Program and Operations Coordination (POC) The National Assessment of Educational Progress (NAEP) program is a complex suite of assessments carried out in an extensive, interconnected, and ongoing operation. The NAEP program includes operational assessments, pilot assessments and studies, other research & development intended to inform the National Center for Education Statistics (NCES) about the efficacy of a given process, and future planning, within a given year.

The Program and Operations Coordination (POC) contractor serves as the study director and provides key roles in supporting NCES in day-to-day program and operations coordination of NAEP, research and development, strategic planning, and the design and development of NAEP.

The POC contractor supports NCES in coordinating the core activities of the NAEP program across multiple contractors, as well as providing intellectual support for strategic development of the program. In this way, the POC contractor serves as the NAEP program’s key organizing force in support of NCES’s direction of the program.

As the contractor responsible for effecting the smooth integration of the various components of the NAEP program across multiple NAEP contractors and providing NCES key support for the strategic direction of NAEP, the POC contractor shall support NCES NAEP objectives to:

· Ensure NAEP’s high standards for quality and its position as the gold-standard in large-scale assessment, including:

· Maintenance of NAEP’s treasured 30-year trend lines in mathematics and reading at grades 4 and 8;

· The highest quality assessment and survey items in large-scale assessment;

· Smooth, trouble-free experiences for the states, districts, schools, educators, students and other stakeholders participating in NAEP studies;

· Accessible, informative, timely, relevant, objective, credible, error-free data for the wide range of stakeholders consuming NAEP data;

· Improve cost-efficiency of processes across NAEP so that NAEP continues to be the best value for money in large-scale assessment;

· Develop methods for constant innovation in all aspects of NAEP to improve NAEP’s utility, the experience of NAEP for participants and data users, and the program’s cost-efficiency; and

· Create the future in large-scale assessment through far-reaching research and development and a deep understanding of developments in education and technology and where NAEP can make the best contribution to the nation’s understanding of educational progress.

Scope The scope of this contract is reflected in the following tasks:

1.1 Support NCES strategic planning for NAEP

The contractor shall support NCES in developing the NAEP Strategic Plan to cover a five-year period from the start of the contract award, and shall also help revise the strategic plan, as necessary. To help develop the plan, the contractor shall develop procedures for scanning and monitoring NAEP's environment for changes in the large-scale assessment industry at the state, district, and international levels. Such procedures shall include identifying opportunities and threats that support or detract from efforts to carry out assessments as planned, to advance NAEP as the leader in large scale assessment, and to promote quality in assessment practices. The contractor shall also help identify and interpret trends and changes in the education environment, assist NCES in implementing procedures for forecasting changes and trends, and identify strategies to effectively manage the impact of such changes on NAEP stakeholders and contractors, through outreach and communication aimed at facilitating an understanding and acceptance of change. These actions should be the basis for creating a new plan, and for providing recommendations to update the plan.

They shall focus on high-level issues, such as changing state and international policies, rapidly emerging technological changes, educational inequity, declining participation in assessments at some grade levels or with some subjects, and the challenge of interpreting performance trends.

1.1.1 Support NAEP Strategic Plan document

The contractor shall support NCES in developing and maintaining the NAEP Strategic Plan. The initial Plan shall cover a five-year period from the start of the contract. Over time, the Plan shall continue to look forward over a five-year horizon. The contractor shall develop a first draft within the first month of the start of the contract and implement a review/revise/resubmit process to produce a final initial plan within three months of the start of the contract. The contractor shall develop and implement a process for revision of the Plan across the contract cycle, as well as tracking the program’s progress against the Plan. The Plan shall be a document that NCES can use effectively to communicate its vision of NAEP in the short and long-term; ensure that the vision is consistent with the program’s goals and developments in education, technology, and in large-scale assessment; plan activities to support development of the program toward this vision; and track the program’s progress. The process for development and maintenance of the Plan must ensure that the Plan continually serves these purposes for NCES.

1.1.2 Support the NAEP Executive Steering Committee

The contractor shall establish and maintain the NAEP Program Executive Steering Committee (Committee), comprising of Corporate Officers from all NAEP functional teams, the NCES Commissioner, NAEP Associate Commissioner, NCES Director of Strategic Partnerships, NCES Chief Psychometrician, and Assessment Division (AD) managers. The Committee’s purpose is to support development, maintenance, and tracking of progress against the Strategic Plan; report to NCES on the status of the implementation of NAEP work under the contracts; and provide other tactical and strategic support to NCES in the direction of the NAEP program.

The contractor shall schedule and convene the Committee meetings. As needed, the Committee shall meet up to four times annually. At the start of the contract and the start of each calendar year thereafter, the contractor will set tentative dates and times and draft preliminary agendas for the upcoming Committee meetings. Dates/times and agenda items may change as events arise throughout the year; however, the initial draft agendas and timing of the meetings shall reflect projected needs for the year as reflected in the Strategic Plan and NAEP assessment schedule.

The contractor shall also convene additional ad-hoc expert panels to meet with the Committee, as the need arises, up to four times per year. In past years NAEP assembled ad-hoc panels of experts from various relevant disciplines and industries beyond the large-scale assessment community to provide input for NAEP’s strategic planning. To ensure a diversity of perspectives on as broad a range of pertinent issues as possible, the composition of any panels of experts convened for such purposes shall vary according to need.

The contractor shall document all meetings (regular Committee meetings as well ad-hoc panels), including deliberations and resultant recommendations, and deliver them to the Contract Officer Representative (COR), and the Committee within two weeks of the meeting. The contractor shall monitor progress on action items identified during the meetings and make recommendations to NCES on approaches to operationalizing resulting recommendations of the Committee and ad-hoc panels. Other NAEP meetings, consisting of NAEP, and other NAEP program staff, will be described in more detail in section 1.2.3 (Coordinate planning and review meetings among NAEP contractors and NCES). The contractor shall report the status of action items and implementation of recommendations as part of the weekly communications with the COR.

1.1.3 Coordinate NAEP Research and Development

NAEP Research and Development (R&D) drives the NAEP program forward by identifying new and better ways to measure and report on student achievement and expanding what NAEP is capable of, enabling NAEP to retain its position as the gold standard for assessment and continue to serve as a model for other large-scale assessments. R&D efforts have in the past focused on validating and improving assessment practices, anticipating, and responding to external policy-related influences on the scope and nature of the assessments, leveraging other data sources to expand what NAEP can say about student achievement, and developing and refining psychometric and statistical procedures.

NAEP’s R&D is conducted by various contractors both inside and outside of the NAEP program and expert groups that focus on different aspects of NAEP. Table 1 shows the types of NAEP R&D that have historically been conducted and the contract(s) or activity under which they have been conducted during the current contract period (though not all activities are conducted under the current contracts). A list of published NAEP studies is available on the NCES website (https://nces.ed.gov/pubsearch/).

Table 1.1 R&D Across the NAEP Program

These many efforts need to be coordinated to ensure that they are, individually and collectively, aligned with the NAEP Strategic Plan and NCES’s priorities, that NAEP’s R&D portfolio is comprehensive and addresses areas of interest and significance, and that R&D overall is moving forward to advance the NAEP program. This coordination is also important for ensuring that outcomes of R&D are used by the NAEP program and communicated to NAEP stakeholders.

The POC contractor shall provide support to NCES in developing and maintaining a coherent R&D program to ensure that NAEP R&D efforts address priority areas, that special studies and research projects are conducted in coordination with other NAEP activities, as appropriate, and that R&D outcomes are communicated to stakeholders internal and external to the NAEP program and incorporated into NAEP processes when appropriate.

1.1.3.1 Develop, monitor, and maintain a comprehensive R&D agenda The contractor shall support NCES in developing, monitoring, and maintaining a coherent and comprehensive research agenda for a rolling five-year horizon that aligns with the NAEP Strategic Plan and is consistent with NAEP operational planning. The research agenda shall identify R&D to support innovations and improvements to NAEP methodologies and processes. Development of the research agenda shall take into consideration the foci of current and recent R&D conducted by NAEP Functional Teams and non-Functional Teams partners and expert groups and identify gaps and opportunities for new R&D efforts to address unmet needs and emerging issues given the state of the field. The R&D agenda shall inform decisions about NAEP R&D commitments. Working with an NCES inhouse R&D advisory committee, the contractor shall make recommendations for creating the required balance across various research issues and needs by taking a broad view of NAEP’s goals and advising on the formulation of a research agenda that will best serve NAEP.

NAEP R&D Landscape Review. To support establishing the R&D agenda, the contractor shall review and summarize current and recent NAEP R&D activities conducted across the Functional Teams and non-Functional Teams contracts and external panels, with input from contractors as needed. This review will provide a comprehensive view of NAEP’s R&D efforts to date, highlight alignment of R&D with NCES goals, identify any duplications of effort, and identify gaps and potential new directions considering the NAEP Strategic Plan. It will also take into consideration the state of the field and emerging technologies and innovations outside of NAEP (for example, artificial intelligence).

Develop and Maintain R&D Agenda. The Executive Steering Committee will consider the R&D summary report and identify priorities for an R&D agenda that aligns with the NAEP Strategic Plan and builds on current efforts and expands R&D to address new innovations and directions for NAEP. The contractor shall prepare a document that outlines the R&D agenda, including objectives, studies, and a timeline. The research agenda shall be reviewed continuously and updated on an annual basis considering NAEP R&D conducted to date, developments in the field, and emerging needs of the NAEP program. To support monitoring of the R&D agenda, the contractor shall work with the Program Support Management (PSM) contractor to (1) establish a searchable database housed on the NAEP IMS that includes information about all NAEP-related research activities conducted by both Functional Teams and non-Functional Teams members, including study proposals, timelines, and interim and final reports, and (2) create a means of showing progress towards accomplishing the R&D agenda, such as a dashboard. The POC contractor shall ensure the tracking database is kept up to date, will monitor progress on all R&D activities, and alert NCES to any risks to meeting R&D milestones.

1.1.3.2 Coordinate reporting on NAEP R&D progress and outcomes The objective of the NAEP R&D agenda is for the outcomes of R&D studies and research to advance the NAEP program by informing assessment design, development, operations, analysis, and reporting and to facilitate improvements in large-scale assessment methodology writ large. For this goal to be met, R&D efforts and outcomes need to be shared and incorporated into NAEP workstreams as appropriate and shared more broadly with the field. The POC contractor shall provide a coordinating role in the communication and dissemination of NAEP R&D across the NAEP Functional Teams and with non-Functional Teams contractors and external stakeholders.

The contractor shall facilitate sharing progress and results from ongoing R&D across the NAEP program with NCES and other internal NAEP stakeholders so that collectively the NAEP program can consider how to incorporate what is learned into NAEP, can identifying synergies across R&D activities, and can identify additional studies to further advance the research agenda. Information shared will also inform updates to the research agenda. The contractor shall recommend an approach that can effectively accomplish these goals.

The contractor shall also support efforts to promote and disseminate NAEP R&D to outside researchers and measurement experts, other large-scale assessment programs, and others for the purposes of informing the broader assessment community about new methodologies and innovations in assessment that NAEP is undertaking. The contractor shall recommend to NCES methods of collating and disseminating information about NAEP R&D to expand the reach of research efforts beyond NAEP, i.e., by publishing thematic compendia or journal issues, or hosting symposia. The contractor shall prepare a plan that provides recommended methods and timelines for sharing NAEP R&D throughout the five-year performance of the contract and describes how the proposed approaches will meet NCES’s goal of informing large-scale assessment more broadly.

The contractor shall provide support to NCES in reporting to Congress and other stakeholders on NAEP R&D plans, costs, and outcomes. The contractor shall prepare summary reports (up to four per year) as requested by NCES on targeted R&D activities, within and across Functional Teams contracts and including at a minimum, for R&D tasks, key milestones, progress, outcomes, and spending.

1.1.3.3 Coordinate Functional Teams special studies and other special studies NCES anticipates conducting various special studies to disseminate NAEP procedures for the operational assessment, during the 2024-2029 contract period. Some of these special studies will be conducted within the NAEP Functional Teams—Functional Teams special studies—while other special studies may include non-Functional Teams contractors—other special studies.

Functional Teams special studies will require input from multiple Functional Teams contractors. In the past, examples of this type of work have included such studies as the 2018 Oral Reading Fluency Study, the 2019 Mathematics Multi-stage Testing study, and proof of concept studies carried out to support NAEP’s transition to online and device agnostic assessment. The contractor shall coordinate effective collaboration among the multiple contractors involved in Functional Teams special studies. Such coordination shall include collaboratively working with Functional Teams contractors to:

· Develop recommendations memos and proposals for the conduct of special studies

· Suggest timelines for COR approval, then coordinate the integration of timelines for special studies into NAEP’s Master Schedule

· Periodically inform NCES on the progress of special studies

· Make recommendations for procedural changes, where necessary

· Make recommendations for the publication and dissemination of special studies reports, or for sharing information on the location of special studies reports The contractor shall undertake similar responsibilities in assisting NCES in coordinating other special studies, i.e., special studies that may include non-Functional Teams contractors.

As part of the R&D coordination function, the contractor shall facilitate collaborations among the various NAEP R&D contractors both within and outside the Functional Teams. The contractor should consider special studies to be studies that require data collection and involve participation of all or some of the Partners. The distinction among large, medium and small studies shall be in the size of the studies. A large study, for example, shall require a sample equivalent to that obtainable from 500 schools, whereas medium and small studies shall be equivalent to that obtainable from 250 and 50 schools, respectively. Studies may involve field staff going to schools or may include remote administration of data collection instruments. An example of a large Functional Teams special study would be a major bridging study to examine the transition to multi-stage testing. The analysis and initial reporting of the Monthly School Survey is an example of a medium-sized general special study. Samples required for special studies shall be obtained from the already defined operational sample, so that there shall be only minimal additional sampling, data collection, and scoring costs for these studies. Each Partner shall indicate such minimal additional costs. NAEP Functional Teams and other special studies shall be funded within the NAEP Functional Teams. Contractors shall conduct approximately three (3) special studies during the 2024-2029 contract period.

1.1.3.4 Propose and conduct innovative R&D studies to advance the NAEP program (Optional task) Since its inception, NAEP has led the way in large-scale assessment methodologies. In recent years, some of the most innovative work to grow the field has been developed through the Survey Assessment Innovation Lab (SAIL) which was established in 2013. Under SAIL, NAEP examined new models of 21st-century skills, new technologies to simulate and capture complex human interactions, and innovations in scoring and reporting.

To continue NAEP’s tradition of expanding what is possible in large-scale assessment, the contractor shall propose a set of R&D activities that are future-oriented and provide the basis for potential innovations in the NAEP program. Such research could reflect recent developments in the learning sciences, cognitive science, technology, and artificial intelligence, with the aim of advancing the state of the art in student assessment. Studies may or may not require data collection but regardless, innovative R&D studies are carried out outside of NAEP’s operational data collections and critical path. These activities shall be consistent with the NAEP Strategic Plan and “fill the gaps” in the NAEP R&D program and research conducted under Functional Teams and non-Functional Teams contracts.

The contractor shall propose a slate of innovative R&D studies, which will be updated annually and considered by an NCES inhouse R&D advisory committee, which will also provide technical guidance on ongoing studies, as needed. Proposals submitted for individual R&D studies shall demonstrate how a given study supports NAEP’s overall strategic plan and R&D agenda and include the study design and analysis and reporting plans. The contractor may seek partnerships and collaborations with external researchers employed by universities and other research organizations who have ongoing research activity in domains related to the proposed research activity. The contractor should not attempt to build everything from scratch, but instead take full advantage of technologies under development at other laboratories and make use of the eNAEP digital platform as appropriate.

The contractor shall:

· Develop recommendations memos and proposals for innovative R&D, consistent with the NAEP Strategic Plan and Research Agenda

· Suggest timelines for COR approval, and then coordinate the integration of timelines into NAEP’s Master Schedule.

· Periodically inform NCES on the progress.

· Make recommendations for procedural changes, where necessary.

· Deliver the final study report to NCES according to the agreed-upon timeline and make recommendations for the publication and dissemination of research.

1.1.3.5 Facilitate execution of ad hoc R&D studies

The contractor shall facilitate small to medium sized R&D projects both within and outside the Functional Teams, not covered under the special studies described in 1.1.3.3 or the longer-term innovative R&D described in 1.1.3.4. Ad hoc R&D studies shall address topics that must be addressed quickly, or that arise over the course of the assessment. An example of a study covered under TC is the study aimed at developing an error response model for critical path NAEP estimation. The contractor shall obtain NCES approval for these ad hoc studies prior to execution. As needed, the contractor shall consult on the execution of these studies, with researchers both within and outside the NAEP Functional Teams.

The contractor shall coordinate the development and facilitate execution of ad hoc studies. Such coordination shall include collaboratively working with NAEP contractors to:

· Develop recommendations memos and proposals for the conduct of ad hoc studies.

· Suggest timelines for COR approval, and then coordinate the integration of timelines for ad hoc studies into NAEP’s Master Schedule.

· Periodically inform NCES on the progress of ad hoc studies.

· Make recommendations for procedural changes, where necessary.

· Deliver the final study report to NCES according to the agreed-upon timeline.

1.2 Provide Program and Operations Coordination across multiple contractors and NCES Because the NAEP program is a complex, extensive, interconnected, and ongoing operation, NCES requires the POC contractor to work closely with other contractors and NCES staff to ensure a smooth interaction among all elements of the assessment, especially from a technical and policy standpoint. NAEP believes to successfully carry out program and operations coordination, the contractor must understand how all the pieces of the overall NAEP process fit together and be able to openly discuss with other NAEP contractors how program and operational changes might be effectively incorporated in the system. The contractor shall work in tandem with the scheduling contractor to share input needed for scheduling and coordinating tasks.

Support operational planning The purpose of operational planning for NAEP is to define the major tasks, estimate the time and resources required to carry out the activities, and provide a high-level process for monitoring, controlling, and reviewing the quality of activities and the progress made in working through the plan. It is developed with reference to the NAEP Strategic Plan.

1.2.1. Support operational planning toward meeting NAEP’s primary goal The contractor shall support NCES in identifying a set of NAEP performance objectives to be achieved during the contract cycle, to meet the primary goal of measuring student progress over time, while maintaining the leading role in large-scale assessments. There may also be a subset of performance objectives that NAEP would like to meet within a single assessment cycle. The contractor shall submit to the COR recommendations for performance objectives that focus on strategic goals.

The contractor shall lead the other NAEP contractors to develop a set of procedures for evaluating progress towards these above objectives, and more. The procedures shall be added to the NAEP Strategic Plan and include targets for gauging progress towards meeting objectives. The procedures shall also state clearly how the contractor will coordinate the work of the Functional Teams towards achievement of these objectives. The contractor shall submit quarterly update reports, coinciding with Executive Steering Committee meetings, to keep NCES abreast of progress. These reports shall include recommended actions to ensure that targets are met.

1.2.2 Coordinate the implementation of tasks across multiple contractors For NAEP projects that need to be coordinated across multiple contractors, the POC contractor shall lead or participate in regular status meetings throughout the contract period to monitor, track, and guide various contractor activities. They shall obtain appropriate input from other NAEP contractors, and from other NAEP support contractors, as needed. The POC contractor shall also provide guidance to and facilitate communication among the contractors. Task 11 (General Activities) outlines the response expected from other contractors to the POC contractor’s requests for information and access to meetings to effectively manage program and operations coordination.

The contractor shall alert the COR and other relevant NCES managers within 24 hours to any problems of which it becomes aware that could jeopardize meeting NAEP contractor project targets and objectives.

Inputs, such as the Master Schedule, other scheduling reports, and meeting minutes, shall be used to populate status updates to be delivered to the COR weekly.

1.2.3 Coordinate planning and review meetings among NAEP contractors and NCES The POC contractor shall coordinate major review and planning meetings among representatives of the other NAEP contractors, NCES staff, and where appropriate, other NAEP support contractors, to review program progress and to plan for future activities. It is estimated that about fifty (50) people will attend such meetings. The first of these meetings shall constitute a project initiation meeting and shall be held within thirty (30) days of the start of the contract. The other key meetings shall include a mid-year NAEP operations meeting to make decisions about the final assessment activities for the upcoming assessment year, a fall NAEP design summit to make decisions about design plans for several assessment years, and a NAEP-wide post- assessment debriefing to discuss significant issues that occurred during assessment administration with NCES, along with potential and applied solutions.

The contractor shall establish regular and as-needed communication with other contractor partners to coordinate aspects of NAEP work and stay apprised of developments and issues. NCES is open to having additional or more frequent meetings if the contractor finds that the meetings help the program with better planning and monitoring. In conducting these meetings, the contractor shall use technology to substitute for face-to- face meetings where appropriate.

The contractor shall arrange meetings between sub-groups of team members, and between team members and NCES staff to address problems and coordinate activities as needed and determined in coordination with the COR. Such meetings may be between contractors only or with NCES, as appropriate. Each of the NAEP contractors would submit items for the agenda, and as required by their respective COR, be responsible for writing a memo of any actions to be taken. The contractor shall provide meeting agendas and minutes, and coordinate and deliver any resultant recommendations memoranda from the other contractors to the COR who was responsible for the contract. Examples of meetings could include weekly project update meetings between contractor and COR, monthly project review and planning meetings between NAEP contractors and NCES staff, regularly scheduled meetings, and NAEP contractors-NCES working group meetings, such as the Socio-Economic Status working group meeting.

The contractor shall deliver recommendations memos to the COR, outlining recommended actions that NCES might pursue following meeting discussions.

1.2.4 Coordinate timely responses to quick turnaround requests At times, the NAEP program must respond quickly to requests from within and outside NCES. NCES requires support from the POC contractor to coordinate activities requiring input from multiple contractors to ensure the timely response to emerging matters for action, as identified by NCES. For each identified matter, the contractor shall submit a memorandum to the COR with the response to the quick turnaround request. The contractor shall deliver a memorandum that is professionally written, quality reviewed and complete. The memo could also include an analysis file or tables, depending on the nature of the request. Sources of emerging matters for action could include the Governing Board quarterly meetings, offices within the Department of Education, other Federal agencies, and state departments of education.

The contractor may also be asked to help NCES provide technical support for activities within the Office of the Associate Commissioner, aimed at ensuring responsiveness to requests for information from sources such as Congress, the Office of the Secretary of Education, the National Assessment Governing Board, officials within the Department and the news media. Quick turnaround requests will usually require a response to NCES within 24 hours. As needed, the contractor shall take the lead on reviewing analytical reports and related documents for the purpose of providing NAEP contractors’ input on responses to such requests.

1.2.5 Coordinate the production of technical documentation on the web Each of the NAEP contractors is responsible for the different processes that make up the NAEP assessment life cycle and each must develop technical documentation on processes and methods. The dissemination of high quality, useful and timely information on these processes is important to the NAEP program, and at present this material is available on the NCES website. The contractor shall coordinate a process to ensure successful production of NAEP web-based technical documentation, not produce the actual documentation. For each cycle of assessments, the NAEP contractors shall produce web-based technical documentation of all phases of the assessment, including assessment design, sampling, data collection, items, and analytic methods. Each cycle of documentation shall document those features of the assessment that constitute changes from previous practice, new elements do not present in previous assessments, and data particular to the assessments being documented (such as sample sizes or numbers of items of different types).

Chapters of the technical documentation shall be written by the contractors responsible for each part of the assessment, such as sampling, data collection, and scoring. As part of NCES Division Review, the contractor shall review all chapters and sections of the technical documentation to ensure coherence of the document. The contractor shall coordinate a process with the web contractor to assemble all chapters into a logically consistent set of electronic files for posting on the NAEP website. Technical documentation shall be written for posting on the website, with an option that allows users to download and print either the entire report or selected sections. The contractor shall ensure that the completed work is of high quality and provide up-to-date production schedules to all NAEP contractors for producing web-based technical documentation.

1.3 Coordinate development of information collection requests (ICR) for OMB approval This task shall include all NAEP information collection request package submissions that are prepared for OMB approval under the Paperwork Reduction Act (PRA). These submissions explain and justify the information collection. The PRA guide, available on the Web, provides general information on preparing packages for OMB approval. Types of collections may be classified as main assessments and special studies, or developmental studies. The main assessments and special studies include operational full-scale NAEP (e.g., national, state, and trial-urban district levels), NAEP pilot tests, and NAEP special studies data collections. The developmental studies include focus groups, cognitive lab and other small-scale studies that use distinct samples that are not part of the NAEP sample and are not designed to be folded into operational NAEP. The contractor shall prepare OMB packages for these different collection types.

For the main assessment data collection package, the contractor shall describe all known assessments and special studies that will be conducted in NAEP for a period, not to exceed three years. These packages require a 60-day notice in the Federal Register to obtain public comment on the proposed collection, prior to submitting the package to OMB, and then a 30-day public notice. See the PRA guide for details. The contractor shall schedule the submission of operational packages to allow for the six to nine months it may take to clear the package. The generic system clearance is no longer used for NAEP operational/pilot collections. NAEP’s current operational clearance (OMB 1850-0928) expires on April 30, 2026.

Requests for developmental data collections are facilitated through a generic clearance for NCES focus groups, cognitive, and pre-test studies. The contractor shall prepare individual clearance packages for these studies that specify the details of each study. These packages, after NCES review, are submitted to OMB, without posting in the Federal Register for public review. The NCES current developmental clearance (OMB 1850-0803) expires on June 30, 2025.

During the five-year period of performance, the contractor shall develop yearly (5) requests for NAEP operational data collections clearances, and any related submissions necessary to clear the main assessments and special studies. The main NAEP package clearance often requires multiple submissions in which draft material is cleared first, and then one or more amendments to the package with substantive or non-substantive content changes are submitted to clear the final version of the package. The contractor shall also develop up to 15 requests per year for NAEP developmental packages. The contractor shall employ a quality review process to ensure the study descriptions in all packages are accurate, complete, and comply with relevant OMB guidelines and NCES statistical standards.

In addition to the clearance packages, the contractor shall prepare memos to accompany each package. Memos are needed to summarize the contents of the package, or document and justify changes to the submitted or cleared OMB package. The contractor shall also prepare memos with recommendations on how to address public comments to the Federal Register Notice and OMB pass-back questions.

2. Sampling and Weighting The National Assessment of Educational Progress (NAEP) Sampling and Weighting (SW) contractor prepares the sampling and variance estimation weights for the operational assessments, pilot and field tests, and special studies required by NAEP. These activities shall be performed by the contractor during the 2025 through 2029 contract period. These activities must continue to be maintained per industry standards and best practices while other competing issues will demand constant attention such as decreasing private school participation rates, adequate representation of minorities in NAEP results both at the national and subgroup levels, linking with other state, national, and international assessments and developing and delivering a state-of-the art student assessment using NAEP-provided equipment, using school equipment or both through a hybrid model. Since the potential benefits and new opportunities are enormous as NAEP continues the transition to using school-provided equipment to collect data, the NAEP SW contractor must be flexible and innovative enough to overcome the challenges inherent during this time of transition.

Scope The expected scope of the NAEP Sampling and Weighting contract primarily falls into three types of activities:

· Design and select samples (section 2.1) for the national main and long-term trend assessments including private schools, state assessments, Trial Urban District Assessments (TUDA), National Indian Education Study (NIES), pilot and field tests and other special studies. When designing and selecting samples, the contractor must ensure that all samples meet the reporting goals defined in the multi-year design and evaluation plans and the operational design plan and are robust enough to be able to maintain historic overall response rates in general (e.g., ensure that there is enough sample for Differential Item Functioning (DIF) analysis to be performed for all major reporting categories) and for minority subgroups (i.e., Native Americans, Blacks and Hispanics). Designing samples must allow for linking to other international, national, and state assessments; creating appropriate samples for TUDA districts supporting state level reporting; systematic linking with special studies (e.g., 12th grade and Career/Technical Centers) and special populations (e.g., homeschooled students); and for bridge studies supporting NAEP’s transition to adaptive testing.

· Generate sampling weights (section 2.2) sufficient to detect errors that interfere with valid inferences being made about groups of students.

· General activities (section 2.3).

· The scope of activities for the contract period are largely determined by the schedule of assessments for the period, as determined by the National Assessment Governing Board (see the current schedule here: https://www.nagb.gov/naep/assessment-schedule.html) and requires in-depth coordination with the Planning and Operations Coordination (POC) and Design, Analysis, and Reporting (DAR) contractors.

2.1 Design and Select Samples

The contractor shall develop sampling plans, based on the assessment design specifications provided by the NAEP Design, Analysis and Reporting (DAR) contractor, for each of 4th, 8th, and 12th grade and ages 9-, 13-, 17-year old Long-Term Trend operational assessments, pilot and field tests, bridge studies and special studies.

Basic design features for NAEP samples are as follows:

· NAEP samples are geographically and regionally representative of the nation including private schools, the reporting jurisdictions, and other reporting subgroups;

· NAEP schools and students are chosen with a measurable probability of selection;

· NAEP samples control for school overlap between assessment cycles across years;

· Appropriate school variables such as type of school, school-level student achievement, and type of location are used to develop a stratified random sample;

· NAEP national, state and district samples are of sufficient size to generate standard errors for relevant populations and subpopulations (e.g., white, black, American Indian and Hispanic students, English Learners) similar to previous NAEP assessments;

· All national NAEP samples shall include, as a component, Large Central Cities;

· NAEP samples are optimized by combining state and Trial Urban District Assessment (TUDA) samples into a nationally representative sample where possible; and

· NAEP sampling methodology for selecting the sample and computing the weights should support continuation of trend reporting.

Furthermore, NAEP samples shall, to the extent possible:

· Minimize effects due to clustered sampling;

· Minimize cost and burden;

· Ensure that sampled students are assessed in only one subject area unless otherwise directed;

· Minimize (or maximize depending on the design requirements) sample overlap with other NCES surveys.

Previous NAEP public school sampling frames have been derived from the annually updated the National Center for Education Statistics (NCES) Common Core of Data (CCD). The CCD provides state and district-level school information including grade spans, enrollment size, and district affiliations. Due to a two- to three-year lag time between the release of CCD files and the current population of schools, the SW contractor shall work with the NAEP state coordinators (through the NAEP Support and Service Center (NSSC) contractor) to update the list of eligible schools. Additionally, the SW contractor shall propose and implement a plan that will use historical frame information from previous NAEP assessment cycles to provide feedback to and update / augment the most recently available CCD data, with the goal of relieving as much of the burden as possible from the NAEP state coordinators when they update their lists of eligible schools.

Previous NAEP private school sampling frames have been derived from the biennial NCES Private School Universe Survey (PSS), information from state departments of education, and a combination of other publicly available lists of private schools. The NAEP program actively partners with interested private school associations to increase participation rates. This partnership may allow for schools associated with certain private school associations to be oversampled to encourage participation.

The list of sampled schools shall be made available in a secure electronic format for easy access by states and jurisdictions, the NSSC, and the other NAEP contractors (as needed) no later than the first week of May of the year prior to the assessment (unless otherwise directed by NCES).

Currently, stratified multistage probability-proportional-to-size sampling methodology is used to develop sampling plans for the nation and the jurisdictions in the state-level assessments.

The contractor shall anticipate and accommodate participation differences in mandated and voluntary assessments (participation information for previous assessments is available in the NAEP technical documentation). The contractor shall develop contingency plans for generating representative national and state samples if some states, school districts and/or private schools refuse to participate and design appropriate procedures for the prompt selection of a substitute school when a school selected for any of the national-only and non-mandated state operational assessments declines to participate. Substitute schools are selected to replace originally sampled non-responding schools and should have the size and characteristics that are the most similar to those of the originally sampled schools. Substitute schools do not need to be selected for assessments where state and district participation is required (i.e., 4th and 8th grade mathematics and reading).

National Sample Designs: The contractor shall produce national samples involving the selection of geographically representative Primary Sampling Units (currently, NAEP PSUs are composed of a county or group of counties) and the selection of public and private schools including charter schools, Bureau of Indian Education (BIE) and the Department of Defense Education Activity (DoDEA) schools.

When the assessment schedule calls for national samples, the sampling plans shall (where applicable): 1) oversample schools with Hispanic, Black and Native American students to ensure the full geographic and regional representation of these groups; and 2) oversample private schools such that their student samples are twice as large as they would be if no oversampling occurred. Sampling designs for oversampling student groups should be innovative and be designed to account for the potential of student group density or other sampling effects on the representation of student performance. For example, NCES studies have revealed lower average scores among Black students in schools with high percentages of Black students. A sampling design that oversampled schools with high percentages of Black students could influence the average score for these students and may overrepresent schools with a high density of Black students rather than fairly representing where these students are in schools.

The SW contractor shall propose innovative ways to efficiently come up with better samples of major reporting groups, including at the state and TUDA levels including (but not limited to) English Learners (ELs) and Students with Disabilities (SWDs). A potential source of innovation could be including large enough samples of ELs and SWDs in national NAEP sample designs to be able to be reported.

Student sample designs: the contractor shall produce student samples that include the random selection of individual students, rather than intact classes, for assessment sessions in schools (except as negotiated with NCES during the development of the DAR contractor’s design plan).

As described in the NAEP technical documentation, NAEP has previously selected students systematically (every nth student) with a random start, selecting individual students from a list of all grade- or age-eligible students in the school (rather than intact classes) for assessment sessions in schools.

The contractor shall obtain the most up-to-date and accurate list of students from sampled schools. In previous assessments, each participating school (or the state) submitted student rosters for a group of schools electronically to the SW contractor. Once the school rosters are received, the contractor shall select the students.

NCES expects a high rate of response from students at all ages and grades. Response rates for mandated assessments have been as high as 100% and the response rates for non-mandated assessments at grades 4 and 8 also have been consistently high (i.e., over 85%). Student response rates in private schools have generally been high (i.e., over 85%). Participation rates for grade 12 students have stabilized (80-85%) after years of decline.

To expand the scope and reach of NAEP and to exploit potential data collection efficiencies, all sampling plans submitted by the SW contractor shall purposefully maximize any potential linking opportunities with other NCES assessments (e.g., the National Teacher and Principal Survey, the High School Longitudinal Study, etc.) and international assessments (e.g., Trends In Mathematics and Science Study, Programme for International Student Assessment, etc.) that are…