Attachment A - NAEP Data Collection Team SOO.pdf

i

Attachment A

Statement of Objectives (SOO)

NAEP Data Collection Team

Task 7: NAEP Support and Service Center (NSSC)

Task 8: Data Collection

National Assessment of Educational Progress

(NAEP)

2024-2029 ii

Table of Contents

Exhibit A…………………………………………………………………………………….…xi

Exhibit B……………………………………………………………………………………….xiii

Exhibit C……………………………………………………………………………………….xiv

Exhibit D……………………………………………………………………………………….xiv

1. Procurement Overview…………………………………………………………………....iii

7.0 Task 4 - The National Assessment of Education Progress

(NAEP)Support and Service Center…………………………………………..……..1

7.1 Orientation and Training of Coordinators……………………………………….4

7.2 Continuous Training of Coordinators…………………………………………

7.3 Technical Assistance and Support to States and Districts……………………

7.4 Conduct Evaluation of Center Products and Services and

Conduct Evaluation of States’ Compliance with NAEP……………...………..11

7.5 Support Private School Recruitment Private School Coordinator……………...12

7.6 State and District Additional Support…………………………………….…….12

7.7 Annual Reports to NAEP……………………………….…………..……….….13

7.8 Educational Assessment Major IT Investment….……………………………...13

7.11 General Activities………………………………………………………………15

8.0 Task 8 Data Collection…………………………………………………………………..46

8.1 Collect Data……………………………………………………………….……47

8.2 Materials, Distribution and Processing…………………………………………58

8.11 General Activities………………………………………………………………69

iii

1. Procurement Overview The U.S. Department of Education intends to enter 5-year contracts with qualified entities to establish a Coalition of Teams of contractors to carry out the design, assessment content development, sampling, materials preparation and distribution, data collection, scoring, analysis, and reporting of one of its flagship programs, the National Assessment of Educational Progress (NAEP). NAEP is the nation’s largest, longest running and most highly regarded nationally representative assessment of student achievement in the United States. The program has maintained trend lines measuring the nation’s student achievement for more than 50 years on its Long-Term Trend assessments in mathematics and reading and more than 30 years on its Main NAEP assessments, which include results for the nation, states, Puerto Rico, and the Department of Defense Education Activity (DODEA), and more than 20 large urban districts that have joined the Trial Urban District Assessment (TUDA) program over the years. It is imperative that the program maintain its core role of reporting educational progress over time, while, at the same time, taking advantage of unprecedented changes in education, in education assessment, and in the learning sciences, as well as rapid advances in technology to innovate, improve efficiency, and otherwise continue to lead the field of large-scale educational assessment.

As the Federal statistical agency charged with implementation of NAEP, the National Center for Education Statistics (NCES) has made a commitment to lead the field of large-scale, technology-based assessments and has identified four goals for the activities defined in this procurement.

They are:

Maintain NAEP’s high standards for quality and its position as the gold-standard in large-scale assessment, including:

o Maintenance of NAEP’s treasured 30-year trend lines in mathematics and reading at grades 4 and 8;

o The highest quality assessment and survey items in large-scale assessment;

o Smooth, trouble-free experiences for the states, districts, schools, educators, students, and other stakeholders participating in NAEP studies; and o Accessible, informative, timely, relevant, objective, credible, error-free data for the wide range of stakeholders consuming NAEP data.

Improve cost-efficiency of processes across NAEP so that NAEP continues to be the best value for money in large-scale assessment.

Develop methods for constant innovation in all aspects of NAEP to improve NAEP’s utility, the experience of NAEP for participants and data users, and the program’s cost-efficiency.

Create the future in large-scale assessment through far-reaching research and development and a deep understanding of developments in education and technology and where NAEP can make the best contribution to the nation’s understanding of educational progress.

There are numerous environmental factors that will influence NAEP during this period, many of which will be referenced throughout several Statements of Objectives (SOO). Examples include:

iv ongoing transition from a paper-and-pencil to technology-based assessment, the emerging interest in international comparisons for states and districts, advances in technology, including artificial intelligence and other developments that could broaden the capabilities of NAEP or improve efficiencies, advances in test development and new item types, development of measures of preparedness for post-secondary education and work, adoption of new content standards across most states that may or may not align with NAEP content frameworks, inclusion of more contextual information in reporting, the integration of NAEP and other NCES surveys and the overall impact on operationalization of the assessment, growth of virtual schools and out-of-school and technology-based learning by students, and potential budget reductions throughout the education and assessment enterprises.

NCES envisions this procurement period both as vital to maintaining a quality assessment and as a continuation of its transition to the future.

Purpose of NAEP

The purpose of NAEP is to serve as a national yardstick of student achievement. NAEP informs the public about what American students know and can do in various subject areas, including civics, mathematics, reading, science, and U.S. history. The assessment results are representative of the nation, regions of the country, and where appropriate, states and selected urban districts at grades 4, 8 and 12. NAEP Long-Term Trend assessment results are representative of the nation and regions of the country for students at ages 9, 13, and 17.

Administration

The Commissioner of Education Statistics (Commissioner) in NCES develops, administers, and reports NAEP while the National Assessment Governing Board (Governing Board), appointed by the Secretary of Education but independent of the Department, sets policy for NAEP.

Main NAEP assessments are made up of the following operational components:

National assessments include mathematics, reading, science, writing, civics, and U.S.

history. These assessments follow the content frameworks developed by the Governing Board and use the latest advances in assessment research. Most subjects are assessed at v grades 4, 8, and 12—although not all grades are assessed each time. Students in both public and private schools are assessed.

State assessments are administered in four subjects (mathematics, reading, science, and writing). Only public schools are included in state-level assessments.

The Trial Urban District Assessment (TUDA) is a multiyear study of the feasibility of a district-level NAEP. Twenty-seven districts currently participate in TUDA.

The NAEP High School Transcript Study (HSTS) is conducted in high schools selected for NAEP mathematics and science assessments. Results are reported primarily at the national level.

The National Indian Education Survey (NIES) is conducted in schools with American Indian/Alaska Native students. Results are reported at the national level and for states with large samples of AI/AN students. (OPTION)

While Main NAEP comprises the bulk of the program, NAEP is a system of assessments that seeks to provide a comprehensive measure of educational progress in the country. Other major components are the long-term trend assessments, special studies, and studies for insuring the validity of NAEP items. They are as follows:

Long-Term Trend assessments (LTT) are given at the national level only and are administered in a different manner from that used for main NAEP assessments. LTT reports results in mathematics and reading that represent trends since the 1970s. LTT assessments are currently administered every four years.

Special studies are periodically conducted to answer a variety of questions. Examples of past special study topics include:

Inclusion and accommodations in NAEP, Math Accessibility Study, NAEP-TIMSS Linking Study, Knowledge and Skills Appropriate study, Socioeconomic Status, and

Various technology-based assessment studies.

Pilot studies are conducted on all items prior to their use in NAEP assessments. All items are reviewed and go through several improvement activities. Cognitive labs and small-scale tryouts are conducted as needed on new and complex item types.

vi

The NAEP Coalition Concept

All NAEP assessments have critical path processes that are conducted by individual contractors as part of the overall team or Coalition of contractors; however, they are dependent upon one another for completion. Broadly, these processes are as follows:

Overall assessment design, Assessment and survey content development, Application development, including assessment and administration platforms for administering assessments, and other tools and applications that support the

NAEP program, Sampling and weighting, Data collection, Scoring, scaling, estimation, and analyses, Reporting and dissemination, and

Research and development across all aspects of the work.

Further, all the processes include the following:

state involvement to ensure participation of schools and the quality of the sampling and data collection processes, the use of the technology in various administrative functions and in the administration of digitally-based assessments, and planning and coordination to ensure that the various processes are aligned and coordinated with one another, and the schedule is maintained for timely reporting and dissemination of results and documentation.

Current Operating Practices

NAEP instruments generally have a common block/booklet structure. Booklets consist of two separately timed 30-minute cognitive blocks followed by separately timed background questions.

NAEP subjects using the common block structure and assessed in the same testing period are spiraled together and administered in the same session.

Reading and mathematics assessments have the highest priority of all assessments for support and scheduling with item development occurring at least three years before an operational assessment. These items are pilot tested up to two years in advance as part of an operational assessment.

vii

NAEP operates on a general principle that pilot tested items are not changed prior to use in an operational assessment. Pilot tested items that are of high quality but not used in the upcoming operational assessment are banked for future use. Pilot test data are not publicly released nor used to provide performance estimates.

Main NAEP assessments are administered during a six-week window beginning the last week of January and concluding the first week of March. In extreme cases, such as a natural disaster or major snowstorm, an additional week or two in March may be used for make-up assessments.

Assessments within states and large urban districts are spread across the entire assessment window.

States and districts are required to participate in grade 4 and 8 reading and mathematics assessments every two years (i.e., mandatory assessments). All other assessments are voluntary for states.

NAEP samples are geographically representative of the reporting jurisdictions and subgroups.

Schools and students are chosen with a measurable probability of selection.

In a state-level assessment, the national sample is composed of the aggregate of state samples.

However, when a state elects not to have the state-representative samples needed to report at the state level, some students in that state may be included to ensure representativeness of the national sample.

Long-Term Trend assessments are administered: in the fall to 13-year-olds, in the winter to 9-year-olds, and in the spring to 17-year-olds.

Testing accommodations are provided to students with disabilities and English learners to remove barriers to their participation if the constructs tested are not altered by the accommodations. For example, the reading assessment cannot be read aloud to students or translated to another language, as the NAEP reading framework defines the construct to include reading printed text in English. Data released from the assessments must report on students with a disability (SD) and English learners (EL) as a percentage of all students who completed the assessment. Within those two populations, the percent excluded, the percent tested with accommodations, and the percent tested without accommodations are reported.

The Governing Board plans and executes the initial release events for assessments after the Commissioner has officially released the results. The current primary means for releasing the initial 4th and 8th grade reading and mathematics results is a summary report intended for a public audience. National and state reading and mathematics reports at grades 4 and 8 are required to be available for release within 6 months after the completion of data collection. All other NAEP initial data releases must be available within 12 months after the completion of data collection, or on a different release schedule provided by the Governing Board. Releases include development of electronic materials and websites for public consumption. NAEP continues its move to more electronic materials and to greater use of the Web and social media for release activities.

Reports and data products from assessments on the Governing Board schedule where data are being released for the first time are identified as “initial release” and are differentiated from other viii

NCES reports and products in the review process. “Initial release” products are reviewed by the Assessment Division (AD), at the NCES center-wide level for signoff by the Chief Statistician, by the NCES Commissioner, and by the Director of the Institute of Education Sciences (IES).

Other products, in addition to the reviews by the Assessment Division, the Chief Statistician, and the NCES Commissioner, undergo a peer review (either internal or external) organized by the

IES.

To the extent possible, NAEP designs, samples and analyses are conducted to maintain trend over time.

Aspects of the NAEP design may change during this procurement period (2024-2029). Inclusion policy and an improved measure of socioeconomic status are some of the practices that are under review for possible changes. Another possible change includes increasing the precision of measurement particularly at the lower and upper ends of the student ability distribution. In addition, as outlined below, there are several challenges that will need to be addressed during the procurement period. Consequently, there will be the need for flexible approaches to design modifications as NAEP’s role continues to evolve either through changes in legislation, Governing Board and Department of Education policies, technology, or the fields of educational measurement, education, or schooling.

Current Operating Cycle

The project consists of preparing for and conducting parallel assessments in different stages1.

Special studies and pilot testing are integrated into the operational assessments, as needed. All Coalition contractors are always actively involved during the calendar year and activities are often completed on tight timelines. The following paragraphs provide a sampling of the scheduling complexities.

The contractors responsible for the design of the assessment must develop a preliminary design up to four years in advance of the assessment so that all other contractors can prepare. The sampling contractor, for example, must begin the sample selection process in January the year before the assessment so states can be notified in May for the subsequent school year.

The item development contractor will be writing, reviewing and pilot testing items for four to six content areas across grades 4, 8 and 12 each year. Items are reviewed by NCES, a standing committee of content experts, the Governing Board, and the states throughout the year.

For the data collection contractor, the peak period is January to March, when substantial numbers of additional people are employed to collect data from students. But other times of the year, staff will be involved in developing training for data collectors, conducting the high school transcript studies, and working with private schools.

1 When there is a new framework being created by the Governing Board, the cycle takes five or more years.

ix

The reporting contractor must meet a tight six-month or twelve-month deadline for all initial report releases. In addition, the contractor analyzes and drafts other reports and conducts other studies or analyses throughout the year.

The contractor for materials and distribution begins their process of compiling materials in the late spring for distribution during the following winter based on sample schools identified in early May and sample students identified in the October – November period.

The contractor is responsible for 24-hour turn-around of additional materials during the data collection in the January – March period.

The contractor supporting the 50 state and 27 district coordinators for their various activities throughout the sampling, data collection, and reporting processes maintain weekly contact via electronic media, conduct three (3) to six (6) in-person workshops each year, and provide orientation to NAEP for the 20-25 percent of new coordinators each year.

The web contractor supports the release of The Nation's Report Card, online technical documentation and other key updates to the public websites regarding NAEP and assessment resources. This contractor also hosts the system on which the assessment platform lives and transmits data and is responsible for developing and maintaining several internal applications and tools that support the NAEP program.

The platform development contractor implements, supports, maintains, and enhances a NAEP digitally-based assessment (DBA) Platform and supporting technology systems.

NAEP’s DBA Platform comprises the full end-to-end technical framework, or environment, that delivers the NAEP assessment to students.

Challenges

NAEP’s primary purpose is to provide a fair and accurate measure of student achievement in various content areas and to report trends over time. At the same time, NAEP must be responsive to environmental and policy changes and to the emerging needs of national and state policymakers. NCES and the Coalition contractors must not only administer required assessments based on an established schedule and with designated populations but also face the competing priority of responding to changes, some of which cannot be anticipated.

For example, innovations in better and cheaper technology and their integration into the classroom and out-of-school learning experiences of the country’s students dictate that developing an assessment for both national and state assessments must be forward-thinking.

Creative development must consider cutting-edge technology, equity and fairness for students, ease of use, timeliness of administration, and flexibility of the jurisdiction environment.

NAEP will be challenged to identify the type of infrastructure that will be necessary to maintain a standard, unbiased administration while maintaining high levels of item security, and low operational burden and cost. The test administration software must allow seamless transition from assessing any of the NAEP digitally-based assessments to allowing these digitally-based items to function in the NAEP Questions Tool.

x

In addition to delivering assessments via appropriate technology, it will become increasingly more important that the application of technology to all parts of the NAEP system is updated to keep pace with the latest industry advances in a pragmatic and cost-effective manner. Many of NAEP’s administrative processes can be paperless.

Whenever commercial-off-the-shelf (COTS) software is appropriate and available, it should be given strong consideration in the evaluation of solutions. In addition, some custom systems may also be developed. All these systems will have to be maintained and enhanced so that NAEP can easily incorporate new innovations in hardware and software with minimal impact on operations and/or budget.

Other challenges that will have to be addressed during this procurement period include the following:

International comparisons. There is a growing demand by national, state, and local policy makers and the public to know how their schools compare not only to the nation and states but to other countries. There is a recognition that our students will be competing with students from this country, as well as students worldwide. NAEP, the only state-representative assessment, is in an excellent position to serve as the link with international assessments to provide state and district estimates for comparison with other countries.

Preparedness for Post-Secondary Education and Work. A different type of challenge arises from the call for the use of the 12th grade NAEP reading and mathematics assessments in a way that captures and measures preparedness for participation in post-secondary education, the military, and/or the workplace.

Universal Design. NAEP assessments should be developed to maximize accessibility for all students to the test items.

Adaptive testing. Adapting which test blocks are administered to an examinee holds the promise of providing better measurement of students across the full range of student performance.

Changing state content standards. States content standards are continuously changing in a post Common Core State Standards initiatives environment. States have continued to modify standards and assessments during a time of changing accountability measures.

Trend. Critical to NAEP’s mission of reporting on the progress of education is maintaining trend. However, to remain relevant, NAEP must respond to changing environments. NAEP routinely faces this challenge through bridge studies that provide statistical evidence for continuing trend lines. However, with the major shifts in what and how NAEP assesses student performance, there may be other more efficient and cost-effective methods that can be used.

Procurement Principles

There are six primary principles the Coalition contracts shall adhere to in this procurement:

1. NAEP data must be accurate, of high quality and on time.

xi

2. NAEP must continue to lead the -field of large-scale assessment, especially in meeting the upcoming challenges.

3. NAEP must adapt to rapidly-changing environmental factors.

4. NAEP must be designed in a way that maintains reporting trends over time.

5. NAEP must improve efficiencies in both costs and burden on participants.

6. NAEP must ensure that the diversity of a workforce that characterizes the full range of NAEP stakeholders is represented at every level of NAEP operations. To do so NAEP must make sure that every attempt possible is made to incorporate professional staff who represent that diversity and that each of the NAEP contractors increase the participation of small and minority-owned businesses within each contract.

Procurement Components

As seen in the following organizational chart (Exhibit A), the new NAEP Coalition Procurements will consist of ten critical path components with each component a separate contract award. Ten of the components are described in chapters 1 through 10 in this SOO, with each chapter corresponding to a separate contract. Chapter 8.3 describes activities common to all contractors and that shall be included in each offeror’s proposal.

The components are as shown in Exhibit A.

Exhibit A: NAEP Coalition Components

1. Program and Operations Coordination

Supports overall strategic planning and design for NAEP;

ensures coordination among Coalition contractors; maintains data for tracking program progress; and provides logistical support for complaints and substantive comments.

2. Sampling and Weighting

Selects samples; prepares sampling weights.

3. Design, Analysis and Reporting

Designs all pilot and field tests, operational assessments, and special studies; analyzes data ensuring reporting of valid results;

proposes and prepares psychometric and statistical analyses compatible with previous NAEP methodologies; specifies data needed to meet the goals for reporting; and prepares reports.

4. Assessment Content Development

Develops cognitive items, scoring rubrics and survey questions;

assists in the training of scorers; conducts cognitive interviews/small-scale pilots of items, rubrics, and survey questions; translates items and survey questions; and conducts item reviews

5. Scoring Human and AI-scoring of student responses.

6. NAEP Platform Development

Develops NAEP assessment delivery platform utilizing a state-of-the-art, age-appropriate user interface (UI) and overall user experience (UX) that is consistent, intuitive, and accessible across all subjects and grade levels xii

7. NAEP Support and Service Center

Provides support, training, and resources to state and TUDA coordinators to ensure the accurate and timely sampling, administration and reporting of NAEP in each state and TUDA district.

8. Data Collection Administers assessments and collects data for pilot and field tests, operational assessments, and special studies; ships completed assessment materials to the scoring sites; conductions High School Transcript Study.)

9. Program Support Management

Manage the NAEP program in alignment with Project Management (PM) best practices to ensure proper scheduling, quality control, risk management and communication across contractors and handoffs.

10. Web Development, Operations, and Maintenance Supports

Develops, implements, and supports public and private web applications and services; identifies and recommends emerging technologies; monitors compliance with all NCES web cybersecurity requirements; and ensures timely and accurate release of web products.

As seen in Exhibit B, the Coalition includes three sets of contracts considered as teams within the Coalition and to be competed as teams and two contracts that will be competed as standalone under separate RFPs. This is in contrast with prior competitions for NAEP contracts in which potential bidders formed teams to bid on the whole set of contracts. Separating the larger Coalition into smaller teams and standalone contracts for the competition is intended to make it easier for potential bidders to put together competitive bids and provide the government the best and lowest cost proposals.

Once the competition for contracts has been completed, all winning contractors must work together as a Coalition to ensure the successful implementation of NAEP. However, the three teams of contracts will be competed as teams because the hand offs between contractors within each team are so critical that vendors must develop relationships and plans before the competition to help ensure success on the critical hand offs within teams.

From left to right in Exhibit B below, the teams are Assessment Content, Design and Coordination, and Data Collection.

Assessment Content: This team includes separate contracts for Assessment Content Development, Scoring, and NAEP Platform Development that require close collaboration to ensure the successful and efficient development of the instruments for undertaking NAEP, including the scoring of student responses.

Design and Coordination: This team includes separate contracts for Program and Operations Coordination, Design, Analysis, and Reporting, and Sampling and Weighting.

This is the core team for supporting the overall design of NAEP and coordination among contractors across the Coalition and requires real-time, close coordination across team contractors.

xiii

Data Collection: This team includes separate contracts for Data Collection and the NAEP Support & Service Center—two contracts that must work hand-in-glove for successful administration of NAEP in the field.

The Program Support Management and Web Development, Operations, and Maintenance Support contracts (shown in gray) will provide key support for the Coalition and are involved with key hand offs within the Coalition but may not require partnership with other vendors pre-bidding to ensure successful participation in the Coalition post-bidding.

Each of the three team sets of contracts will be bid as its own team, with contractors working together to submit a unified group of proposals that will be accepted or rejected in their entirety.

Any organization interested in making an offer on one or more components within a team must seek out other organizations that are recognized for complementary expertise to collaborate on a team proposal. The other two contracts will be standalone. NCES will award ten contracts, one for each component, with more than one contract being awarded to a single organization as appropriate. The contractors of all ten contracts will be required to develop agreements to ensure successful collaboration as an overall coalition.

Exhibit B. The NAEP Coalition

Please note that other, non-Coalition contracts also support NAEP. See Exhibit C for a list of non-Coalition contracts. Additional collaboration between Coalition and non-Coalition contracts—not just within the Coalition contracts—is vital to the successful implementation of

NAEP.

xiv

Exhibit C: NAEP Contracts not Included in the Coalition

Publications, Outreach and Dissemination Support (POODS)

Procurement of Research Evaluation and Statistics Task Orders (PRESTO)

NAEP Logistics NAEP-State Technical and

Operations Cooperative National Indian Education Study

(NIES) Support Services Contact

Overview of the Work

Independently, and not as an agent of the United States Government, the Contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below. The work to be covered under this SOO begins with the award of the contracts in 2024 with completion of the 2024 assessments and ends in 2029 with development work for the 2030 assessments and beyond. Each assessment cycle includes field/pilot tests, and the operational assessments.

In November 2023, the Governing Board set the current schedule of assessments as presented in Exhibit D (see https://www.nagb.gov/naep/assessment-schedule.html).

Exhibit D: NAEP Schedule of Assessments

Year Subject National Grades Assessed

State Grades Assessed

TUDA Grades Assessed

2024 Reading Mathematics Science Transcript Studies

4, 8, 12 4, 8, 12

4, 8

2025 Long-term Trend ~

2026 READING

MATHEMATICS

Civics U.S. History

4, 8 4, 8

4, 8 4, 8

2028 Reading

Mathematics

SCIENCE

Transcript Studies

4, 8, 12 4, 8, 12 4, 8

4, 8, 12 4, 8, 12

4, 8 4, 8

2029 Long-term Trend ~ 2030 Reading

Mathematics

CIVICS

U.S. HISTORY

4, 8 4, 8 8, 12 8, 12

4, 8 4, 8 xv

2032 Reading

Mathematics Science

WRITING

Transcript Studies

4, 8, 12 4, 8, 12 4, 8, 12 4, 8, 12

4, 8, 12 4, 8, 12

2033 Long-term Trend ~

Notes:

(1) Grades tested are 4, 8, and 12 unless otherwise indicated, except that long-term trend assessments sample students at ages 9, 13, and 17 and are conducted in reading and mathematics.

(2) Subjects in BOLD ALL CAPS indicate the years in which a new framework is planned for implementation.

Assessments to be Implemented The assessment schedule (Exhibit D) provides a list of the assessments to be administered during this contract period, as well as future assessments for which development may be required in the contract period. However, please note that the schedule is subject to change. For example, the schedule may change to place NAEP assessments back on an “odd-year” schedule, as it was before the pandemic caused postponement of the 2021 NAEP assessments to 2022. If this occurs, assessments planned for fall 2025 and beyond are likely to be moved out one year. Any changes in assessment schedule will be reflected in updates to the NAGB assessment schedule webpage (https://www.nagb.gov/naep/assessment-schedule.html).

The flow of work on a NAEP assessment begins long before the operational assessment year and continues afterwards. Assessment content development begins several years before an assessment is fielded operationally; items are piloted typically two years before fielding operationally; and most operational innovations undergo proof-of-concept and field test stages in the years before operational implementation. Analysis and sampling plans must be developed well in advance of assessments and data collection operations, including contacting states, districts, and schools, as well as recruiting and training field staff will occur well before assessments go into the field. Reporting and dissemination, including the preparation of data files and documentation to be shared with researchers, as well as in-depth analysis by the NAEP program, extends well beyond the initial release of an assessment’s results. Further details on the scope and timing of work are included in the SOO chapters for each contract.

About the Statement of Objectives

The next part of the SOO is organized into two chapters, Task 7 the NAEP Support and Service Center (NSSC) and Task 8 the Data Collection (DC). While each contract will have specific responsibilities, there is a set of more general activities and associated tasks that are common to all the contracts described at the end of each chapter. Each general activity and the associated tasks are relevant to each contract and shall be included in each contract offer. The work below includes expectations for the NSSC and DC team. This team will work with the other teams and Coalition members to ensure a successful implementation of the NAEP program.

7. The National Assessment of Educational Progress (NAEP) Support and Service Center

The purpose of the NAEP support and service center (NSSC) is to fully support the administration and implementation of the NAEP assessments as well as the partnership between the NAEP program, states and participating large urban districts. The service center will provide the necessary support to these jurisdictions primarily through the training and assistance given to NAEP states and district coordinators. The service center will provide products, processes, materials, and trainings for all phases of the various NAEP assessments. The goal of training is to ensure that the coordinators can complete the responsibilities for the successful implementation of their required activities for NAEP in their respective states. The support and training of NAEP coordinators should extend beyond the operational implementation of NAEP and should provide meaningful training and resources for state level analysis, reporting and dissemination of NAEP results.

NCES works with states and participating districts at many levels to ensure the successful implementation of NAEP assessments, international assessments, and other surveys. States and districts play critical roles in NAEP’s ability to administer an assessment, report valid and reliable results and stay at the forefront of large-scale assessment. This mutually beneficial relationship is key to ensuring that NAEP and other NCES activities are supported at both the local and Federal level. The states and large urban districts are seen as partners with NCES.

The service center will also support NCES in the achievement of its goals including overall engagement with states and large urban districts surrounding NAEP, international assessments and other possible surveys. One important goal for NCES is to maintain its ongoing efforts to inform states, districts, researchers, and other key stakeholders regarding the value and purpose of NAEP is in the education and large-scale assessment environment. The service center will support NCES in its understanding of NAEP’s position with key stakeholders at the state and district level as well as support NCES in its outreach to these audiences. The service center will support NCES in understanding the educational assessment landscape and its position within that landscape.

State and district coordinators serve as the liaison between NCES and the state and district education agencies (SEA). NCES utilizes individualized contracts with state education agencies to create the NAEP states and Trial Urban District Assessment (TUDA) coordinator position and to establish a closer link between a United States Department of Education program and the schools and students that are sampled to participate in NAEP assessments. The coordinators are responsible for creating an increased level of awareness, promoting an overall general understanding of NAEP within their agency, districts, schools, and with the public.

NCES provides funding to states, the District of Columbia, Puerto Rico, and the Department of Defense Schools to employ a full time NAEP state coordinator (NSC). The coordinators, while funded by NCES, are considered employees of their respective states, and therefore are integrated into the state departments of education. NSC’s are engaged and supported in NAEP activities for 90 percent of their time. The SEA is responsible for the hiring and maintenance of the coordinator position for their state. This specific location of the position within the SEA can vary from the data and accountability to assessment departments. Each SEA determines the location of the position based on its own organizational structure. There is a predictable amount of turnover in the coordinator position. The training and support provided needs to be reflective of varying level of experience of those holding the position. (A copy of the general performance works statement for the NSC’s outlining the requirements is available in Attachment 3, NCES support and service center technical exhibit)

The TUDA program was expanded from five in 2002 to twenty- seven large urban districts in 2017. As a result of this ongoing expansion, the TUDA program is now approximately half the size of the state program; making it essential that the support and training for TUDA districts reflects its emergence as one of the key components of the NAEP program. Funding is not provided for the district coordinators. The TUDA districts volunteer to participate in the program. The TUDA coordinators often hold other positions within their districts and are not solely responsible for the NAEP activities; a team within each district often completes the required tasks.

Therefore, the training and support provided to the districts needs to reflect the complex organizational of the participating districts.

State and district coordinators have a variety of responsibilities as part of the NAEP program.

The role of the state and district coordinator includes:

Ensuring the accuracy of school and sampling information, Encouraging and managing the participation of sampled schools, Coordinating the NAEP administration in the states and districts, Analyzing and reporting NAEP results at the state and district level, Coordinating the use of NAEP results to inform policy and program planning, Assisting in the recruitment and participation of districts and schools in international assessments including Trends in International Mathematics and Science Survey (TIMSS), the Programme for International Student Assessment (PISA) and others, and, Enhancing assessment literacy among educators, policy makers, and the public within the individual states.

Currently, all but one states have employed a federally funded full-time NAEP coordinator. The remaining state provide a point of contact to serve as a coordinator for required NAEP activities, who is often referred to as a non-funded coordinator. NAEP experience suggests that 20 to 25 percent of coordinators may be replaced in any calendar year. The TUDA coordinator position experiences a greater fluidity in the fulfillment of the role, so the support and training provided needs to meet the continuous turnover of district coordinators. Ongoing turnover is an expectation for the NAEP state coordinator program, and the support and training developed needs to reflect the ever-changing experience and level of familiarity with NAEP of the current and incoming coordinators. Support and service activities need to ensure the successful transition of the NSC in any state and/or district and provide a comprehensive onboarding process.

Additionally, coordination activities are instituted surrounding the recruitment of private schools in the NAEP assessment. The support center will provide training and support of recruitment of private schools through a private school coordinator that will work with a variety of organizations to gain support from the private school community.

Support and Service Center Scope

The contractor shall develop and maintain a close working relationship with each state and TUDA coordinator and provide processes and procedures for a virtual work environment. The service center shall be responsible for training and supporting the coordinators in all aspects of their jobs. The contractor shall maintain relationships with TUDA coordinators and as well as members of the districts’ internal NAEP/TUDA assessment teams. Finally, the contractor shall maintain ongoing relationships with private schools and organizations and associations affiliated with the private school community.

The contractor shall fully support all coordinators in the fulfillment of their responsibilities as outlined in their performance work statement. The contractor shall also support non- funded coordinators by providing them the same training and resources as their funded counterparts. The support for non-funded coordinators shall include additional support for operational tasks, continuous outreach, and guidance as well as coverage for travel to all trainings sponsored by NCES. The contractor shall provide the same support for the TUDA coordinators and other contractor staff, specifically those working on the National Indian Education Survey (NIES). The contractor shall cover all costs for travel to NCES-sponsored trainings for the TUDA coordinators.

The SSC will perform each of the following program activities to support the NAEP state and TUDA coordinators.

There are eight major activities in this procurement:

Task 7.1: Orientation and Training of Coordinators

Task 7.2: Continuous Training of Coordinators

Task 7.3: Technical Assistance and Support to States and Districts

Task 7.4: Conduct Evaluation of Center Products and Services and Conduct Evaluation of States’ Compliance with NAEP

Task 7.5: Support Private School Recruitment Private School Coordinator

Task 7.6: State and District Additional Support

Task 7.7: Annual Reports to NAEP

Task 7.8: Major IT Education Investment

Task 7.9: General Activities

The contractor shall employ innovative technologies can be used to enhance the usefulness and sustained impact of all trainings and workshops.

7.1 Orientation and Training of Coordinators

The NSSC is the mechanism for providing all the training, orientation, technical assistance and support necessary to make the coordinators successful in their respective states and districts. As previously described, there is a certain level of turnover expected for the NSC position. New NSC’s need to be introduced to the NAEP program and the operational activities required at the point in the NAEP cycle that they begin the position within their state and/or district. Orientation and onboarding are the opportunity to ensure that new coordinators are introduced to their roles and responsibilities as the NAEP coordinator and to ensure the continued and consistent implementation of the NAEP program in states and districts.

Training does not end after the orientation but continues throughout the NAEP assessment cycle and the contractor shall support the coordinators with ongoing training and support.

7.1.1 Orientation for New Coordinators

For new coordinators the contractor shall provide information packets that give the new coordinator an orientation to NAEP and to the job expectations within 30 days of employment of new staff.

The contractor shall hold in-person or virtual orientations for new coordinators to provide them with a comprehensive overview of their position expectations and resources available. The purpose of these orientation workshops is to give new coordinators the information, skills, and resources they need to perform their jobs as well as to give to them the opportunity to meet and begin to network among their colleagues.

In the past, orientations have in the past been three days in length but should be flexible enough to meet the needs of new coordinators. Whenever possible, NCES staff will attend and participate in orientation training, and will provide an overview of NAEP contractual activities.

The contractor shall work with the other members of the Functional Team to provide the necessary resource materials and technical support for these meetings.

TUDA orientations should occur when new districts are added to the TUDA program. The contractor shall conduct a separate, TUDA-specific orientation to discuss the unique needs and challenges of participating districts. The orientation will provide an overview of the NAEP program, the requirements for participation and an outline of the major roles and responsibilities of the TUDA coordinator and any district level team they establish. The contractor shall provide new TUDA coordinators with an onboarding experience or virtual orientation when joining their respective district teams to allow for a gap in timing from start date to an in- person orientation.

7.1.2 Develop and Conduct In- Person Training Activities

The contractor shall hold the three main types of in-person trainings required for the coordinators to successfully implement the NAEP program in their respective state/district:

NAEP Report Card Pre-release meetings to prepare coordinators for the release of district, state and national NAEP results, Training to support use of NAEP tools, processes and procedures related to operational activities including recruitment of schools and data collection activities and

In-depth training designed to improve coordinators’ abilities to report, analyze and communicate about NAEP results to key stakeholders within their respective jurisdictions effectively and efficiently.

The scheduling of these meetings is dependent on the NAEP assessment cycle. A schedule of the expected NAEP assessments is included in Exhibit C and should be referenced in determining the required number of workshops. Funding for the travel to these training sessions is included in the budgets provided to the states except for the orientation session. Most of these trainings, when possible, should be in the Washington DC metro area to ensure the inclusion of NCES staff.

7.1.2.1 Conduct NAEP Report Card Pre-Release Workshops

The contractor shall hold a workshop prior to the release of the state and district level NAEP data. The purpose of this type of workshop is to give state and district coordinators the opportunity to learn more about the assessment(s), to review their state/district NAEP results prior to the official U.S. Department of Education’s release, to prepare for their states/districts own release activities and to help states/districts plan to conduct their own analysis of their data using NAEP data and tools. These workshops are typically attended by the coordinators and up to two additional state education agency staff and three district staff. The contractor shall support the coordinators and additional staff at the workshop; including providing the funding for the two additional staff members to attend. Report Card pre-release workshops will be scheduled in conjunction with the release of Nation’s Report Card reports that contain state and district-level data.

The release of TUDA results has been streamlined to occur at the same time as the release of the national and state report cards. The contractor shall conduct the pre-release workshop for the participating TUDA districts prior to the release of the TUDA results. For the district workshops the NSSC will be responsible for the total funding for all participants.

7.1.2.2 Conduct Assessment Operations Training

Prior to NAEP assessments with state and district level components, the contractor shall provide training to support state and TUDA coordinators in the operational components of NAEP. The contractor shall conduct the operations training to prepare state and district coordinators on the variety of tasks associated with the administrative implementation of a large state/district year of NAEP. The contractor shall prepare all presentations, resources and activities that will be used during the workshops. Coordinators need to understand and be able to complete required communications to districts and schools, understand the systems that will be used for communication with participating districts and schools and understand their overall role in the preparation for a NAEP state/district year.

As described in the Sampling and Data Collection (SDC) portion of the performance work statement , the expectation is that the NSSC and SDC contractors will work together to train the coordinators on the following procedures used for all NAEP data collections covered by this procurement: notifying and transmitting lists of sampled schools to districts; securing participation of the schools; ensuring the complete and accurate student list submission for each eligible sampled school; and corresponding with schools and field staff regarding NAEP assessment issues. Additional support shall be provided (as needed) to states with a large sample of schools and for states without a full-time and/or permanent NAEP state coordinator.

7.1.2.3 Professional Development Workshops

The contractor shall provide the coordinators with training to further enhance their understanding of…