91990018R0018_Amendment_0002_FormSF30(1).pdf

NSN 7540-01-152-8070 STANDARD FORM 30. (Rev. 10-83) Previous Edition unusable Prescribed by GSA FAR (48 CFR) 53.243

Page of Pages AMENDMENT OF SOLICITATION/MODIFICATION OF CONTRACT 1. Contract ID Code

2. AMENDMENT MODIFICATION NO. 3. EFFECTIVE DATE 4. REQUISITION/PURCHASE REQ. NO. 5. PROJECT NO. (if applicable)

6. ISSUED BY CODE 7. ADMINISTERED BY (If other than item 6) CODE

(x) 9A. AMENDMENT OF SOLICITATION NO.

9B. DATED (SEE ITEM 11)

10A. MODIFICATION OF CONTRACT/ORDER NO.

8. NAME AND ADDRESS OF CONTRACTOR (NO., Street, Country, State and ZIP Code)

CODE FACILITY CODE

10B. DATED (SEE ITEM 13)

11. THIS ITEM ONLY APPLIES TO AMENDMENTS OF SOLICITATIONS

The above numbered solicitation is amended as set forth in item 14. The hour and date specified for receipt of offers is extended, is not extended.

Offers must acknowledge receipt of this amendment prior to the hour and date specified in the solicitation or as amended, by one of the following methods:

(a) By completing items 8 and 15, and returning ______ copies of amendment; (b) By acknowledging receipt of this amendment on each copy of the offer submitted;

or (c) By separate letter or telegram which includes a reference to the solicitation and amendment numbers. FAILURE OF YOUR ACKNOWLEDGMENT TO BE RECEIVED AT THE PLACE DESIGNATED FOR THE RECEIPT OR OFFERS PRIOR TO THE HOUR AND DATE SPECIFIED MAY RESULT IN REJECTION OF YOUR OFFER. If by virtue of this amendment your desire to change an offer already submitted, such change may be made by telegram or letter, provided each telegram or letter makes reference to the solicitation and this amendment, and is received prior to the opening hour and date specified.

12. ACCOUNTING AND APPROPRIATION DATA (If required)

13. THIS ITEM ONLY APPLIES TO MODIFICATION OF CONTRACTS/ORDERS. IT MODIFIES THE CONTRACT/ORDER NO. AS DESCRIBED IN ITEM 14.

Check One

A. THIS CHANGE ORDER IS ISSUED PURSUANT TO: (Specify authority) THE CHANGES SET FORTH IN ITEM 14 ARE MADE IN THE

CONTRACT ORDER NO. IN ITEM 10A.

B. THE ABOVE NUMBERED CONTRACT/ORDER IS MODIFIED TO REFLECT THE ADMINISTRATIVE CHANGES (such as changes in paying office, appropriation date, etc) SET FORTH IN ITEM 14, PURSUANT TO THE AUTHORITY OF FAR 43.103(b).

C. THIS SUPPLEMENTAL AGREEMENT IS ENTERED INTO PURSUANT TO AUTHORITY OF:

D. OTHER (Specify type of modification and authority)

E. IMPORTANT: Contractor is not, is required to sign this document and return _______ copies to the issuing office.

14. DESCRIPTION OF AMENDMENT/MODIFICATION (Organized by UCF section headings, including solicitation/contract subject matter where feasible)

Except as provided herein, all terms and conditions of the document referenced in item 9A or 10A, as heretofore changed, remains unchanged and in full force and effect.

15A. NAME AND TITLE OF SIGNER (Type or print)

16A. NAME AND TITLE OF CONTRACTING OFFICER (Type or print)

15B. CONTRACTOR/OFFEROR 16B. UNITED STATES OF AMERICA

(Signature of person authorized to sign)

15C. DATE SIGNED

(Signature of Contracting Officer)

16C. DATE SIGNED

See Attachment Page.

1 452

91990018R00180002 FEB 21, 2019

91990018R0018

JAN 22, 2019

See Schedule

Attachment Page

PAGE 2 OF 452 91990018R00180002

The purpose of this amendment is to: 1) update L.14 Addendum to 3452.239-71 ED Security Requirements; 2) update L.15 Instructions to Offerors; 3) update Attachment A Performance Work Statement; 4) update Attachment B Quality Assurance Surveillance Plan/Award Fee Evaluation Plan; and 5) to provide Government responses to clarification questions received.

Accordingly, the solicitation is amended as follows:

J.1 309-1a List of Attachments

Attachment A – Performance Work Statement Attachment B – Quality Assurance Surveillance Plan/Award Fee Evaluation Plan Government Responses to Clarification Questions

PAGE 3 OF 452 91990018R00180002

i

(Amendment 0002)

Attachment A

Performance Work Statement

National Assessment of Educational Progress

(NAEP)

2019-2024

PAGE 4 OF 452 91990018R00180002

ii

Amendment 0002

ATTACHMENT A ..................................................................................................... I

1 ALLIANCE PLANNING AND COORDINATION (PC)

1.1 Conduct strategic planning

1.2 Support operational programmatic planning

1.3 Coordinate NAEP assessment activities across multiple contractors and

NCES

1.4 Coordinate NAEP R&D

2 PROGRAM SUPPORT MANAGEMENT

2.1 Project Liaison and Support

2.2 Project Liaison and Support

2.3 Develop and Maintain Contract and Financial Management Support Tools

3 DESIGN, ANALYSIS AND REPORTING

3.1 Design

3.2 Psychometric and Statistical Analysis

3.3 Research and Special Projects

3.4 Reporting

3.5 Additional General Activities

4 ITEM DEVELOPMENT

4.1 Cognitive Assessment Development

4.2 Develop questions for the survey questionnaires

4.3 Item/survey questions reviews

4.4 Item Development Tracking System (IDTS)

5 SAMPLING AND DATA COLLECTION

5.1 Design and Select Samples

5.2 Collect Data

5.3 Generate Sampling Weights

5.4 Conduct and Report the High School Transcript Study

6 THE NAEP SUPPORT AND SERVICE CENTER

6.1 Orientation and Training for Coordinators

6.2 Technical Assistance and Support to State and Districts

6.3 Supporting the NAEP Network Support Private School Recruitment

6.4 State and District Additional Support

6.5 Annual Reports to NAEP

7 MATERIALS, DISTRIBUTION, PROCESSING AND SCORING

3. TRACK, RECEIVE COMPLETED AND UNUSED MATERIALS BACK

FROM THE FIELD, AND PROCESS AND INVENTORY FOR ACCURACY.

THIS DEPENDS ON ONGOING QC AND RECEIPT CONTROL

7.1 Materials Preparation and Distribution

7.2 Collection, Receipt Control, Data Capture and Processing

7.3 Scoring and Special Studies

PAGE 5 OF 452 91990018R00180002

ii

8 NAEP WEB/TECHNOLOGY DEVELOPMENT, OPERATIONS AND

MAINTENANCE

8.1 Provide Public Websites Support and Coordination

8.2 Develop and Maintain Private Websites and Applications

8.3 Provide Hosting, Hardware, and Operations Support to AD Systems

Please refer to General Section, Chapter 11 for detailed information on Compliance with ED IT Security Policy, Internet Protocol version 6 (IPv6) and Reporting of Data Security Breaches for the NAEP System

8.4 Provide Training and Demonstrations for Applications, Data Releases, and Website Refreshes

8.5 Support Assessment Division Business Processes

9 NAEP PLATFORM DEVELOPMENT CONTRACT

9.1 Develop the DBA Platform

9.2 Ensure Quality of Platform and Supporting Systems

9.3 Operate and Support the Platform and Supporting Systems

9.4 Conduct Studies and R&D for the Platform and Supporting Systems

9.5 Develop the NAEP Item Bank

9.6 Convene Platform Development Steering Meeting

10 NATIONAL ASSESSMENT FOR EDUCATIONAL PROGRESS DATA

WAREHOUSE

INTRODUCTION-THE NAEP DATA WAREHOUSE

10.1 Project Management

10.2 Develop NAEP Data Warehouse Prototype

10.3 Develop and Maintain NAEP Data Warehouse

10.4 (Optional) Develop and Maintain NCES Data Warehouse

10.5 Optional) Develop and Maintain IES Data Warehouse

11 GENERAL ACTIVITIES

11.1 Project Oversight and Management

11.2 Coordination with other NAEP Contractors on Additional Task Component

Assignments

11.3 Provide Support

11.4 Transfer of Project Materials

11.5 Other Requirements

Appendix 11.1

Data Submission Requirements

File: Annual Budget Forecast

File: Annual Percent Complete/Schedule Model

File: Monthly Actual Cost

File: Monthly Schedule Status

PAGE 6 OF 452 91990018R00180002

Amendment 0002

Procurement Overview

The U.S. Department of Education intends to enter into a five-year contracts with qualified entities to establish an Alliance/team of contractors to carry out the design, item development, sampling, materials preparation and distribution, data collection, scoring, analysis, and reporting of one of its flagship programs, the National Assessment of Educational Progress (NAEP). NAEP is approaching its 50th anniversary as a mature student assessment program in a rapidly changing environment. It is incumbent on the program to maintain its historical role of reporting educational progress while at the same time reflecting changes in education and taking advantage of advances in education assessment, learning sciences, and in technology.

The National Center For Education Statistics (NCES) has made a commitment to lead the field of large-scale, technology-based assessments and has identified five goals for the activities defined in this. They are:

Strengthen the utility of NAEP: enhancing NAEP’s current functionality to ensure its continued leadership in large scale assessment.

Adopt new approaches to assessment: integrate new and thoughtful approaches to large scale assessment, including but not limited to design, analysis, and administration and reporting to ensure successful and cost effective implementation of NAEP.

Lead in assessment innovation: lead the field in research and development of new item types, expand measurement of cognitive and non-cognitive skills and enhancement of assessment delivery Enhance communication: enhance and effectively communicate results, reports and finding from NAEP and related research efforts , and Improve operational processes.

There are numerous environmental factors that will influence NAEP during this period, many of which will be referenced throughout this Performance Work Statement (PWS).

Examples include:

ongoing transition from a paper-and-pencil to digitally--based assessment, the emerging interest in international comparisons for states and districts, advances in assessment development and new item types, development of measures of preparedness for post-secondary education and work, adoption of new content standards across most states, inclusion of more contextual information in reporting, PAGE 7 OF 452 91990018R00180002 the linking of NAEP and other NCES surveys, such as international assessments and the National Teacher and Principal Survey (NTPS), growth of virtual schools and out-of-school and technology-based learning by students, and budget reductions throughout the education and assessment enterprises.

NCES envisions this procurement period both as vital to maintaining a quality assessment and as a continuation of its transition to the future. There will have to be balance between the requirements specified herein and the on-going planning for the future of NAEP.

NCES has undertaken several ongoing activities to develop a vision for NAEP 2020 and beyond; those activities are documented at http://nces.ed.gov/nationsreportcard/future.

Purpose of NAEP

The purpose of NAEP is to provide periodic, fair and accurate measurement of educational achievement in reading, mathematics, science and writing – the core assessments – as well as other content areas including United States history, geography, civics, economics, foreign languages, technology and engineering literacy, and the arts.

The assessment results are representative of the nation, regions of the country, and where appropriate, states and selected urban districts at grades 4, 8 and 12. NAEP also reports on national long term trend assessments in reading and mathematics for students at ages 9, 13, and 17 in order to measure educational progress since the 1970s.

Legislative Authority for NAEP

The Every Student Succeeds Act (ESSA) signed December 2015, reauthorizes the following provisions for NAEP:

‘‘(D) the State will participate in the biennial State academic assessments in reading and mathematics in grades 4 and 8 of the National Assessment of Educational Progress carried out under section 303(b)(3) of the National Assessment of Educational Progress Authorization Act (20 U.S.C. 9622(b)(3)) if the Secretary pays the costs of administering such assessments;

‘‘(1) SUBGRANTS.—A local educational agency may receive a subgrant under this part for any fiscal year only if such agency has on file with the State educational agency a plan, approved by the State educational agency, that— they participate, if selected, in the National Assessment of Educational Progress in reading and mathematics in grades 4 and 8 carried out under section 303(b)(3) of the National Assessment of Educational Progress Authorization Act (20 U.S.C. 9622(b)(3));

The National Assessment of Educational Progress Improvement Act, Part C of the Public Law 100-297, which is now part of U.S. Code (20 U.S.C. 9010 and 9011), legislatively

PAGE 8 OF 452 91990018R00180002

defines the purpose of the NAEP “to improve the effectiveness of our nation's schools by making objective information about student performance in selected learning areas available to policymakers at the national, state and local levels. To enhance its utility, such information should be both representative and comparable, and should be maintained in a manner that ensures the privacy of individual students and their families.”

A critical aspect of NAEP is the reporting of trends in student performance over time.

Administration

The Commissioner of Education Statistics (Commissioner) of NCES develops, administers, and reports NAEP while the National Assessment Governing Board (Governing Board) , appointed by the Secretary of Education but independent of the Department, sets policy for NAEP.

Main NAEP assessments are made up of the following operational components:

National assessments include mathematics, reading, science, writing, the arts, civics, economics, geography, U.S. history and the technology and engineering literacy assessment (TEL). These assessments follow the content frameworks developed by the Governing Board and use the latest advances in assessment research. Most subjects are assessed at grades 4, 8, and 12— although not all grades are assessed each time. Students in public and private schools are assessed.

State assessments are administered in four subjects (mathematics, reading, science, and writing). Only public schools are included in state-level assessments.

The trial urban district assessment (TUDA) is a multiyear study of the feasibility of a district-level NAEP administered in four subjects (mathematics, reading, science and writing). Twenty-seven districts currently participate in TUDA.

The NAEP High School Transcript Study (HSTS) is conducted in high schools selected for NAEP mathematics and science assessments. Results are reported primarily at the national level.

The NAEP Middle School Transcript Study (MSTS) is conducted in middle schools selected for NAEP mathematics and science assessments. Results are reported primarily at the participating urban district level.

The National Indian Education Survey (NIES) is conducted in schools with American Indian/Alaska Native students. Results are reported at the national level and for states with large samples of AI/AN students. (OPTION)

While Main NAEP comprises the bulk of the program, NAEP is a system of assessments that seeks to provide a comprehensive measure of educational progress in the country.

PAGE 9 OF 452 91990018R00180002

Other major components are the long-term trend assessments, special studies and studies for insuring the validity of NAEP items. They are as follows:

Long-term trend assessments (LTT) are given at the national level only, and are administered in a different manner from that used for main NAEP assessments. LTT reports results in mathematics and reading that represent trends since the 1970s. LTT assessments are scheduled to be administered every four years.

Special studies are periodically conducted to answer a variety of questions.

Examples of special study topics include the following:

Inclusion and accommodations in NAEP, Accessibility of assessments items to measure across ability distribution, Linking between NAEP and international studies-Trends in Mathematics and Science Study (TIMSS)Linking Study, Defining measurement of Socioeconomic Status, and

Various technology-based assessment studies.

Pilot studies are conducted on all items prior to their use in NAEP assessments. All items are reviewed and go through several improvement activities. Cognitive labs and small-scale tryouts are conducted as needed on new and complex item types.

The NAEP Alliance Concept

All NAEP assessments have critical path processes that are conducted by individual contractors as part of the NAEP Alliance; however they are dependent upon one another for completion. Broadly, these processes are as follows:

assessment design, item and platform development, sampling, weighting and data collection, materials, distribution and scoring, scaling, estimation and analyses, and reporting

Further, all of the processes include the following:

PAGE 10 OF 452 91990018R00180002

state involvement to ensure participation of schools and the quality of the sampling and data collection processes, the use of the technology in various administrative functions and in the administration of digitally-based assessments, and planning and coordination to ensure that the various processes are aligned and coordinated with one another and the schedule is maintained for the required six and twelve-month reporting.

Current Operating Practices

The NAEP assessment generally consists of two separately timed 30-minute cognitive blocks followed by separately timed contextual questions. NAEP subjects using the common block structure that are spiraled together and administered in the same session.

Reading and mathematics assessments have the highest priority of all assessments for support and scheduling with item development occurring at least three years before an operational assessment. These items are pilot tested up to two years in advance as part of an operational assessment.

NAEP operates on a general principle that pilot tested items are not changed prior to use in an operational assessment. Pilot tested items that are of high quality but not used in the upcoming operational assessment are banked for future use. Pilot test data are not publicly released nor used to provide performance estimates.

Main NAEP assessments are administered during a six-week window beginning the last week of January and concluding the first week of March. In extreme cases, such as a natural disaster or major snowstorm, the second week in March may be used for make-up assessments. Assessments within states and large urban districts are spread across the entire assessment window.

States and districts receiving Title I funds are required to participate in grade 4 and 8 reading and mathematics assessments every two years (i.e., mandatory assessments). All other assessments are voluntary for states. School and student participation are voluntary for all assessments.

NAEP samples are geographically representative of the reporting jurisdictions and subgroups. Schools and students are chosen with a measurable probability of selection.

In a state-level assessment, the national sample is composed of the aggregate of state samples. However, when a state elects not to have the state-representative samples needed to report at the state level, some students in that state may be included to ensure representativeness of the national sample.

Long-term trend assessments are administered: in the fall to 13-year-olds, in the winter to 9-year-olds, and in the spring to 17-year-olds.

PAGE 11 OF 452 91990018R00180002

Testing accommodations are provided to students with disabilities and English learners to remove barriers to their participation, as long as the constructs tested are not altered by the accommodations. For example, the reading assessment cannot be read aloud to students or translated to another language, as the NAEP reading framework defines the construct to include reading printed text in English. Data released from the assessments must report on students with a disability (SD) and English learners (EL) as a percentage of all students who completed the assessment. Within those two populations, the percent excluded, the percent tested with accommodations, and the percent tested without accommodations are reported.

The Governing Board plans and executes the initial release of reports of assessments after approval by the Commissioner.. National and state reading and mathematics reports at grades 4 and 8 are required to be available for release within 6 months after the completion of data collection. All other NAEP initial data releases must be available within 12 months after the completion of data collection, or on a different release schedule by the Governing Board. Releases include development of electronic materials and websites for public consumption. NAEP continues its move to more electronic materials and to greater use of the Web and social media for release activities.

Reports and data products from assessments on the Governing Board schedule where data are being released for the first time are identified as “initial release” and are differentiated from other NCES reports and products in the review process. “Initial release” products are reviewed by the Assessment Division (AD), at the NCES center-wide level for signoff by the Chief Statistician, by the NCES Commissioner, and by the Director of the Institute of Education Sciences (IES). Other products, in addition to the reviews by the Assessment Division, the Chief Statistician, and the NCES Commissioner, undergo a peer review (either internal or external) organized by the IES.

To the extent possible, NAEP designs, samples and analyses are conducted to maintain trend over time.

Aspects of the NAEP design may change during this procurement period (2019-2024).

As outlined below, there are a number of challenges that will need to be addressed during the procurement period. Consequently, there will be the need for flexible approaches to design modifications as NAEP’s role continues to evolve either through changes in legislation, Governing Board and Department of Education policies or changes in the fields of educational measurement, education, or schooling.

Current Operating Cycle

The project consists of preparing for and conducting parallel assessments in different stages.1 In addition to at least four overlapping assessment cycles progressing at the same time, the NAEP project also alternates every other year in terms of size of the data collection. During odd-numbered years (2019, 2021, and 2023), the NAEP data contractors will collect, score, analyze and report information representative of not only

1 When there is a new framework being created by the Governing Board, the cycle takes five or more years.

PAGE 12 OF 452 91990018R00180002

the nation and geographic regions, but also all states and selected large urban districts in reading and mathematics and alternately science or writing. In even-numbered years (2020, 2022 and 2024), NAEP conducts assessments with samples large enough to provide national and regional ability estimates only. Special studies and pilot testing are integrated into the operational assessments, as needed.

All Alliance contractors are actively involved at all times during the calendar year and activities are often completed on tight timelines. The following paragraphs provide a sampling of the scheduling complexities.

The contractor responsible for the design of the assessment must develop and deliver a preliminary design up to four years in advance of the assessment so that all other contractors can prepare. The sampling contractor, for example, must begin the sample selection process in January the year before the assessment so states can be notified in May for the subsequent school year.

The item development contractor will be writing, reviewing and pilot testing items for four to six content areas across grades 4, 8 and 12 each year. Items are reviewed by NCES, a standing committee of content experts, the Governing Board and the states throughout the year.

For the data collection contractor, the peak period is January to March, when substantial numbers of additional people are employed to collect data from students. But other times of the year, staff will be involved in developing training for data collectors, identifying schools to be sampled in the next year’s assessment, creating sampling weights, conducting the high school and middle school transcript studies, and working with private schools.

The reporting contractor must meet a tight six-month or twelve-month deadline for all initial report releases. In addition, the contractor analyzes and drafts other reports and conducts other studies or analyses throughout the year.

The materials and distribution contractor begins their process of compiling materials in the late spring for distribution during the following winter based on sample schools identified in early May and sample students identified in the October – November period. The contractor is responsible for 24-hour turn-around of additional materials during the data collection in the January – March period.

The contractor supporting the 50 state and 27 district coordinators for their various activities throughout the sampling, data collection, and reporting processes maintain weekly contact via electronic media, conducts three (3) to six (6) in-person workshops each year, and provide orientation to NAEP for the 20-25 percent of new coordinators each year.

The web contractor supports the release of reports, new items for the item tools, the latest data for the NAEP Data Explorer, as well as updating the online technical documentation for the assessment process. This contractor

PAGE 13 OF 452 91990018R00180002

periodically updates the Alliance’s intranet site, as well as the web page review site.

The platform development contractor implements, supports, maintains, and enhances a NAEP digitally based assessment (DBA) platform and supporting technology systems. NAEP’s DBA Platform comprises the full end-to-end technical framework, or environment, that delivers the NAEP assessment to students.

Challenges

NAEP’s primary purpose is to provide a fair and accurate measure of student achievement in various content areas and to report trends over time. At the same time, NAEP has to be responsive to environmental and policy changes and to the emerging needs have national and state policymakers. For example, the emergence of better and cheaper technology and their integration into the classroom and out-of-school learning experiences of the nation’s students dictate the need to develop national and state assessment that are innovative, technology-based and forward thinking. NAEP will be challenged to identify the type of infrastructure that will be necessary to deliver innovative assessments while maintaining a standardized administration model high levels of item security, and low operational burden and cost.

In addition to delivering assessments via appropriate technology, it will become increasingly more important that the application of technology to all parts of the NAEP system is updated to keep pace with the latest industry advances in a pragmatic and cost-effective manner. Many of NAEP’s administrative processes can be paperless.

Whenever commercial-off-the-shelf (COTS) software is appropriate and available, it should be given strong consideration in the evaluation of solutions. In addition some custom systems may also be developed. All of these systems will have to be maintained and enhanced so that NAEP can easily incorporate new innovations in hardware and software with minimal impact on operations and/or budget.

Other challenges that will have to be addressed during this procurement period include the following:

International comparisons. There is a growing demand by national, state and local policy makers and the general public to know how their schools compare not only to the nation and states but to other countries. NAEP, the only state-representative assessment, is in an excellent position to serve as the link with international assessments to provide state and district estimates for comparison with other countries. NAEP implemented a comprehensive study in 2011 to link NAEP with the Trend in Mathematics and Science Survey (TIMSS). However, there are calls to link NAEP with other international assessments.

Preparedness for Post-Secondary Education and Work. A different type of challenge arises from the call for the use of the 12th grade NAEP reading and mathematics assessments in a way that captures and measures preparedness for participation in post-

PAGE 14 OF 452 91990018R00180002

secondary education, the military, and/or the workplace. https://www.nagb.gov/focus-areas/reports/preparedness-research.html

Accessibility. NAEP assessments should be developed to maximize accessibility for all students to the assessment items.

Improving measurement precision. Adapting which test blocks are administered to an examinee holds the promise of providing better measurement of students across the full range of student performance.

Changing state content standards. States’ content standards are continuously changing in a post Common Core State Standards initiatives environment. States have continued to modify standards and assessments during a time of changing accountability measures.

Maintaining Trend. Critical to NAEP’s mission of reporting on the progress of education is maintaining trend. However, to remain relevant NAEP has to respond to changing environments. NAEP routinely faces this challenge through bridge studies that provide statistical evidence for continuing trend lines. However, with the major shifts in what and how NAEP assesses student performance, there may be other more efficient and cost-effective methods that can be used.

Procurement Principles

There are six primary principles the Alliance contracts shall adhere to in this procurement:

NAEP data must be accurate, of high quality and on time.

NAEP must continue to be a leader in large-scale assessment especially in meeting the upcoming challenges.

NAEP must adapt to rapidly changing environmental factors.

NAEP must be designed in a way that maintains reporting trends over time.

NAEP must improve efficiencies in both costs and burden on participants.

NAEP must ensure that the diversity of a workforce that characterizes the full range of NAEP stake-holders is represented at every level of NAEP operations. To do so NAEP must make sure that every attempt possible is made to incorporate professional staff who represent that diversity and that each of the NAEP contractors increase the participation of small and minority-owned businesses within each contract.

Procurement Components

As seen in the following organizational chart (Exhibit A), the new NAEP Alliance Procurements will consist of ten critical path components with each component a separate

PAGE 15 OF 452 91990018R00180002

contract award. Ten of the components are described in chapters 1 through 10 in this PWS, with each chapter corresponding to a separate contract. Chapter 11 describes activities common to all contractors and that shall be included in each offeror’s proposal.

The components are as follows:

1. Planning and Coordination (PC)

2. Program Support Management (PSM)

3. Design, Analysis and Reporting (DAR)

4. Item Development (ID)

5. Sampling and Data Collection (SDC)

6. NAEP Support and Service Center (NSSC)

7. Materials, Distribution, Processing and Scoring (MDPS)

8. Web/Technology Development, Operations, and Maintenance (WTDOM)

9. NAEP Platform Development (NPD)

10. NAEP Data Warehouse (NDW)

The shading, or groupings, in Exhibit A indicate relationships that NCES believes provide the best quality and greatest cost and time efficiencies for the government. For example, The Design, Analysis and Reporting contract (DAR) develops the design for each assessment including specifications for framework coverage and number of items and content blocks that are to be developed for each assessment and passes that information to the Item Development contractor (ID). ID and NAEP Platform Development (NPD) collaborate to develop the items using content specialists that are then available to DAR for use in the design phase, thus eliminating the need for staffing duplication. DAR conducts the item analysis after pilot testing and works with ID and NPD to determine, based on the item data, which items are suitable for inclusion in the assessment. They work together to identify items that need additional work and re-pilot testing.

The sharing of DAR analysts for this specific portion of the development process eliminates the need for ID to have analysts on staff. Close proximity of the two teams makes it easier to share staff at time-critical junctures in the program.

Having the Planning and Coordination contractor (PC) located with DAR provides similar time and cost efficiencies. PC is responsible for overall strategic planning for NAEP, coordinating the implementation plan for

PAGE 16 OF 452 91990018R00180002

individual assessments built on the DAR design document, and for planning and managing NAEP’s research agenda. These functions require technical and psychometric expertise which can be matrix from the DAR analysts.

Similarly, the relationship between the Sampling and Data Collection contract (SDC) and the NAEP Support Services Center contract (NSSC) eliminate the need for staffing and resource duplication. SDC selects the sample based on a sampling frame that has been reviewed by the states under the supervision of the NSSC making the initial samples more accurate and representative. After the samples are selected and forwarded to the states, the NSSC works with the NAEP state and district coordinators to add/eliminate schools that have opened/closed or are otherwise unavailable for the assessment. The NSSC trains and supervises the state and district NAEP coordinators in the processes and quality standards necessary for a high quality NAEP and in the recruitment of schools and ensures that they know the procedures and have the materials needed to participate in the assessment including the legislatively required parental notification. The NSSC does not need to employ sampling statisticians to develop training for the states and the SDC does not need to employ, train, and maintain state-based personnel to recruit and work with schools.

The ten critical path components shall be bid as an alliance, with contractors working together to submit a unified group of proposals that will be accepted or rejected in their entirety. NCES will award ten contracts, one for each component with more than one contract being awarded to a single organization as appropriate.

PAGE 17 OF 452 91990018R00180002

Exhibit A. The NAEP Alliance

1. Alliance Planning and Coordination

2. Design, Analysis and Reporting

3. Item Development

4. Program Support Management

5. Sampling and Data Collection

National Center for Education

Statistics

6. Support and Service Center

7. Materials, Distribution, Processing, and Scoring

8. Web/Technology Development, Operations

9. NAEP Platform Development

10. NAEP Data Warehouse

PAGE 18 OF 452 91990018R00180002

NAEP Contracts not Included in the Alliance

NAEP Quality Assurance Technical Panel

NAEP Logistics

NAEP Validity Studies Panel Publications, Outreach and Dissemination Support (PODS) National Indian Education Support Education Statistics Support Institution Network

The alliance components, described in Exhibit B, represent all of the NAEP assessments and special studies. The Alliance contractors shall work together as a team, with each team member responsible for processes and products that interface with the processes and products of one or more of the other team members. A failure by one contractor may well affect the performance of all. Therefore, all components are included in one consolidated PWS, and proposals must be in the form of a team bid that covers the alliance in its entirety. Any organization interested in making an offer on one or more components shall seek out other organizations that are recognized for complementary expertise to collaborate on a team proposal.

PAGE 19 OF 452 91990018R00180002

Exhibit B: NAEP Alliance Components

1. Planning and Coordination

Ensures coordination among Alliance contractors; maintains data for tracking program progress; and provides logistical support for complaints and substantive comments.

2. Program Support Management

Manages the NAEP program in alignment with Project Management (PM) best practices to ensure proper scheduling, quality control, risk management and communication across contractors and handoffs.

3. Design, Analysis and Reporting

Designs all pilot and field tests, operational assessments, and special studies; analyzes data ensuring reporting of valid results;

proposes and prepares psychometric and statistical analyses compatible with previous NAEP methodologies; specifies data needed to meet the goals for reporting; and prepares reports.

4. Item Development

Develops cognitive items, scoring rubrics and survey questions;

assists in the training of scorers; conducts cognitive interviews/small-scale pilots of items, rubrics, and survey questions; translates items and survey questions; and conducts item reviews

5. Sampling and Data Collection

Selects samples; prepares sampling weights; administers assessments and collects data for pilot and field tests, operational assessments, and special studies; and ships completed assessment materials to the scoring sites. Conducts the High School Transcript Study and the Middle School Transcript Study.

6. Support and Service Center

Provides support, training, and resources to state and TUDA coordinators to ensure the accurate and timely sampling, administration and reporting of NAEP in each state and TUDA district.

7. Materials, Distribution, Processing and Scoring

Prepares and packages all assessment and auxiliary materials;

distributes assessment booklets and materials to the test administrators for each school; receives the materials from the schools; with ID and DAR contractor develops scoring training materials; and scores all assessments.

8. Web/Technology Development, Operations, and Maintenance

Develops, implements, and supports Internet-related applications and services; identifies and deploys emerging technologies and new products to improve NAEP’s web and other computer-based products and services.; monitors compliance with all NCES web requirements, and ensures timely release of quality products and services using Web technologies.

9. NAEP Platform Development

Develops NAEP assessment delivery platform utilizing a state-of-the-art, age-appropriate user interface (UI) and overall user experience (UX) that is consistent, intuitive, and accessible across all subjects and grade levels

10. NAEP Data Builds a NAEP Data Warehouse and Data Analytics to extract, PAGE 20 OF 452 91990018R00180002

Warehouse data load/store, integrate, analyze, index and search all of its datasets, databases, published reports, social media feeds, geographical information systems, Census.

Overview of the Work

Independently and not as an agent of the United States Government, the Contractor shall provide all personnel, materials, services, and facilities necessary for the project and perform the tasks as described below. The work to be covered under this PWS begins with the award of the contracts in 2019 with completion of the 2019 assessment, and ends in 2024 with development work for the 2025 assessments. Each assessment cycle includes pilot tests, and the operational assessments.

In November 2015, the Governing Board set the current schedule of assessments as presented in Exhibit C:

Exhibit C: NAEP Schedule of Assessments

Year Subject

National Grades Assessed

State Grades Assessed

TUDA

Grades Assessed

U.S. History* 8 Civics* 8

2018 Geography* Technology and

Engineering Literacy Reading 4,8,12 4,8 Mathematics 4,8,12 4,8

2019 Science 4,8,12 4,8 High School Transcript ~ Study

Reading Mathematics Writing

4,8 4,8 4,8,12

4,8 4,8

4,8 4,8

U.S. HISTORY 8,12

8,12 8,12 8,12

CIVICS

GEOGRAPHY

2022 Economics Technology and Engineering Literacy

PAGE 21 OF 452 91990018R00180002

Reading Mathematics Science High School Transcript Study

4,8,12 4,8,12 4,8,12

4,8 4,8 4,8

4,8 4,8 4,8

ARTS

FOREIGN LANGUAGE

Long-Term Trend

Notes:

(1) Grades tested are 4, 8, and 12 unless otherwise indicated, except that long-term trend assessments sample students at ages 9, 13, and 17 and are conducted in reading and mathematics.

(2) Subjects in BOLD ALL CAPS indicate the years in which a new framework is planned for implementation.

(3) The Governing Board intends to conduct assessments at the 12th grade in World History and Foreign Language during the assessment period 2019-2024.

Assessments to be implemented

The full NAEP assessment schedule adopted by the Governing Board, shown in Exhibit C above, outlines the range of activities that are planned during this contract period.

However, the current appropriations do not cover the full range of work implied by that schedule. As a consequence, the schedule is subject to change.

As depicted in Exhibit D, the flow of work in NAEP begins long before the assessment year. This chart reflects a sample of a schedule of assessments and indicates when development work must begin. There are a few implicit assumptions associated with the operational assessments. With each new framework or move to a technology-based format, NCES will conduct a bridge study to determine whether that trend can be maintained.

Beyond the scope of work described above, several special study options as outlined in Chapter 11 are under consideration. Additionally, World History and Foreign Language assessments may be considered at sometime within the cycle.

PAGE 22 OF 452 91990018R00180002

Exhibit D: NAEP Assessment Development Activities: Sample NAEP Work Plan

Year Item Development Pilot Testing Operational Assessments Analysis & Reporting

2023 Reading (4,8,12) 2021 Mathematics (4,8,12) Reading (4,8,12) 2018 U. S. History 2023 Mathematics (4,8,12) 2021 Reading (4,8,12) Mathematics (4,8,12) 2018 Civics 2023 Science (4,8, 12) Science (4,8,12) 2018 Geography

High School Transcript Study 2018 TEL 2019 Reading 2019 Mathematics

2024 Arts (8) 2021 Writing (DBA; 4,8,12) 2020 U.S. History (Start up 12)

2024 Foreign Language (12) 2022 U.S. History (DBA; 8, 12) 2020 Civics (Startup 12) 2019 Science 2022 Civics (DBA; 8, 12) 2020 Geography(Stratup 12) 2022 Geography (DBA; 8, 12) 2022 Economics (DBA, 12) 2022 Technology and Engineering Literacy (TEL) (8, 12)

2023 Reading (4,8,12) Reading (4,8) 2021 Reading 2023 Mathematics (4,8,12) Mathematics (4,8) 2021 Mathematics 2023 Science (4,8,12) Writing (4,8,12) 2021 Wrting

2025 Reading (4,8,12) 2024 Arts (8) U.S. History (8,12) 2025 Mathematics (4,8,12) 2024 Foreign Language (12) Civics (8,12) 2025 Writing (4,8,12) Geography (8,12)

Economics (12)

TEL (8,12)

2026 U.S. History (8,12) Reading (4,8,12) 2022 U. S. History 2026 Civics (8,12) Mathematics (4,8,12) 2022 Civics 2026 Geography (8,12) Science (4,8,12) 2022 Geography 2026 Economics (12) High School Transcript Study 2022 TEL 2026 TEL (8,12) 2023 Reading (4,8,12)

2023 Mathematics (4,8, 12)

2027 Science (4, 8, 12) 2025 Reading (4,8) Arts (8) 2023 Science (4,8) 2027 Reading (4,8,12) 2025 Mathematics (4,8) Foreign Language (12) 2027 Math (4,8,12) 2025 Wrting (4,8,12) Long-Term Trend

2026 U.S. History (8,12) Reading (4,8) 2026 Civics (8,12) Math (4,8)

2025 2026 Geography (8,12) Writing (4,8,12) 2026 Economics (12)

2026 TEL (8,12)

2026 US History (8,12) Civics (8,12) Geography (8,12) Economics (12)

TEL (8,12)

State and District Level Operational Assessments

Activities shown above generally begin in the fall of the first year through September/October of the second year listed. Assessments of grade 4, 8 and 12 unless otherwise noted. Long-term trend (LTT) assessments are ages 9, 13, and 17. Special studies are not shown.

During the contract period of performance, the Governing Board may revise the schedule, which would affect the activities above.

About the Performance Work Statement

The next part of the PWS is organized into eleven chapters. The first ten correspond to each of the Alliance contracts. While each contract will have specific responsibilities, there is a set of more general activities and associated tasks that are common to all the contracts described once in Chapter 11. Each general activity and the associated tasks are relevant to each contract and shall be included in each contract offer.

PAGE 23 OF 452 91990018R00180002

1 Alliance Planning and Coordination (PC)

Overview

The NAEP program is a complex suite of assessments carried out in an extensive, interconnected, and ongoing operation. The NAEP program includes operational assessments, pilot assessments and studies, other research intended to inform NCES about the efficacy of a given process, and future planning, within a given year. An alliance of key contractors, the NAEP Alliance (Alliance), supports NCES in carrying out the major processes of the NAEP program.

The Planning and Coordination (PC) contractor supports NCES in conducting strategic and operational planning for NAEP, coordinating the core activities of the NAEP program across multiple contractors and NCES, and coordinating research and development activities aimed at improving the assessments.

As the contractor responsible for planning and for effecting the smooth integration of the various components of the NAEP program across contractors both within and outside the Alliance, the PC contractor shall support NCES in achieving the overarching strategic NAEP goals to:

Strengthen the utility of NAEP, Adopt new approaches to assessment, Lead in assessment innovation, and Improve operational processes.

Scope

The scope of this contract is reflected in the following tasks:

Task 1.1 Conduct strategic planning

Task 1.2 Provide operational support/planning

Task 1.3 Coordinate assessment tasks across multiple contractors and NCES

Task 1.4 Coordinate NAEP Research and Development

1.1 Conduct strategic planning

Throughout this procurement cycle, NAEP must maintain its high-quality assessments while making strategic decisions that will influence the program for decades. The PC contractor shall support NCES in ensuring alignment between program objectives and

PAGE 24 OF 452 91990018R00180002

general activities, and shall possess a clear understanding of the technological, policy and legislative contexts of NAEP.

1.1.1 Support strategic planning

The contractor shall support NCES in developing the NAEP Strategic Plan to cover a five-year period from the start of the contract year, and shall also help revise the plan, as necessary. To help develop the plan, the contractor shall develop procedures for scanning NAEP's environment for changes in the large-scale assessment industry at the state, district and international levels. Such procedures shall include identifying opportunities and threats that support or detract from efforts to carry out assessments as planned, to advance NAEP as the leader in large scale assessment, and to promote currency in assessment practices. The contractor shall also help identify and interpret trends and changes in the education environment, assist NCES in implementing procedures for forecasting changes and trends, and identify strategies to effectively manage the impact of such changes on NAEP stakeholders and contractors, through outreach and communication aimed at facilitating an understanding and acceptance of change. These actions should be the basis for creating a new plan, and for providing recommendations to update the plan.

They shall focus on high-level issues, such as changing state and international policies, rapidly emerging technological changes, educational inequity, declining participation in assessments at some grade levels or with some subjects, and the challenge of interpreting performance trends.

1.1.2 Facilitate development of NAEP Executive Steering Committee

The contractor shall establish or maintain the NAEP Program Executive Steering Committee (Committee), comprising Corporate Officers from all Alliance contractors, the NCES NAEP Associate Commissioner, and Assessment Division (AD) managers, to report out on the status of strategic planning and implementation, and present the rationale for any proposed new directions and actions that are aligned with NAEP’s mission and vision. The contractor shall schedule and convene the Committee meetings.

As needed, the Committee shall meet up to four times annually to focus on high-level issues, and to develop broad recommendations for addressing those issues.

The contractor shall also convene additional ad-hoc expert panel to meet with the Executive Steering Committee, as the need arises, up to four times per year. In past years NAEP assembled ad-hoc panels of experts from various relevant disciplines and industries beyond the large-scale assessment community to provide input for NAEP’s strategic planning. To ensure a diversity of perspectives on as broad a range of pertinent issues as possible, the composition of any panels of experts convened for such purposes shall vary according to need.

PAGE 25 OF 452 91990018R00180002

The contractor shall document all expert panel deliberations and resultant recommendations, and deliver them to the Contract Officer Representative (COR), and the Executive Steering Committee. The contractor shall monitor progress on action items identified during NAEP Alliance meetings, and make recommendations to NCES on approaches to operationalizing the recommendations of the panel. NAEP Alliance meetings, consisting of NAEP, Alliance and non-Alliance staff, will be described in more detail in section 1.2.2 (Coordinate planning and review meetings among NAEP contractors and NCES). The contractor shall report the status of action items and implementation of recommendations as part of the weekly communications with the

COR.

1.1.3 Convene NAEP Innovations Symposium

The contractor shall prepare annually for a NAEP Innovations Symposium with expert consultants that focus on potential future initiatives for NAEP. This meeting would likely include the AD management team, other NCES staff, the Executive Steering Committee, and other NCES contractor staff as directed by the COR. The contractor shall support NAEP in identifying a topic for the symposium, draft an agenda, and in bringing together the ad-hoc panels of experts.

1.1.4 Facilitate Digital Transitions Advisory Council meetings

The contractor shall convene the Digital Transitions Advisory Council (DTAC)…