Attachment K - eNAEP overview.pdf

NAEP Platform Development (NPD)

Next-Generation Platform

System Design Document

Chapter 1 – Introduction

Chapter 2 – System Overview

Submitted: November 14, 2023

Attachment K

November 2023 Page 2

Contents

1. Introduction

1.1 Identification and Purpose

1.2 Audience and Stakeholders

2. System Overview

2.1 The Purpose of the Next-gen Platform

2.2 System Design Goals

2.3 Next-gen DBA Platform Conceptual Design

2.3.1 Phase 1: Assessment assembly – pre-administration

2.3.2 Phase 2: Platform distribution and deployment

2.3.3 Phase 3: Field administration

2.3.4 Phase 4: Post-administration

November 2023 Page 3

List of Figures Figure 1-1. Key NAEP users and how they interact with the Next-gen Platform

Figure 2-1. The Next-gen Platform high-level system overview Figure 2-2. The Next-gen Platform core goals and capabilities Figure 2-3. Alliance coordination for the Next-gen Platform Figure 2-4. The four stages of assessment assembly for the Next-gen Platform during pre-administration Figure 2-5. The Next-gen Platform distribution and deployment process Figure 2-6. Registration system conceptual data entities for NAEP field administration Figure 2-7. Online assessment delivery mode for the Next-gen Platform Figure 2-8. NPN assessment delivery mode for the Next-gen Platform Figure 2-9. Next-gen Platform NAEP Operational Dashboard Figure 2-10. Post-administration data flow

November 2023 Page 4

1. Introduction

1.1 Identification and Purpose

This system design document (SDD) provides a high-level description of the architecture and system design of the Next-generation (Next-gen) Platform for the National Assessment of Educational Progress (NAEP) program under the National Center for Education Statistics (NCES) at the Department of Education. The Next-gen Platform was developed under the NAEP Platform Development (NPD) contract during the contract period 2019-2024.

For the 2024 Operational Administration, two modes of assessment delivery are supported versus the four modes that were available for the 2023 Field Test to simplify the setup process for field staff: (1) Online Mode, which is synonymous with what will be operationally referred to in 2024 as School Wi-Fi Mode, and (2) NAEP Private Network (NPN) Mode, which serves as a backup mode to online assessment delivery by leveraging a NAEP Classroom Server (NCS).

Online Mode will require School Wi-Fi internet connectivity. In contrast, NPN Mode will require internet connectivity only after the assessment session(s) are completed. The NPN Mode will continue to leverage the NAEP Classroom Server (NCS) device and application that was utilized to support the NCS Mode in the 2023 Field Test.

1.2 Audience and Stakeholders

The intended audience for this document is anyone who wants to understand the system architecture and design of the Next-gen NAEP Platform, including but not limited to the following groups:

• NCES NAEP contract management staff – Evaluate that the system architecture and design meet the requirements for the NAEP program.

• Department of Education IT managers – Be informed of the system architecture of the NAEP Platform and core design goals.

• NPD development and quality control (QC) engineers – Understand the system architecture and follow the design to build and test the Next-gen Platform.

• NPD security and operation engineers – Understand the system’s internal workings to administer it effectively and securely.

• Future platform maintenance personnel and contractors – Understand the internal workings of the system to perform on the Next-gen Platform.

• The NAEP Alliance partners – Understand the architectural approach of how the Platform helps NAEP achieve its goals.

The SDD also identifies several key groups of platform users with important interests (concerns in IEEE standards) in the design and implementation of the NAEP Platform. These key users are outlined in figure 1-1. The architecture and design of the Next-gen Platform must meet their specific needs.

Figure 1-1. Key NAEP users and how they interact with the Next-gen Platform

November 2023 Page 5

The stakeholders of the Next-gen Platform and their concerns include the following:

1. Students sampled to participate in the NAEP assessment are the most important users of the NAEP Platform. They may take the NAEP assessment on a NAEP-provided Windows or Chromebook device or use a familiar Windows or Chromebook device from their school (school-based equipment [SBE]). Some students may need accommodation(s) in the assessment platform (such as extended time or bilingual support) or accommodations offered by other means (e.g., a braille version of the test). Regardless of how they take the NAEP assessment, the Next-gen Platform aims to provide an accessible and consistent student experience that supports a fair and reliable measurement of what students know and can do.

2. NCES and Alliance reviewers are responsible for conducting content reviews of individual items, collections of items (e.g., a 30-minute block of items), and completing assessment forms on the Platform. NCES certifies the assessment system (including item content and the Platform that supports the content) through a user acceptance testing (UAT) process. All of these activities require a flexible, high-fidelity method to review the assessment content and a system user interface and user experience (UI/UX) that reflects exactly what students will see and experience when they take the assessment.

3. The National Assessment Governing Board and other reviewers such as subject matter experts and content development standing committees often need to review specific items. Unlike NCES users who are authorized to review any item in the pool, these users will have access to only prespecified items for their review activity. Therefore, the technical and security requirements for supporting these users are distinct.

4. NAEP field staff are hired by Sampling and Data Collection (SDC) to administer the NAEP assessment using a registration system in schools on administrator devices. Their responsibilities include updating assessment settings and accommodations on assessment day, setting up

November 2023 Page 6 devices and the network during the assessment day, answering student questions and reporting field issues, and uploading data after the assessment administration in NAEP Private Network (NPN) Mode. Reducing field staff burden before, during, and after assessment administration is an important goal for the Next-gen Platform.

5. School administrators and teachers participating in the Teacher and School survey questionnaires will be directed to the NAEPq website, an online survey application of the NAEP Platform. Unlike students, the teachers and school administrative staff can complete the NAEPq survey using a standard web browser on any device that meets the minimum requirements (e.g., a minimum screen size of 7 inches).

6. NAEP reports and released NAEP items on the Nation’s Report Card website (nationsreportcard.gov) to the public. Some of the sample items require a special version of the NAEP Platform (the content on the web [COTW] service) to preserve their interactivity as live digital content. Some of the digitally released items also need Platform support in public-facing services such as Take NAEP Now. Accuracy, accessibility, performance, and device- and browser-agnostic design are Platform requirements for serving these users.

In addition, certain authorized users of the Next-gen Platform exchange data with NAEP via application programming interfaces (APIs) to view data and perform analytics or to develop and operate the Next-gen Platform infrastructure. They include the following:

• NPD develops, operates, maintains, and secures the Next-gen Platform and the cloud-based infrastructure on which it is developed and operated.

• NCES and the Alliance use the NAEP Operational Dashboard during the administration window to monitor the progress of the assessment and make important operational decisions based on the data. The Dashboard, which is part of the Platform, is intended to alert stakeholders and decision-makers about potential technical or field issues (e.g., hardware failures or software bugs) and enable rapid data forensic analyses to support decisions. In addition, authorized users from NCES and the Alliance will have access to the data analytic engine (Databricks 0F

1) supporting the Dashboard, which they can use to run ad hoc data analyses or export Platform data for research and investigation.

• Item Development (ID) develops and authors the NAEP assessment content using the item authoring system and/or other tools. Item developers integrate item content and associated metadata into the Platform to help ensure that content is rendered and functioning in the Platform as intended. ID is also responsible for supporting item reviews by NCES, the National Assessment Governing Board (Governing Board), and external expert panels, which require Platform support for viewing select items.

• SDC teams and their registration system interact with the Next-gen Platform closely. SDC is responsible for installing the Next-gen Platform software on NAEP-provided devices. SDC operates the registration system, in which the schools and login information for student assessments in the NAEP sample must be synchronized with the Next-gen Platform before the field administration. Improvements to the Next-gen Platform and its interaction with the Next-gen registration systems will bring much-desired efficiencies to the overall NAEP program’s

1 Databricks is a tool used to process, store, clean, share, analyze, model, and transform assessment data in the AWS NPD Cloud Operational Data Lake.

http://nationsreportcard.gov/

November 2023 Page 7 activities. The changes to the Next-gen Platform must be carefully coordinated between NPD and SDC to maintain the assessment's integrity.

• MDPS prepares assessment materials (e.g., creating student login resources) and receives the student responses that the Next-gen Platform exports after the assessment window. The MDPS systems support the human scoring activities of student responses. The human scored data and responses are sent from MDPS to DAR for machine scoring and reporting.

• DAR produces the mapping that assigns students to test forms (aka the bookmap) that must be integrated into the Platform before the administration. During the field administration, selected data engineers and data scientists from DAR also carry out Platform validation tasks and data forensics. Post administration, DAR conducts machine scoring of non-human-scored responses, process data analyses, and psychometric modeling using various systems that receive data from the Platform.

The Next-gen Platform supports secure access to the assessment delivery system either using browsers and Okta authentication or via a secured NAEP app on either NAEP-provided or school-provided devices.

Students and reviewers from NCES, the Alliance, or Committee and Governing Board members utilize these secure access mechanisms to interact with the Assessment Delivery Platform (ADP) to support the administrative actions in the registration system by the field staff.

2. System Overview

2.1 The Purpose of the Next-gen Platform

The high-level purpose and goals of the Next-gen Platform include the following:

• Support online assessment delivery on NAEP-provided devices and on school-based equipment

(SBE).

• Support a backup mode of assessment delivery for schools without sufficient internet bandwidth for online assessments.

• Utilize app-based approach with auto-update features.

• Device-agnostic approach to support a variety of operating systems including Windows, ChromeOS, and macOS (planned for future support).

• Use a single codebase to provide a consistent and uniform UI/UX across a variety of content types, including discrete items, scenario-based tasks (SBT), reading discrete items (RDI), interactive item components (IIC), and interactive computer tasks (ICT).

• Modernize Alliance interaction touchpoints with real-time data transfer where possible.

• Generate standardized, consistent, and well-documented response data for all item types in the assessment platform.

• Generate standardized, consistent, modernized, and well documented process data from the assessment platform.

• Develop and maintain a modern and reliable cloud-native Operational Data Lake that serves as a single-source of truth for current and historical assessment data.

November 2023 Page 8

• Develop and maintain a modern cloud-native Operational Dashboard to meet NAEP’s operational analytic reporting needs to support active monitoring and troubleshooting.

• Comply with FISMA, FedRAMP, and NIST standards required for federal compliance. Deploy under the NAEP System ATO by working with the WTDOM Alliance partner.

The section below outlines how each of the above goals are achieved in the Next-gen Platform and the advantages of such an approach.

Online and NPN (Backup) Assessment Modes

The Next-gen Platform has modernized NAEP assessment delivery by supporting online assessment.

Using cloud-native and serverless secure components, the Next-gen Platform can seamlessly scale to meet the needs of hundreds of thousands of students who take NAEP assessments across the U.S.

mainland and Puerto Rico. Using progressive web app (PWA) and caching principles, these online assessments are supported using school networks of varying bandwidth and throughput quality. The Next-gen online ADP is designed and built to be resilient to handle temporary disruptions and unreliable Internet connections. This Platform also supports a backup delivery mode to support schools that do not have sufficient internet bandwidth to deliver the complex eNAEP assessment.

Application and Updates

The software components utilize an app-based approach for Windows (and, in the future, macOS) devices and a Chrome kiosk app-based approach for Chromebook devices. A robust, secure, and self-contained electron-based lockdown Google Chrome browser is used for ADP apps in Windows (and, in the future, macOS) devices. This app-based approach eliminates dependencies on the host device. These apps support automatic software update features to ensure the latest software and content are used during assessment delivery. This standardized, device-agnostic, modular, and component-based app approach for the Next-gen Platform’s deployment and distribution process streamlines the milestone build releases, simplifies content and software patching, and supports automated software updates. The Next-gen Platform has been built to operate on NAEP-provided and school-based equipment (SBE) and moves away from using pre-imaged devices for the assessment.

Device Agnostic

The Next-gen Platform is engineered to be device agnostic, with support for Windows, Chromebook, and in the future, macOS, each with different display aspect ratios1F

2 and screen resolutions2F

3. To be device agnostic and still provide a comparable student experience across device types continues to be an evolving and difficult challenge for the Next-gen Platform’s UI components given the importance of maintaining trend.

Content Support

The Next-gen Platform utilizes a single codebase to support a variety of content including discrete items, reading discrete items, interactive computer tasks (ICTs), interactive item components (IICs), and

2 Aspect ratio of a display device is the proportional relationship between the physical width and the height. It is expressed as two numbers separated by a colon (x:y), where x corresponds to the width and y to the height. Common aspect ratios for displays, past and present, include 5:4, 4:3, 16:10, and 16:9.

3 The display resolution of a computer monitor or display device is the number of distinct pixels in each dimension that can be displayed. It is usually quoted as width × height, with the units in pixels: for example, 1366 × 768 means the width is 1366 pixels and the height is 768 pixels.

November 2023 Page 9 scenario-based tasks (SBTs) using the Benchmark Component Library and Next-gen Platform frameworks. Toward this goal, previously authored and rendered opaque-box/black-box items are iteratively re-engineered to be natively supported in the Platform. This approach ensures that the response, process, and diagnostic data captured during an assessment remains consistent across the diverse content of NAEP while meeting the consistency of appearance and behavior expectations across these content types. Using the same codebase also provides agility, cost, and quality efficiencies for the Next-gen Platform and NAEP overall. Using the power of a custom-developed automation framework, the Platform continues to be able to develop new features using agile DevSecOps while ensuring no/reduced impact on existing functionalities.

Real-time Data Transfer

The Next-gen Platform modernizes application interfaces with Alliance partner systems with support for real-time data transfer between the systems where feasible. For example, the eNAEP software applications, including the NAEP Classroom Server (NCS) application on the NCS server (in the backup mode) and the ADP running on the student devices interact with SDC’s registration system in real time during a field administration. NAEP involves multiple Alliance partners whose application interfaces are modernized with real-time data transfer capabilities in the Next-gen Platform.

Response Data

The Next-gen Platform provides consistent response data from field administration, by natively supporting a variety of item types, including, Discrete Items, Reading Discrete Items (RDI), Interactive Computer Tasks (ICT), and Scenario Based Tasks (SBT).

Process Data

The Next-gen Platform has been engineered to produce student experience data from a field administration in the form of process/event data in a standardized, modernized, and well-documented format.

Data Lake

Using Databricks and modern, cloud-native components, the Next-gen Platform implements an Operational Data Lake, which forms the single source of truth for assessment data from current and historical years.

Dashboard

The Next-gen Platform also implements a cloud-native business intelligence NAEP Operational Dashboard that provides insight into the operational metrics required for monitoring NAEP assessment administrations.

Data Security

The NAEP Platform complies with the FISMA and associated National Institute of Standards and Technology (NIST) standards required for federal compliance. The Next-gen NAEP Platform production system will be deployed to and operated under the "NAEP System" ATO by working with the WTDOM NAEP Alliance partner.

The NAEP ecosystem requires many tools and applications that serve NCES and Alliance partners with their content review, authoring, survey questionnaires, data import, diagnostic, audit, logging, reporting, and dashboard needs. NAEP also exposes various experience tools to the public through released

November 2023 Page 10 tutorials, content, and assessment data. The Platform uses a single codebase, standardized processes with style guides, development standards, and integrated DevSecOps processes. The Platform adheres to recommended best practices for infrastructure and security.

These core requirements of the NAEP ecosystem are served using the right mix of an assessment delivery platform, assessment delivery client (UI), NCS application, back-end API services, back-end assessment storage, Operational Data Lake, and Alliance data exchange service software components.

Figure 2-1. The Next-gen Platform high-level system overview

The following is a brief description of the (mostly software) components depicted in the high-level system overview diagram (figure 2-1).

• Assessment Delivery Client (ADC) – Student-facing, front-end, single-page application (SPA).

This SPA is a progressive web app (PWA) built using the Benchmark Component Library and renders content produced for RDI, SBT, ICTs, and Discrete items. The custom-developed Papago transformation library transforms XML produced by the authoring platform to JSON required by Benchmark and the Platform for rendering item components. Though depicted in figure 2-1 as a single application, this component includes a smaller application called the Student Check-In (SCI) PWA SPA, whose responsibility is to offer a login screen to the students for entering their participantId information before launching the ADC itself.

November 2023 Page 11

• Assessment Delivery Platform App (ADP App) – Packaged as a PWA App to support device-agnostic (DA) Online and NPN Modes of assessment delivery using a lockdown secure browser for Windows and Kiosk mode in Chromebooks.

• Assessment Back-end Service APIs – Comprises various service APIs to support the ADC, the Content transformation library (aka Papago), the ADP, the registration system during assessment delivery, the NAEP Classroom Server (NCS) during the backup delivery mode (NAEP Private Network Mode), and the Operational Data Lake.

• Assessment Back-end Storage – The assessment data is persisted to the AWS NPD Cloud (NPD Cloud) Assessment Delivery Storage for online assessment delivery. For NPN Mode, the assessment data is stored on the NCS device (see NCS description below) during the assessment.

Assessment data from the NCS device is transferred to the NPD Cloud after the field administration once the NCS device is connected to the internet.

• NAEP Classroom Server (NCS) Application – This application is used to support the NPN Mode.

It contains prepackaged application components, including the assessment content, delivery platform, and back-end service APIs with temporary storage of generated data (e.g., process, response, logs, metrics, diagnostics, etc.). These application software components contain enough information to collect assessment data from the student devices. The NCS device acts as temporary storage during the assessment that collects all of the assessment-generated data, including response, process, and diagnostic data from the student devices until the NCS device is connected to the internet and data is transferred to the AWS NPD Cloud. The SDC-provided registration system interacts with the NCS application for monitoring and administrative actions such as Release from the waiting room, Pause, and Advance assessment during the field administration. The Next-gen Platform can thus support Online and NPN Modes of assessment delivery; the latter of which is supported without internet connectivity during the assessment.

• Registration System – SDC-developed application used by the field staff (Administrative Coordinators [AC] and Administrative Assistants [AA]) to administer, monitor, and ensure smooth NAEP assessment delivery in the school network. This application component interacts with the assessment back-end services on the AWS NPD Cloud for Online Mode and with the NCS application for NPN Mode.

• Assessment Delivery Storage – Pre-administration assessment data (such as item content and bookmap data) and assessment delivery transactional data (such as student responses and observable data) are stored in this component by the assessment back-end services before getting ingested into the Operational Data Lake.

• Operational Data Lake – A central repository to store pre-administration, administration, and post-administration assessment data from the current and historic administration years. The core capability of the Operational Data Lake is to be the definitive source of truth platform to meet stakeholder, Alliance contractors, and NCES needs such as advanced analytics, ad hoc reporting, operational support, and dashboard monitoring.

• Data Exchange Services – These services support data export from the Next-gen Platform for consumption by Alliance contractors. For response data, the industry-standard Question and Test Interoperability (QTI) format is used for both human and machine scoring. MathML responses are exported as images to aid human scoring. The Next-gen Platform defines a NAEP-specific Event Specification format for the process (aka observable) data to aid student experience analytics. The Next-gen Platform can be extended to export any assessment data for

November 2023 Page 12 advanced analytics by external systems. NPD will work with NCES to prioritize activities that support the educational research community and export to any other identified analytics schema.

Listed below are the key tools/applications that are a part of the Next-gen Platform and support the overall NAEP Assessment Delivery.

Tools

• Content and review tools – The Next-gen Platform provides content-review tools using supported web browsers and devices. These tools include the Random Access Viewer to aid the review of active items, the Random Access Launcher (to aid the searching and launching of active items and blocks), and the item authoring system (to allow the content authors to preview the items from within the item authoring system). Both Random Access Viewer and the Random Access Launcher are tools used by ID in constructing Electronic Review Packages (ERPs) to facilitate content reviews with NCES, standing committees, the Governing Board, and other external reviewers.

• NAEP Experience tools – The Next-gen Platform supports applications such as Take NAEP Now (to aid in providing the experience of taking a subset of the NAEP assessment comprised of released digital content and having responses scored on the fly), Content on the Web (to aid in reviewing and experiencing released digital content), and Tutorial On The Web (to aid in experiencing tutorials for the NAEP assessments) for the general public. These applications can be accessed via a standard web browser on supported devices and will not require any installation of NAEP-specific apps. As outlined in section 3.1.3, these applications do not require the level of content security, data capture and session state management, and platform processes as compared to the application required for the official NAEP assessment.

• NAEP teacher and school survey questionnaire (NAEPq) tools– Teacher and school survey questionnaires are integrated into the NAEPq Platform through coordination and collaboration with ID. NAEPq is accessible by participants via a custom URL generated by the Platform based on data shared by SDC and the sample drawn for a given NAEP administration. The NAEPq application may be accessed from various devices and browsers, as outlined in section 3.1.2. For the 2024 Administration, the Current-generation (Current-gen) NAEPq Platform will be utilized as NPD begins to transition NAEPq to the Next-gen Platform for subsequent administrations.

• Content onboarding tools – The Next-gen Platform provides tools that support assessment content integration. This assessment content includes the bookmaps (that aid in communicating the Platform configuration requirements for assessment content), item content (to aid in the batch importing of item content, including metadata and item assets [e.g., media files], into the Platform), and other content found in the Platform, such as Student Help, Adjust Device, and dialog screens, (to aid in conveying instructional and support information to students before and during the assessment). These tools leverage parts of the assessment service APIs and their storage to support critical NAEP onboarding functionalities.

• Operational Dashboard tool – The Next-gen NAEP Operational Dashboard is a vital tool that provides NCES and the Alliance with near-real-time views of the collected data before, during, and after a field administration. The Dashboard offers enhanced custom reporting, analytics, and troubleshooting dashboard features using the QuickSight AWS cloud-native tool. Diagnostic data is directly consumed from the assessment data storage for dashboarding needs. The Operational Data Lake serves as the major ingestion source for the NAEP Operational Dashboard for pre-

November 2023 Page 13 administration assessment metadata, the assessment data during the administration, and the post-administration data processed by the Data Lake. Some of the pre-administration information such as the schools, students, and sessions data originates from the SDC's registration system.

• Diagnostic, audit, and logging tools – The industry-recommended Splunk tool is used to proactively monitor, search, alert, report, visualize, and analyze the application log and diagnostics data. Real-time data analytics, monitoring, auditing, and troubleshooting are achieved using AWS cloud-native services such as XRAY3F

4, CloudWatch4F

5, and CloudTrail.5F

To aid the design and development of these applications, components, and tools in the NAEP ecosystem, the Next-gen Platform uses the following high-level processes, standards, and guidelines.

• Development codebase and processes – Industry-recommended tools such as Visual Studio Code (integrated development environment for source code development), GitLab Source Code Management (SCM), MonoRepos (standardized repository design and organization methodology to seamlessly support multiple launch modes and applications from a single codebase) and TypeScript (strongly typed programming language that enforces sound programming practices) are used in the entire stack, including the assessment delivery front end, the back-end services, and the Platform app. The ADP, the student check-in screen, and the assessment delivery clients are engineered to use progressive web app (PWA) principles with prefetching of content, assets, data, and software to ensure seamless operation of the assessment delivery under various network conditions. The front-end assessment delivery client uses functional programming paradigms, ReactJS (for modularized development using components) and RxJS (for reactive programming with improved performance, modularity, and better debuggable call stacks). The back-end APIs use NestJS (for efficient, reliable, and scalable server-side APIs with TypeScript) and Docker (for fast, easy, and portable server-side interfaces).

The APIs use serverless AWS cloud-native technologies like Lambda, FarGate, and Elastic Container Service to ensure on-demand resource availability, reduced vulnerabilities, no maintenance, no patching, reduced cost, and automatically scalable server-side infrastructure wherever feasible.

• Style Guide and standards – Benchmark components, style guide, coding standards, linting standards, code-review standards, code-approval standards, role-based restrictions on who can promote to what environment and validation using various quality controls (e.g., unit test, manual test, automation smoke test, automation regression test, performance tests) help ensure that the Next-gen Platform can support all the apps and launch modes from a single codebase.

• DevOps and automated CI/CD pipelines/processes – The industry-recommended tools/practices such as GitLab CI/CD, AWS Cloud formation (to ease the spin up and tear down of AWS Cloud infrastructure and environments), JFrog Artifactory (for storing the build artifacts), 4 AWS XRAY provides a complete view of requests as they travel through the NPD Cloud, and it filters visual data across payloads, functions, traces, services, APIs, and more with no-code and low-code motions.

5 AWS CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. CloudWatch metrics help NPD, NCES, and other Alliance users make operational decisions.

6 AWS CloudTrail is a service that enables operational and risk auditing, governance, and compliance of the Next-gen AWS NPD Cloud Platform.

November 2023 Page 14 integration with automation pipelines, and version management are used during the development life cycle of the Next-gen Platform.

• Infrastructure best practices – Serverless best practices, industry-recommended, well-architected framework guidelines, multi-account AWS Cloud strategies, separation of production and development environments, and infrastructure as code (IaC) best practices are leveraged during the entire development life cycle of the Next-gen Platform.

• Security best practices – FISMA and FedRAMP-compliant AWS Cloud services are used in the Next-gen Platform. Sections 7.3 and 7.4 describe the security controls in place for the Platform.

• Quality control tools and processes – A combination of unit tests, manual functional tests, accessibility/navigability tests, and automation tests, ensures that new functionality and bug fixes do not break existing functionality across the Next-gen Platform’s ecosystem.

2.2 System Design Goals

NAEP needs to support a variety of assessment delivery modes, content types, tools, and operating environments while serving different school networks in the U.S. mainland and Puerto Rico locations. To support these various needs, the core Next-gen Platform is built using foundational functional components that are online first and cloud-native, with quality control processes and integrated DevSecOps principles, and uses an Operational Data Lake. Figure 2-2 visually represents the Next-gen Platform's foundational goals and how they serve the various needs of NAEP.

Figure 2-2. The Next-gen Platform core goals and capabilities

November 2023 Page 15

As depicted in figure 2-2, the Next-gen Platform supports the following use cases of NAEP:

• Different Assessment Delivery Modes – The Online and NPN Modes of assessment delivery are supported with the AWS Cloud back end. A Local private NAEP network is set up using the NAEP-provided router for NPN Mode. The backup NPN Mode also supports data transfer after successful field administration.

• Multiple Content types, subjects, and grades – The Next-gen Platform supports various ID-authored (via the item authoring system), ID-IT-authored, and ID-plus-vendor-authored content.

The rendered content supported includes discrete items, reading discrete items (RDIs), scenario-based tasks (SBTs), and interactive item components (IICs). A variety of subjects spanning mathematics, mathematics for Puerto Rico (MPR), reading, science, and social studies, to name a few, are to be supported by a single platform. Students from grades 4, 8, and 12 are required to be supported by the Platform.

• Variety of tools – Content authoring/review tools (e.g., Item Viewer, Random Access Viewer, Random Access Launcher), NAEP Experience tools (e.g., Take NAEP Now, COTW, TOTW), teacher and school survey questionnaire (NAEPq) tools, and content onboarding tools (e.g., for item import, content import, and bookmap management) are supported by the Next-gen Platform.

• A variety of operating environments – The Next-gen Platform assessment delivery system and its accompanying tools are supported on a variety of devices (e.g., Chromebooks, Surface Pros), and operating systems (e.g., ChromeOS, Windows) and are engineered to operate under a

November 2023 Page 16 variety of field network conditions. The Platform app approach can also support macOS as per future NAEP needs.

• Several types of networks – The assessment delivery system can operate during the administration either online with school-provided Wi-Fi or in backup mode with NAEP-provided routers.

• Different assessment delivery locations – The students using the Next-gen assessment delivery system are in the U.S. mainland or Puerto Rico.

To support these use cases, the Next-gen Platform is built on the following core principles:

• Functional components – The building blocks selected for the Next-gen Platform are as follows:

o Modular – Each component in the Next-gen Platform is engineered to be used as a building block.

o Reusable – Each such modular component may be reused to serve diverse needs.

o Extensible – The components may be customized by extending their core functionality.

o Configurable – The components may be configured to customize their operation based on where it is used.

o Integrable – The integrated components and their sub-parts are engineered to interact with other components or external systems.

o Internationalized – The components support U.S. English (en-US), U.S. Spanish (es-US), and Spanish Puerto Rico (es-PR) languages.

o Localized – The component behavior is customized for native behavior for the supported languages.

• Cloud native – Engineered to be cloud native for online-first delivery mode to satisfy the following intents:

o Security – To meet or strive to exceed the FISMA Moderate compliance level and the Department of Education regulations. The cloud-native Next-gen Platform uses only FedRAMP-certified cloud services in compliance with federal IT regulations.

o Responsiveness – To respond in a timely manner to student/user actions.

o Elasticity – To shrink and to grow infrastructure based on user load where possible.

o Scalability – To provide predictive and reactive scaling mechanisms where possible.

o Availability – To be available with sufficient capacity, e.g., during assessment delivery.

o Reliability – To be able to operate with reduced failure conditions.

o Serverless – To operate in a cloud-native development model, which provides on-demand, as-needed capacity without preprovisioned servers that require maintenance.

o Easily deployable – To easily deploy the applications and tools to multiple environments and deployment modes.

o Resilient – To stay as responsive as possible even under failure conditions.

November 2023 Page 17

• Strong quality control methodologies using a combination of the following:

o Unit tests – Using testing code directly into the development repositories, to ensure that previously working functionality is not affected by newly introduced code.

o Automated tests – Using a custom automation framework to ensure end-to-end functionality across delivery and launch modes, reducing manual QC burden.

o Functional tests – Using a combination of unit, automated, and manual QC.

o Visual Validation tests – Integrating visual comparison tools across a variety of environments and years to identify issues not easily comprehendible by the naked eye.

o Performance tests – Using tools to ensure optimal functionality under a variety of load conditions.

o Manual validations – Evaluating new functionality which is not yet/cannot be automated.

• Integrated DevSecOps methodologies to seamlessly work in the following ways:

o On multiple accounts – Best AWS-recommended practices of account and environment separation.

o On multiple environments.

o Support for multiple assessment years in parallel – To support access to previously assessed, currently being developed/assessed tools/applications/content, and future development/applications/content in parallel.

o Support for multiple versions within the same assessment year – To support differing QC, authoring, development, and review needs across various environments for a given assessment year.

o Integrated Infrastructure as Code (IaC) – IaC software and configuration is integrated and maintained in the same source repositories as the Next-gen Platform itself.

o A uniform standard approach for applications using MonoRepositories (version-controlled code repositories) − The codebase is built with standardized branching/versioning methodologies to support multiple environments, years, and accounts.

o Integrated quality control checks in the development pipeline – Automated linting, unit testing, performance testing, automation testing, and security scanning tools are integrated into the DevSecOps pipelines.

• With various datasets, including the following:

o Current and historic data o Pre-administration bookmap, item, and supporting content o School, session, and student data o Events/process data o Response data o Diagnostic, audit, logging, metrics, tracing and troubleshooting data

November 2023 Page 18

• Operational Data Lake to support the following:

o Data analytics needs of NCES and Alliance o Reporting needs of NCES and Alliance o Operational Dashboard monitoring o Processing responses in preparation for MDPS’s human scoring, DAR’s machine scoring, and DAR’s overall analysis, troubleshooting, analytics, and reporting needs o Data management needs of the Platform o Assessment data required to support NCES and Alliance needs o Data governance needs of the Platform to control access by NCES and the Alliance to the Operational Data Lake.

2.3 Next-gen DBA Platform Conceptual Design

The Next-gen Platform is used throughout the assessment development and administration processes, and its functionalities and user interactions differ depending on the stage of the NAEP assessment process. Figure 2-3 illustrates the collaboration between the NAEP Alliance partners for end-to-end assessment delivery.

• ID (includes ID-IT and vendors) – Develops the assessment content to meet requirements and prepares scoring materials.

• DAR – Designs assessment specifications, conducts analysis, performs algorithmic/machine-scoring activities, and publishes reports.

• MDPS – Coordinates and manages operations for human scoring activities.

• SDC – Administers NAEP assessments, manages NAEP devices, manages the Help Desk, designs and selects student samples, collects data, and generates sampling weights.

• NPD – Designs and develops assessment delivery platforms and systems for data collection, transfer, and monitoring of assessment administration.

Figure 2-3. Alliance coordination for the Next-gen Platform

November 2023 Page 19

The conceptual design for the eNAEP assessment delivery application can be divided into four phases illustrating how the Platform supports the NAEP operational and business aspects.

• Phase 1: Assessment Assembly (Pre-administration)

• Phase 2: Platform Distribution and Deployment

• Phase 3: Field Administration

• Phase 4: Post-admin

The following subsections provide detailed summaries of how the Next-gen Platform will support each of these identified areas of the NAEP administration.

2.3.1 Phase 1: Assessment assembly – pre-administration

NAEP content is developed by the ID contractor (in association with ID-IT and vendors) in a separate item-authoring system and is then assembled and integrated into the Next-gen Platform. This action initiates the assessment assembly and review process before the assessment administration. Figure 2-4 depicts the Next-gen Platform support of the four stages of assessment assembly that occur before an administration.

Figure 2-4. The four stages of assessment assembly for the Next-gen Platform during pre-administration

November 2023 Page 20

Item authoring – ID uses a centralized item authoring system for managing the metadata associated with NAEP items. A new NAEP Item Viewer tool integrated with the item authoring system supports an improved review process. A single UI rendering codebase ensures consistency between assessment content delivery and the item previews. The Item Viewer implements a quality control step allowing item authors to preview content with the NAEP secure browser and can be extended to integrate with other authoring systems in the future.

Item content and bookmap import – The data import tool allows content migration of all NAEP item types in all supported languages via an automated API-based approach. A bookmap is a configuration file that provides the rules for constructing student-delivered items into assessment blocks and forms.

The eNAEP Next-gen Platform Data Uploader tool built using serverless technologies imports the bookmap and the content into the Platform for assessment delivery.

Prepare supporting materials – The content management tool prepares and integrates any supporting materials for the assessment used during the assessment delivery. The content management tool will support the preparation and integration of assessment-supporting materials consumed by the Platform for assessment delivery (e.g., tutorials, Student Help, OMB statements).

Assessment/content review – The Next-gen Platform supports the Random Access Viewer and Random Access Launcher tools for content preview. These tools can also be used with an Electronic Review Package (ERP)—a NAEP tool for item content reviews—to support the content package reviews. NPD will provide separate environments to support various reviews by committees, NCES, and Alliance partners as needed. The Next-gen Platform will provide environments for online assessment review by NCES and Alliance partners.

November 2023 Page 21

2.3.2 Phase 2: Platform distribution and deployment

Once the assessment items and other content are integrated with the Platform and certified by NCES and the Alliance, the ADP app needs to be installed and tested on NAEP-provided devices and NCS devices prior to field operations. Figure 2-5 depicts the Platform distribution and deployment process that will occur leading into assessment day.

Figure 2-5. The Next-gen Platform distribution and deployment process

The Next-gen Platform currently supports Windows and ChromeOS devices with Mac OS support planned in the future as NAEP transitions to SBE. The applicable packages, installers, applications, configurations, and tools will be deployed to the AWS-hosted NPD Download Center. Software and content are updated from the AWS NPD Cloud. The ADP application is deployed to the appropriate student devices for assessment delivery. To support NPN Mode assessment delivery, the appropriate NCS application is deployed to the NCS devices. At a later stage (typically just before the administration), software, content, and configurations will be auto-updated with the latest data from the AWS NPD Cloud.

2.3.3 Phase 3: Field administration

During the field administration, the Next-gen Platform is used on NAEP-provided devices within a school environment. Both Online and NPN Modes of assessment delivery are supported by the Next-gen Platform. Students participating in the NAEP assessment interact with the Platform on their designated eNAEP student device. Trained field administrators will use the registration system administration

November 2023 Page 22 device to monitor the assessment using the registration system. Figure 2-6 depicts the following conceptual data entities for which the registration system is the source of truth:

• Administration codes

• Teachers list

• Accommodation codes

• Booklets

• Schools

• SQ settings

Figure 2-6. Registration system conceptual data entities for NAEP field administration

As depicted in figure 2-6, the registration system and its cloud back end send the school, session, and student data to the Next-gen eNAEP Platform for successful Online or NPN Mode field administration.

The Platform APIs provide the session monitoring data required for field staff to use the registration system effectively.

2.3.3.1 Online Mode of assessment delivery

As seen in figure 2-7, the Online Mode relies on the list of test takers being sent to the AWS NPD Cloud from the registration system running on the registration system device. The student devices access this student information from the AWS NPD Cloud just in time for the online assessment. The eNAEP Student

November 2023 Page 23

Device exchanges real-time data with the AWS NPD Cloud for assessment content, device status, and data capture during online assessments. Administrators can monitor the status on the registration system administration device (using the registration system), which pulls student status information from the AWS NPD Cloud.

Figure 2-7. Online assessment delivery mode for the Next-gen Platform

November 2023 Page 24

2.3.3.2 NAEP Private Network Mode of assessment delivery

As depicted in figure 2-8, the NPN Mode of assessment delivery involves using the eNAEP router, NAEP Classroom Server, registration system administration device, and the eNAEP student devices. Before the assessment, the administrators ensure the school, session, and student data are available in the NAEP Classroom Server. During the assessment, the student devices communicate exclusively with the NCS device for configurations from the NCS, or to send the assessment progress, response, and diagnostic data to the NCS in real time. The NCS device sends assessment progress and monitoring information to the registration system so that the field staff can effectively administer and monitor the assessment.

This backup mode setup relies only on the eNAEP router, student devices, NCS device, and registration system device for assessment delivery, without the need for establishing an internet connection during assessment delivery. Data transfer to the AWS NPD Cloud for all the student sessions in the backup mode (NPN) is done by the field staff using the Data Transfer service after the assessment administration.

Figure 2-8. NPN assessment delivery mode for the Next-gen Platform

November 2023 Page 25

2.3.3.3 Field staff support

The Next-gen Platform provides support for field staff through the registration system with an interface for administrative functions and automated data exchange with the Platform’s back end. It also supports the SDC and NPD Help Desk staff during field administration by providing them with dashboards and data analytic capabilities. More details of the field administration support are provided in Section 3.3.3 Field Operations Support Components, after related components of the software are introduced.

As the NPD solution evolves throughout further releases and updates, these troubleshooting tools will be refined and better defined by working with NCES, SDC, and other Alliance partners. These updates will be captured throughout the lifecycle of this living document.

2.3.3.4 Operational dashboard

The Alliance and NCES rely on the NAEP Operational Dashboard to monitor and track the progress of assessments during an administration in order to respond to operational issues in a timely and consistent manner.

November 2023 Page 26

Figure 2-9 shows the data sources for the Next-gen Platform NAEP Operational Dashboard, the data pipeline that supports enrichment and aggregation of the real-time data, and the tools to be leveraged for the Dashboard views.

Figure 2-9. Next-gen…