Attachment C - Assessment Content Team SOOs (4-6).docx

Tasks 4, 5, & 6 (Team Contracts)

Statement of Objectives

Assessment Content Team

Assessment Content Development

Scoring

Platform Development

National Assessment of Educational Progress

(NAEP)

2024-2029

4. Assessment Content Development Contract The purpose of this contract is to develop content for NAEP’s Digitally Based Assessments (DBA), including development and maintenance of the cognitive assessments, survey questionnaires, and supporting content required to successfully administer NAEP (e.g., student directions, ancillary materials).

The assessment content development contractor shall:

· Implement efficient, effective, and systematic processes for the development and review of new cognitive content (including scoring rubrics), survey questions, and supporting assessment materials (e.g., ancillary documents) that align with NAEP’s goals for online and device agnostic delivery and NAEP framework requirements.

· Develop and maintain pools of content that seamlessly operate within NAEP’s assessment and questionnaire delivery platforms provided by the NAEP Platform Development (NPD) contractor (including the platform’s digital universal design elements, tools, and accommodations), support measurement and reporting across the full range of student performance, and are fair, accessible, valid, and engaging to students.

· Develop and maintain survey questionnaires that support the reporting and understanding of student achievement, and are efficient, engaging, fair, and accessible to respondents.

· Configure cognitive content and survey questions into blocks, including supported accommodations blocks, according to the assessment design provided by the Design, Analysis and Reporting (DAR) contractor.

· Review and certify content and supporting ancillary materials ahead of each administration.

Overview NCES has a commitment to ensuring all digital content developed for the NAEP assessments is accessible, innovative, and engaging, while efficiently, validly, and reliably measuring the full range of student performance and aligning to NAEP frameworks. As part of the next generation of NAEP digital assessment development, this includes – but is not limited to – the following goals for the Assessment Content Development (ACD) contract:

· Integrate considerations for diversity, equity, inclusivity, and accessibility within all digital assessment development processes and procedures and ensure that NAEP content is free from bias.

· Utilize mechanisms and approaches to efficiently develop robust pools of items and scoring rubrics (e.g., automated item generation, multiple item versions with varying difficulty) while also maintaining fidelity to validity and reliability.

· Advance the development of survey questions that support diagnostic reporting and understanding of student achievement, and achievement trends over time.

· Support the NAEP Platform Development (NPD) contractor to create efficiencies and technical solutions for the development and implementation of content for NAEP’s digital assessment and questionnaire platforms.

· Implement approaches for development of content that measure the full range of student performance to improve the capacity of NAEP to report performance for all students, and particularly for improving measurement at the lower end of the performance scale.

· Propose technical approaches, solutions, and innovations that advance automation and other efficiencies in assessment development processes (e.g., automated item and content generation, translation).

· Examine assessment data (pilot and operational) to enhance the quality of the cognitive content and survey questions.

Scope of Work The following tasks represent the scope of this contract. The contractor must perform the tasks for each operational assessment, pilot test, and special study (e.g., National Indian Education Study (NIES)) to be conducted in the 2024-2029 period of performance (see Exhibit TBD for Assessment Development Activities). To accomplish these tasks, the ACD contractor will need to work closely with all NAEP contractors, and particularly with Design, Analysis and Reporting (DAR), Scoring, NAEP Platform Development (PD), and NAEP Data Collection (DC). Exhibit TBD, NAEP Schedule of Assessments on page TBD in the Overview chapter, indicates the assessments that are to be considered in this procurement. Note that the schedule is subject to change. The assessments to be conducted during this contract period will be based on frameworks established by the National Assessment Governing Board (NAGB).

The activities required in this statement of objectives fall within the following tasks:

· Manage Assessment Content Development (section 4.1)

· Develop Content for the Digital Cognitive Assessments (section 4.2)

· Develop Content for the Digital Survey Questionnaires (section 4.3)

· Prepare for the Assessment Administration (section 4.4)

· Convene External Reviews (section 4.5)

· Conduct Research and Development to support Assessment Development (section 4.6)

· Support IT Business Processes (4.7)

4.1 Manage Assessment Content Development

The NAEP assessment development cycle includes the following phases: (1) assessment design and development planning, (2) content development, including iterative reviews by a wide range of experts (e.g., practitioners), pretesting in front of target audiences, and platform implementation reviews, (3) block and booklet assembly, (4) pilot administration preparation, including in-system platform milestone reviews and certification, (5) piloting and post-piloting activities, including scoring and analyses, (6) selection of items for the operational assessment and block and booklet assembly, (7) operational administration preparation (i.e., in-system platform milestone reviews and certification), and (8) operational administration and post-operational activities, including scoring, analyses, and item release. The development cycle ends with a systematic review of the item pool to identify the item development needs for the next development cycle. Collaboration and coordination with other NAEP contractors are essential to ensuring the success of the NAEP program. Furthermore, research and development are integrated throughout each phase of the assessment development cycle. For example, moving towards multi-stage adaptive testing may require additional targeted development and changes in the block assembly approach.

Under this task, the contractor shall manage all assessment content development activities as part of the NAEP assessment development cycle, across the various tasks within this contract and in collaboration other NAEP contractors. The contractor shall develop and monitor detailed processes and procedures for implementing NAEP’s assessment development cycle that ensures that all NAEP assessment content is valid, reliable, accessible, and equitable, and meets best practices for assessment development. The contractor shall also establish and maintain, with approval from NCES, development and review protocols to ensure that all assessment development content (e.g., items, questions, indices, tasks, passages, tools) are constructed to reflect and enhance NCES’s program goals for diversity, equity, inclusivity, and accessibility.

The contractor shall develop a Project Management Plan that provides details on how the end-to-end assessment content development activities will be executed, monitored, and controlled. The plan shall document robust processes, protocols, and staffing to develop, implement, coordinate, and monitor all content development activities in support of this contract. The plan shall also include an overall assessment content development schedule that integrates key milestones and hand-offs from various assessment content project schedules and other NAEP contractor schedule dependencies.

4.2 Develop Content for the Digital Cognitive Assessments

Under this task, the contractor shall develop and maintain digital cognitive content for each pilot test, operational assessment, and special study administration for the period of performance. Development of digital cognitive content shall include developing items, tasks, and scoring rubrics. The contractor shall adhere to the highest standards in digital assessment development practices and implement efficient, effective, and standardized development processes across all NAEP assessments.

The contractor shall ensure that NAEP cognitive assessment content are:

· Aligned with the NAEP frameworks and assessment item specifications,

· Valid measures of subject-specific knowledge and skills that span the range of student performance,

· Equitable and free from bias against any group of students,

· Grade appropriate,

· Linguistically accessible to all students,

· Consistent with the principles of universal design and comply with Section 508 requirements,

· Compatible with platform and delivery requirements, i.e., online and device agnostic,

· Statistically sound, i.e., meet commonly accepted item statistical standards (e.g., difficulty, discrimination, and differential item functioning (DIF)),

· Engage students to do their best and allow students the opportunity to demonstrate their skills and knowledge in the construct of interest, including students with disabilities (SDs) and English learners (ELs),

· State of the art and follow best practices in education technology for digital content development.

The development and maintenance of assessment content shall be conducted in close collaboration with the DAR and NPD contractors. New development shall be purposeful and targeted to fulfill the reporting and design objectives for each assessment identified by the DAR contractor. The ACD contractor shall collaborate with the DAR contractor to determine:

· The number and types of items/tasks needed to fulfill measurement objectives and reporting goals established by the DAR contractor.

· The number and types of items/tasks needed to achieve sound measurement at each NAEP achievement level and across the student performance distribution.

· The number of piloted items/tasks in the item pool that are available for use.

· The number of items/tasks that require substantive revisions after pilot testing (such items must be re-piloted to verify their psychometric properties before being used in an operational assessment).

· The number of and which items to release after each operational assessment.

New development shall employ an efficient development process that utilizes technological advances in content development, including authoring tools (e.g., content authoring systems, benchmark components) to ensure new content, whether developed internally or externally, can be implemented efficiently and effectively into the digital assessment platform. The ACD contractor shall collaborate with the NPD contractor on the following:

· Development and execution of end-to-end assessment development processes for authoring, implementing, and reviewing content within the digital assessment platform.

· Development of content and platform specifications and new features to support NAEP assessment content and NAEP’s goals for device agnostic assessment, including updating NAEP style guides as needed.

· Development of externally developed content, if applicable (e.g., passages, multi-media stimulus, scenario-based tasks).

· Development of digital assessment resources (e.g., universal design elements, digital tools, interactive item components, simulations).

· Confirmation of the accuracy and functionality of the content in the NAEP platform, following implementation of content by the NPD contractor.

· Development of a schedule to support content development and platform implementation across all stages of development (e.g., external reviews, pretesting) and multiple assessment cycles.

NCES’s goals for NAEP digital cognitive assessment development are to leverage technological approaches to develop and administer content that aligns with NCES’s standards and NAEP Frameworks and maintains NAEP as the gold standard for large-scale assessments efficiently and effectively. The contractor shall employ innovative approaches that reflect technological advances in assessment development to achieve these goals.

4.2.1 Develop the Cognitive Assessment Content Development Plan Typically, the ACD contractor initiates development planning at least two years prior to the piloting of new items. The current practice has been to develop an overage of new cognitive content given the expectation that some content will not survive the reviews and testing, typically developing at a ratio of two to one. The contractor shall recommend to NCES the appropriate ratio and overage for the pilot assessment, considering goals for developing enough high-quality items for operational assessments and cost efficiencies for the program.

Development planning is a collaborative endeavor among multiple contractors and is guided by NAEP’s assessment frameworks and the NAEP Strategic Plan and vision for NAEP assessments to ensure development meets measurement, design and reporting goals. The contractor, in collaboration with the DAR contractor, shall develop a Cognitive Assessment Content Development Plan for each NAEP subject area assessment. The Development Plan serves as a comprehensive roadmap that specifies the content development needs for the pilot and operational assessment and guides the development of new cognitive content.

The Cognitive Assessment Content Development Plan shall include, at a minimum, the following components:

· A detailed project schedule for the end-to-end assessment development cycle, that also includes key milestones and hand-offs with other NAEP contractors.

· Content and psychometric characteristics of the complete item pool (including the alignment between student ability distribution and item difficulty distribution), to include the operational items and items previously piloted but not used in an operational assessment.

· A description of how the development targets satisfy measurement, design, and reporting goals for new development.

· A description of the approach for developing items of varying difficulty to enhance the alignment between student ability distribution and item difficulty distribution.

· A description of methods for recruiting and training content developers and item writers that includes an emphasis on equity, diversity and inclusivity, the composition of the pool of content developers/item writers, including their qualifications, and procedures for evaluating and monitoring their work.

· A description of how lessons learned in previous development cycles, including lessons learned from scoring and other relevant NAEP contract activities as appropriate, will be incorporated into item writer training and the development effort overall.

· Details on how new content will align to platform specifications and delivery specifications (e.g., device agnostic), and utilize the platform development tools, as applicable.

· High-level plans for conducting pretesting to evaluate the quality of new item types and features, including the rationales for the types of pretesting activities.

· In collaboration with NPD, a description of any new or updated content and platform specifications based on new development targets and reporting goals, including requirements and specifications for the use of digital resources (e.g., audio, video, texts, art, interactive item components, virtual tools, etc.).

· Robust plans for ensuring that all new content is valid, reliable, accessible, and equitable, and meets best practices for assessment development.

· Plans for conducting content quality external reviews (e.g., NCES, Subject area Standing Committee, Governing Board, State and Trial Urban District, etc.).

· Plans for obtaining, maintaining, and tracking copyright licenses for digital resources (e.g., audio, video, texts, art, etc.), as applicable.

· Detailed qualitative and quantitative plans for ensuring that all content selected for inclusion in the operational assessment measures what it is intended to measure.

· Plans for the release of content after an operational assessment.

The contractor shall use and reference as applicable the following in preparing the Cognitive Assessment Content Development Plan: (1) subject-specific frameworks and assessment item specifications documents developed by the National Assessment Governing Board (e.g., NAEP Mathematics Framework), (2) internal documentation developed and maintained by the contractor that clarifies and interprets the guidance from the Governing Board’s framework documents, (3) multi-year design and evaluation plan developed by the DAR contractor, and the (4) NAEP Strategic Plan maintained by the Program and Operations Coordination (POC) contractor .

4.2.1.1 Implement new and updated NAEP frameworks

The National Assessment Governing Board periodically reviews existing NAEP frameworks to determine whether updates are required. If a substantial update has been made to a NAEP framework or a new framework has been created, the contractor shall conduct necessary steps and procedures to implement the framework for the relevant content area assessment (e.g., review and code existing item pool to the new framework). As part of this task, the contractor shall also develop a Framework Implementation Plan. The Framework Implementation Plan serves as a comprehensive road map for carrying out the relevant activities for assessment design, development, and implementation, and record of the actions and procedures necessary to ensure that the successful implementation of NAEP assessments are aligned with new and updated frameworks.

The Framework Implementation Plan shall include, at a minimum, the following components:

· A review of the requirements from the framework that can be implemented within NAEP’s existing design and technology.

· Identification of new or updated requirements in the framework that require additional research to ensure correct implementation and a plan for researching these requirements.

· Alternative methods for achieving the requirements in the framework if existing NAEP design and technology do not allow for the attainment of these goals.

· An analysis of the trend item pool (including both cognitive and survey questionnaires) to determine the amount of trend content that can be carried forward under the new or updated framework and the resulting gaps in the item pool to accomplish the framework’s requirements.

· A roadmap across multiple development cycles for implementing the framework requirements.

The ACD contractor shall develop the Framework Implementation Plan in collaboration with other relevant NAEP contractors.

4.2.1.1.1 Framework Implementation Plan – Year 1

The contract shall perform tasks described above for framework implementation tasks required in the contract’s year 1.

4.2.1.1.2 Framework Implementation Plan – Year 2

The contract shall perform tasks described above for framework implementation tasks required in the contract’s year 2.

4.2.1.1.3 Framework Implementation Plan – Year 3

The contract shall perform tasks described above for framework implementation tasks required in the contract’s year 3.

4.2.1.1.4 Framework Implementation Plan – Year 4

The contract shall perform tasks described above for framework implementation tasks required in the contract’s year 4.

4.2.1.1.5 Framework Implementation Plan – Year 5

The contract shall perform tasks described above for framework implementation tasks required in the contract’s year 5.

4.2.2 Develop New Content and Scoring Rubrics

The contractor shall develop new content, including selected response items, constructed responsive items, scenario-based tasks, and associated scoring rubrics as specified in the item development plan (task 4.1.1).

NAEP is moving towards online and school-based equipment delivery methods, and as such new assessment content shall align to these delivery methods. Content developed shall be optimized for online delivery with the flexibility to be administered on devices of varying types and specifications, while continuing to maintain high standards for equity, validity, and accessibility and ensuring the assessments are valid measures of the intended constructs. Further, new assessment content shall be developed so it can be seamlessly implemented within NAEP’s digital assessment platform that is managed by NPD and aligns with NAEP’s development and style guidelines. To that end, NCES is interested in improvements to assessment content development that employ standardization and automation, leverage in-system development, prototyping, and authoring, and utilize NAEP’s benchmark component library. Please see Exhibit TBD for more information on NAEP’s current digital assessment platform and the content development tools available, such as the benchmark component library.

NAEP assessments typically include a range of digitally based discrete item types, item sets, Scenario-Based Tasks (SBTs), tools, and features to enable students to demonstrate their knowledge and skills. Exhibit TBD provides further details on the types of content available in NAEP assessments and NAEP’s current style guidelines. The ACD contractor shall be responsible for the selection of stimuli and texts, content design, development and authoring or programming of items and tasks, and the development of scoring rubrics as required by each NAEP subject area framework. Some content may require new or modified platform features and tools. The contractor shall collaborate closely with the NPD contractor and NCES to review new platform features to support new content development. Some content or content features requires advanced technical support and expertise to build (e.g., scenario-based tasks). Given the increasingly technical nature of NAEP’s digital-based content and to ensure development approaches follow technical standards and best practices, the development and programming of NAEP content shall be conducted and directed by a dedicated team of highly qualified staff with technical expertise in digital-based content development.

The following components shall also be completed as part of this subtask.

Training item writers.

Creating effective, valid, and high-quality content is a challenging process that depends on the quality of the content developers and item writers. The contractor shall conduct training consistent with the Contracts Office Representative (COR)-approved assessment content development plan and include the following information:

· NAEP subject-specific framework and item specifications,

· Development targets (e.g., content objectives, response format, difficulty, etc.),

· Platform specifications and NAEP content user interface and experience style guidelines

· Technical supports to standardize and automate development (e.g., benchmark components, templates for SBTs),

· Online and device agnostic authoring guidelines,

· Standards for equity and accessibility to ensure content is fair, relevant and avoids bias and sensitivity issues.

The contractor shall submit an agenda and proposed approach in advance of item writer training.

Scoring rubrics Student responses to the NAEP assessments are currently either algorithmically-scored or human-scored, depending on the item type, although NCES is transitioning towards more automated scoring. As such, NAEP content and scoring rubrics shall comply with best practices and standards to support automated scoring. The contractor shall develop scoring rubrics alongside item design and development that include: (1) scoring guides that describe criteria for each score level, including, when warranted, coding of different approaches or types of responses at a given score level, (2) expected prototypical responses for each score level, and (3) composite maps, where relevant. The contractor shall also provide rationales for item distractors and correct answer(s) (i.e., key), when appropriate.

Records of development.

The contractor shall maintain records for all assessment development content to document item history and maintain appropriate item and block-level metadata. The record of development (RoD) shall contain a link to the digital item, a brief description of item, item type, content area, variable label name, cognitive processes tested, tested skill(s), predicted difficulty level, relevant framework specifications, changes made to item or task, item reviewers’ comments, pretesting results (where applicable), item performance data (where applicable), and copyright status. Once an NCES-approved approach for developing equitable, diverse, and inclusive NAEP assessments is established, the RoDs shall also include relevant metadata for content alignment with this approach. The contractor shall submit on a schedule negotiated with the COR an interim RoD at specific stages of item development.

Item variable labels.

The contractor shall ensure that all cognitive item variable names are skill-based rather than explicit descriptors, to reduce the risk of item/task content disclosure.

Diversity, equity, inclusivity, and accessibility.

The contractor shall establish and maintain development and review protocols to ensure that cognitive assessment content (e.g., items, tasks, passages, tools) are constructed to reflect and enhance NCES’s program goals for diversity, equity, inclusivity, and accessibility (see task 4.1). Both individual content and the pool of content shall be developed and periodically reviewed for alignment with an NCES-approved approach that considers the diverse student populations taking NAEP assessments (see task 4.1). Further, all NAEP assessment content shall comply with the latest WCAG and 508 compliancy standards for digital content.

Acquire copyright licenses.

NAEP assessments include texts, passages, audio, video, and artwork that are acquired from published materials. All assessments with copyrighted material must have licenses before it is administered for the pilot and operational assessments. The contractor shall propose to NCES a timeline and process for acquiring copyright licenses for NAEP assessment content that accounts for the content development process and length of time to obtain permissions. The contractor shall routinely request the copyright releases and maintain the agreements not only for inclusion in the assessment materials themselves, but also for use on the NAEP website once items are released. The contractor shall ensure that status of copyright information for relevant items are included in RoDs and item inventories and shared with NCES periodically.

Framework Clarifications documents.

Framework Clarifications documents are internal documents that documents implementation decisions based on guidance from the Governing Board’s framework and specifications documents to provide implementation consistency across assessment development cycles. Assessment developers use the clarifications documents in tandem with the NAEP framework and specifications documents to apply implementation decision during the assessment content development cycle. As part of this task, the contractor shall develop and/or maintain the Framework Clarifications document for each subject area assessment, The clarifications document shall be periodically updated at specific, agreed-upon junctures during the assessment development cycle to reflect implementation and clarification decisions made.

4.2.2.1 Develop new content and scoring rubrics – Year 1

The contract shall perform tasks described above for content required in the contract’s year 1

4.2.2.2 Develop new content and scoring rubrics – Year 2

The contract shall perform tasks described above for content required in the contract’s year 2.

4.2.2.3 Develop new content and scoring rubrics – Year 3

The contract shall perform tasks described above for content required in the contract’s year 3.

4.2.2.4 Develop new content and scoring rubrics – Year 4

The contract shall perform tasks described above for content required in the contract’s year 4.

4.2.2.5 Develop new content and scoring rubrics – Year 5

The contract shall perform tasks described above for content required in the contract’s year 5.

4.2.3 Pretest new content and scoring rubrics

A subset of newly developed cognitive content shall be pretested in front of target audiences to: (1) validate that new items, tasks, formats, or item response tools are measuring what they intend to measure and are eliciting the expected responses, (2) confirm that there are no sources of construct irrelevant variance, (3) obtain preliminary quantitative data (e.g., percent correct, r-biserial, subgroup differences), and (4) determine whether the items and tasks are familiar, engaging and accessible to students. The contractor shall conduct pretesting to evaluate the quality of new or revised cognitive item types, response formats, features or constructs to ascertain item quality, language accessibility, accuracy of translations, accessibility, item difficulty, performance of response categories, timing, and other issues to be specified by the contractor. The contractor shall determine the appropriate method to use for pretesting, depending on the pretesting goals and research questions. In the past, NAEP pretesting has included one-on-one cognitive interviews and administration of items in situations mimicking an operational administration, as well as variations of those approaches. Regardless of the method, pretesting shall be conducted within NPD’s digital assessment platform, therefore coordination with the NPD contractor is critical for the success of this activity.

The contractor shall submit a pretesting plan prior to conducting pretesting that includes the research questions, the number of participants, a description of the methods and procedures for recruiting and/or sampling diverse student populations, the selection and rationales for the proposed content, pretesting methods to be used, the proposed interviewer script and probes, and a project schedule for pretesting that includes project milestones (e.g., OMB ICR clearance, integration of content in the platform) and timelines. The ACD contractor shall collaborate with the POC contractor in preparing the Office of Management and Budget (OMB) Information Collection Review (ICR) Clearance Forms required for pretesting.

The contractor shall recruit participants who reflect the diverse population of students participating in NAEP. While there are no restrictions to the location or mode of pretesting (i.e., virtual or in-person), considerations should be given to achieving a diverse sample of students while also maintaining assessment content security. NCES shall have final authority on the location and mode of pretesting. The contractor shall provide NCES regular updates on the progress of recruitment and implementation and provide methods for allowing NCES, or an NCES representative, to observe pretesting sessions.

Following the pretesting activity, the contractor shall submit a report outline and final report on the findings from pretesting. The final report shall include, at a minimum, a description of the participants, the study procedures, a summary of findings with supporting evidence related to key pretesting goals, detected problems within items, tasks and scoring rubrics, sources of difficulty that respondents had with the items and tasks, and suggestions to revise and improve problematic items. Item-level recommendations for revising items ahead of the pilot administration shall be included in the RoDs.

4.2.3.1 Pretest new content and scoring rubric – Year 1

The contractor shall perform the work described above for contract year 1.

4.2.3.2 Pretest new content and scoring rubric – Year 2

The contractor shall perform the work described above for contract year 2.

4.2.3.3 Pretest new content and scoring rubric – Year 3

The contractor shall perform the work described above for contract year 3.

4.2.3.4 Pretest new content and scoring rubric – Year 4

The contractor shall perform the work described above for contract year 4.

4.2.3.5 Pretest new content and scoring rubric – Year 5

The contractor shall perform the work described above for contract year 15.

4.2.4 Maintain the Pool of Cognitive Assessment Items

In addition to the development of new content, the contractor shall maintain the trend content item pools to ensure that individual items and the pool of items maintain their validity and rigor continue to meet measurement and reporting goals. As part of this task, the contractor shall conduct thorough reviews of each trend item and the pool of items prior to each administration.

The trend content reviews shall include, but are not limited to:

· Alignment to NAEP frameworks,

· Continued accuracy of content,

· Content datedness,

· Fairness, bias, and sensitivity,

· Distribution of item difficulty,

· Accessibility and compliance with web standards,

· Consistency with NAEP visual style,

· Device agnostic rendering and functionality, and

· Alignment with NCES’s approach for diversity, equity, inclusivity, and accessibility.

Following the review, the contractor shall document issues identified during the review and propose recommended action to address issues. Recommendations that require changes to content shall obtain DAR contractor and NCES approval prior to implementation.

In addition, the contractor shall maintain inventories of the item pools that contain links to items and blocks in the digital assessment platform, scoring rubrics, the item variable labels, item type, framework classifications, item psychometric data (i.e., classical and Item Response Theory (IRT) data, Differential Item Functioning (DIF)), copyrights, and any changes introduced to the item or scoring rubrics during a given administration.

4.2.5 Support Scoring of Pilot and Operational Items

The scoring of NAEP assessments is a collaborative effort among NAEP contractors. NAEP currently employs three methods for scoring student responses to cognitive content: (1) algorithmic scoring of single response items, (2) algorithmic scoring of items with a finite number of correct response items (e.g., multiple-selection items), and (3) human scoring of open-ended response items.

The contractor shall collaborate closely with both the Scoring and DAR contractors in the scoring of the assessment items.

The contractor shall provide support to the Scoring in the training of trainers and scorers by providing subject-matter support staff. In collaboration with Scoring, the ACD contractor shall discuss plans for scoring, handoffs, and collaborations for training trainers to train scorers, training set construction, and shall identify and provide any materials or documents required to train scorers.

The ACD contractor shall collaborate with the Scoring contractor on the development of training sets for each item administered in the pilot assessments and for each new item in the operational assessments. The current practice is to develop training sets for all items with four score points or higher and as needed for items with fewer than four score points. Training sets include a selection of exemplar student responses for each score category within an item’s scoring rubric, a set of anchor papers and scoring practice sets, and qualifying sets as needed (typically for scoring rubrics with four score points or higher) for each constructed response item to guide training of scorers in preparation for pilot scoring. The ACD contractor shall develop draft training sets and conduct iterative reviews and revisions with the Scoring contractor and NCES.

During the scoring window, the contractor shall provide, at a minimum, one subject-matter specialist to provide content expertise and to monitor scoring activities throughout the course of scoring.

The contractor shall collaborate with the Scoring contractor to review and approve all changes to the scoring rubrics and training materials that arise during scoring.

The contractor shall actively participate in the following:

· Scoring planning meetings,

· Daily scoring calls during scoring,

· Scoring debriefing meetings at the conclusion of scoring activities where summary statistics, difficulties in scoring items, lessons learned etc. are presented, and

· Fulfillment of relevant action items identified in the Action Item Report resulting from the scoring debriefing meeting.

At the end of scoring, the contractor shall provide a summary report of the scoring activity that includes, but is not limited to, a summary of the outcomes of scoring and detailed lessons learned to improve the development of content and rubrics.

4.2.6 Configure Cognitive Content into Assessment Blocks

New cognitive assessment items and tasks should be assembled into assessment blocks for both the pilot and operational assessments using efficient and effective methods. Assessment blocks shall be assembled to be in alignment with the DAR contractor’s design plan, which describes the design approach, (i.e., linear, or adaptive), the total number of items and blocks required, and the configuration of those blocks to meet the measurement and reporting goals for each assessment.

In collaboration with DAR, the ACD contractor shall develop a block assembly design plan that allows the DAR contractor to implement automated block assembly (e.g., AI, use of algorithms). The block assembly design plan shall include an overview of the assessment design, summaries of the item pool, the block assembly requirements to meet this design, and the proposed structure for the blocks. Once the block assembly design plan is approved, the blocks shall be assembled, reviewed, and provided to NCES for approval.

4.2.6.1 Configure cognitive content into assessment blocks – Year 1 The contractor shall perform the work described above for contract year 1.

4.2.6.2 Configure cognitive content into assessment blocks – Year 2 The contractor shall perform the work described above for contract year 2.

4.2.6.3 Configure cognitive content into assessment blocks – Year 3 The contractor shall perform the work described above for contract year 3.

4.2.6.4 Configure cognitive content into assessment blocks – Year 4 The contractor shall perform the work described above for contract year 4.

4.2.6.5 Configure cognitive content into assessment blocks – Year 5 The contractor shall perform the work described above for contract year 5.

4.2.7 Translate Cognitive Assessment Content into Spanish

NAEP administers assessments in Spanish through bilingual Spanish accommodation booklets in all subjects except reading and a Spanish version of the grades 4 and 8 mathematics assessments in Puerto Rico. There are also accommodations booklets that provide Spanish-English bilingual directions only for all NAEP subjects. Ensuring the validity of the translated versions is important to NAEP. Equivalent and valid English and Spanish assessments for cultures and dialects in the mainland United States and Puerto Rico are essential to avoiding errors in interpretation and faulty conclusions.

The contractor shall develop equivalent and valid bilingual and Spanish NAEP assessments for use as accommodation booklets and for administration in Puerto Rico. Development of new content shall consider equity, accessibility and measuring the full range of student performance for bilingual and Puerto Rican students. The contractor shall implement efficient and valid processes and procedures for the translations of assessments into Spanish that embed the development of bilingual and Puerto Rico assessments within the end-to-end assessment development process. NCES is interested in technical approaches that leverage automation for the translation of NAEP assessment content, while maintaining reliability, validity, and equivalency.

To support the collection of reliable and valid reporting of student performance on the bilingual and the Puerto Rico assessments, the contractor shall develop a Translation Assessment Development Plan for each administration that includes, at a minimum, the proposed scope of translation activities, detailed translation and review processes that reflect current best practices and take advantage of technological efficiencies, and a detailed project schedule.

4.2.7.1 Translate cognitive assessment content into Spanish – Year 1 The contractor shall perform the work described above for contract year 1.

4.2.7.2 Translate cognitive assessment content into Spanish – Year 2 The contractor shall perform the work described above for contract year 2.

4.2.7.3 Translate cognitive assessment content into Spanish – Year 3 The contractor shall perform the work described above for contract year 3

4.2.7.4 Translate cognitive assessment content into Spanish – Year 4 The contractor shall perform the work described above for contract year 4.

4.2.7.5 Translate cognitive assessment content into Spanish – Year 5

4.2.8 Conduct post-data collection cognitive assessment quality reviews After each pilot and operational assessment (and special studies as applicable), in collaboration with the DAR contractor, the ACD contractor shall conduct a systematic review of each assessment instrument, including the items, tasks, blocks, to examine the quality of the assessment item pool and to inform item development needs for the next development cycle. An Item Development Quality Control (ID QC) meeting shall be held to present the review findings to NCES.

Post-pilot data collection reviews shall focus on how well the new pilot cognitive content performed in relation to the measurement goals for the pilot assessment. For cognitive content, this includes the evaluation of (1) the success of items and item pool in aligning to the subject area frameworks and specifications, (2) the psychometric properties of the items and item pool, including process data, (3) the validity of individual items, the item pool broadly, and for groups of students, including statistical evaluations for fairness and bias, and (4) lessons learned for the next development cycle. Following the pilot item quality reviews, the contractor shall identify which items can be selected for operational use and which items do not meet quality standards and should be revised or dropped from the item pool.

Post-operational data collection reviews shall focus on the cognitive content and psychometric characteristics of the assessment, to include evaluation of (1) alignment and coverage of the items and item pool to the subject area frameworks and assessment item specifications, and to the reporting and measurement goals, (2) the psychometric properties of items and item pool, including process data, (3) the validity and reliability of the items and item pool broadly, including statistical evaluations for fairness, bias, and sensitivity, and (4) lessons learned for the next development cycle. Following the operational item quality reviews, the contractor shall identify which items should be retired from the item pool because they do not meet qualitative or psychometric specifications. The contractor shall also support any additional post-data collection reviews to support special studies, field tests, or other data collection activities for NAEP.

In preparing for the quality review meetings, the contractor shall submit a draft agenda and provide materials in advance of the meeting. The materials shall include documentation of the design, development, psychometric properties of each cognitive/subject matter operational assessment. Following the quality review meetings, the contractor shall prepare a meeting report that includes meeting notes, key issues and action items discussed, and recommendations for item pools, i.e., incorporating piloted items in an operational assessment, items to retire, and items to revise.

In tandem with task 4.3.8, the contractor shall also develop and maintain an action item tracking system (e.g., tracking table) that can be easily accessible to NCES. The tracking system shall be periodically updated, as determined by the COR, to reflect the progress of the action items. The COR and relevant NCES staff shall be notified of any completed action items for review and approval.

4.2.9 Determine cognitive content to release

NAEP assessment items, tasks, and associated scoring rubrics are typically released as part of reporting to showcase the types of content and features included in operational assessments and to provide further context for the student performance data. Current practice is to release approximately two blocks from each operational assessment when the assessment results are published in the Nation’s Report Card. Released items currently reside on the NAEP Questions Tool (NQT).

In collaboration with the DAR contractor, the ACD contractor shall determine which items and/or blocks to recommend for release. Past factors that have influenced item release decisions include, but are not limited, to the following:

· Blocks of items that have been in the field for a long time,

· Blocks of items with special features or that are more difficult to replace (e.g., Bilingual, Braille, tools) are more likely to be retained longer than other blocks,

· Content coverage and framework representativeness across the collection of released blocks, and

· Psychometric properties of blocks.

The contractor shall submit an item release plan that includes the items and scoring rubrics proposed for release to the public and the rationale with supporting qualitative and psychometric statistics. The release plan shall include all necessary information to obtain approval of the content by NCES and the National Assessment Governing Board. Final approval of any released content rests with the NCES Commissioner and the National Assessment Governing Board.

The contractor shall support the DAR and Web Development, Operations, and Maintenance Support (WDOMS) contractors with releasing items and tasks on the NAEP website, including the NAEP Questions Tool or other tools developed to showcase released NAEP items, and as part of the release of NAEP results. The ACD contractor shall provide the DAR and WDOMS contractors with the released items, tasks, scoring rubrics, student responses for each score level, commentary for each student response, and other additional information or support as needed with COR-approval for implementation within the NAEP Questions Tool. The ACD contractor shall support the DAR contractor as requested to facilitate content interpretations and reviews as part of reporting and dissemination activities for the release of NAEP results.

4.2.10 (Optional) Develop Additional Cognitive Items At the exercise of this option, the contractor shall develop for a given subject and grade up to 20 percent additional cognitive items and associated scoring rubrics, pretest them, support scoring, configure blocks, translate, and perform post-pilot quality reviews as described above.

4.2.10.1 (Optional) Develop Additional Cognitive Items – Year 1 The contractor shall perform the work described above for contract year 1.

4.2.10.2 (Optional) Develop Additional Cognitive Items – Year 2 The contractor shall perform the work described above for contract year 2.

4.2.10.3 (Optional) Develop Additional Cognitive Items – Year 3 The contractor shall perform the work described above for contract year 3.

4.2.10.4 (Optional) Develop Additional Cognitive Items – Year 4 The contractor shall perform the work described above for contract year 4.

4.2.10.5 (Optional) Develop Additional Cognitive Items – Year 5

4.3 Develop Content for the Digital Survey Questionnaires

NAEP collects contextual information to fulfill the reporting requirements of federal legislation and National Assessment Governing Board policy, and to provide context for the reporting of student performance on the NAEP assessments. NAEP collects contextual information from students (grades 4, 8 and 12; ages 9, 13 and 17), teachers (grades 4 and 8), and schools (grades 4, 8 and 12; ages 9, 13 and 17) with survey questionnaires. Previously administered survey questionnaire items are available on the NCES website. Under this task, the contractor shall develop and maintain digital survey questionnaires for NAEP assessments.

As part of this task, NCES is interested in innovations that leverage digital and technological approaches for the development of student, teacher, and school survey questionnaire content design that aligns with NCES’s program goals for online and device agnostic administration, NCES’s statistical standards, and NAEP Frameworks while also considering the need for maintaining trends in reporting.

Development of new survey questions may relate to the two types of information collected by the survey questionnaires—core information and subject-specific information.

Core questions. Core questions, such as race/ethnicity and educational resources at home, and teacher education have been in place for multiple administrations. The contractor shall focus work in this task on a review of the quality of current NAEP core questions (as described in task 4.2.1), monitoring changes in trends due to the impact of policy implementation coming from other agencies (e.g., changes in collecting and reporting on race/ethnicity), and searching for alternative sources of data to reduce the burden on NAEP participants.

Subject-specific questions. Subject-specific questions represent factors that may be associated with student achievement in the assessed subject such as relevant course content and requirements, curricular offerings, and teacher preparation. Increasingly, NAEP subject area frameworks are providing guidance to the development of subject-specific survey questions. Prior to each administration, or at a frequency determined by the COR and relevant NCES staff, the contractor shall annually review these factors and update the questions in the student, teacher, and school questionnaires prior to pilot testing.

The contractor shall leverage the highest standards in assessment content development practices…