Attachment R-2023 Scoring Memo 10.26.23.pdf

N A E P 2 0 2 3 S c o r i n g M e m o r a n d u m

NAEP 2023

Scoring Memorandum

Constructed Response Scoring of

2023 NAEP

Long-Term Trend and Field Test Assessments

Prepared by Pearson for NCES

10-26-23 version

2023 NAEP Scoring Calendar

Activity Math Reading Math Reading Science

LTT 13 Field Test

Test Administration

10/10 - 12/16

3/20 - 4/14

Pearson Scoring Center

Pearson / ETS Train the Trainer I

1/3 - 1/4 1/3 - 1/4

6/13 - 6/14 3/29 - 3/30 N/A

ETS delivers rubrics to Pearson

12/13 N/A

N/A

Samples delivered for Training Set creation

N/A N/A

Training set Assembly, Review, Approval

Standing Committee Reviews

N/A N/A N/A

Train the Trainer II

1/10 - 1/11 1/10 - 1/11

7/5 - 7/6 4/24 - 4/28 7/5 - 7/6

Scoring Starts 1/17 1/17 7/10 5/1 7/10

Scoring Ends 1/23 1/23 7/14 5/12 7/21

MDPS Delivers Scored Data to DAR

1/30 1/30

7/21 5/19 7/28

ID delivers ROA sets to

MDPS

N/A N/A N/A

Table of Contents

1 INTRODUCTION TO NAEP SCORING

1.1 THE NAEP PROGRAM FOR 2023

1.1.1 NAEP Components

1.1.2 Long-Term Trend

1.2 CHANGES FOR 2023

1.3 ROLES AND RESPONSIBILITIES IN NAEP SCORING

1.4 OTHER RESOURCES

1.5 PEARSON SCORING SYSTEM -- OVERVIEW

1.5.1 Late Receipts

2 NAEP SCORING QUALITY ASSURANCE

2.1 DEBRIEFING AND PLANNING MEETINGS

2.1.1 Lessons Learned

2.2 NAEP SCORING STAFF QUALIFICATIONS AND HIRING

2.3 PLACEMENT TOOLS

2.4 MONITORING TREND

2.5 NAEP SCORING GUIDES (RUBRICS)

2.6 TRAINING SETS

2.7 MONITORING SCORER ACCURACY

3 OVERVIEW OF 2023 SCORING

3.1 STUDENT SAMPLE SIZES

3.2 NUMBER AND TYPES OF ITEMS

3.3 NUMBER OF STUDENT RESPONSES

4 TRAINING SET ASSEMBLY AND APPROVAL

4.1 TRAINING SET DESIGN

4.2 SAMPLE SELECTION

4.3 TRAINING SET ASSEMBLY

4.3.1 Pilot Training Set Assembly

4.3.2 New Operational Training Set Assembly

4.4 STANDING COMMITTEE REVIEW

5 PRELIMINARY SCORING ACTIVITIES

5.1 OE SPREADSHEET

5.1.1 SRDS (Student Response Definitions and Specifications)

5.2 TREND POOLS

5.2.1 Training Trend Pool

5.2.2 Monitoring Trend Pool

5.2.3 Creation of the Trend Pools

5.3 ITEM-LEVEL SCHEDULING

5.4 TRAIN THE TRAINER I

5.5 OPERATIONAL TREND ITEM PREPARATION

5.6 NEW OPERATIONAL ITEM PREPARATION

5.7 PILOT ITEM PREPARATION

5.8 TRAIN THE TRAINER II

6 SCORING PROCEDURES AND MONITORING

6.1 GENERAL SCORER TRAINING PROCEDURES

6.1.1 Scorer Orientation

6.1.2 Scorer Content Training

6.1.3 Scorer Qualification

6.1.4 Training Trend

6.2 TRAINER AND SUPERVISOR ROLES IN SCORING MONITORING

6.2.1 Backreading procedures

6.2.2 Backreading scores

6.3 WITHIN-YEAR INTERRATER RELIABILITY (SECOND SCORING)

6.4 CALIBRATION

6.5 SCORING PROCEDURES – PILOT ITEMS

6.6 SCORING PROCEDURES – NEW OPERATIONAL ITEMS

6.7 SCORING PROCEDURES – OPERATIONAL TREND ITEMS

6.8 SCORING PROCEDURES – BILINGUAL ITEMS

6.9 REPORTS USED TO MONITOR SCORING

6.10 GUIDELINES FOR ESTABLISHING MONITORING STATISTICS

6.11 TREND MONITORING

6.11.1 T-test

6.11.2 Cross-year percentage exact agreement (trend year reliability index)

6.11.3 Unweighted Kappa

6.11.4 Conditions for trend flagging

6.11.5 Binomial Test

6.11.6 Rescoring guidelines

6.12 CONDITION CODES IN MONITORING

6.13 DAILY CONFERENCE CALLS

6.14 A COMPREHENSIVE REVIEW OF KAPPA

7 POST-SCORING ACTIVITIES

7.1 DATA DELIVERY

7.2 PEARSON ROA

7.3 ETS MEMORANDA

APPENDIX A: GLOSSARY OF TERMS

APPENDIX B: FACILITIES AND STAFFING

APPENDIX C: HISTORY OF NAEP SCORING

APPENDIX D: EPEN REPORT DEFINITIONS

APPENDIX E: CONDITION CODES

APPENDIX F: UNIVERSAL PLACEHOLDERS

APPENDIX G: FEEDBACK LOOP

APPENDIX H: PROCESS FOR MACHINE SCORED ITEMS

2023 SCORING MEMO ADDENDUM

TRAINER HANDBOOK

1. OVERVIEW OF NAEP

1.1. BACKGROUND OF NAEP

1.2. NAEP ORGANIZATIONAL STRUCTURE

1.3. THE NAEP PROGRAM FOR 2023

1.3.1. 2023 Scoring from Home

1.4. LONG-TERM TREND

1.5. SECURITY AND CONFIDENTIALITY

1.6. NAEP WEB SITE

2. ROLES AND RESPONSIBILITIES

2.1. ETS CONTENT LEAD ROLES & RESPONSIBILITIES

2.2. PEARSON CONTENT SPECIALIST ROLES & RESPONSIBILITIES

2.3. TRAINER ROLES & RESPONSIBILITIES

2.3.1. NAEP Trainer

2.4. SUPERVISOR ROLES & RESPONSIBILITIES

2.4.1. NAEP Scoring Supervisor

2.5. SCORER ROLES & RESPONSIBILITIES

2.6. ADDITIONAL PEARSON STAFF

3. NAEP READING PROJECT OVERVIEW

3.1. OVERVIEW OF FRAMEWORK AND ASSESSMENT DESIGN

3.2. NAEP 2023 READING SCORING GUIDE OVERVIEW

GENERAL GUIDELINES FOR THE SCORING OF CONSTRUCTED RESPONSE QUESTIONS

3. NAEP MATHEMATICS PROJECT OVERVIEW

3.1. OVERVIEW OF FRAMEWORK AND ASSESSMENT DESIGN

3.2. NAEP 2023 MATHEMATICS SCORING GUIDE OVERVIEW

GENERAL GUIDELINES FOR THE SCORING OF CONSTRUCTED RESPONSE QUESTIONS

3. NAEP SCIENCE PROJECT OVERVIEW

3.1. OVERVIEW OF FRAMEWORK AND ASSESSMENT DESIGN

3.2. NAEP 2023 SCIENCE SCORING GUIDE OVERVIEW

GENERAL GUIDELINES FOR THE SCORING OF CONSTRUCTED RESPONSE QUESTIONS

4. TRAINING SET ASSEMBLY (PILOT & NEW OPERATIONAL ITEMS)

4.1. GUIDELINES FOR PRODUCING TRAINING SETS

4.2. TRAINING SET OVERVIEW AND MATERIALS

3.3. OVERVIEW OF SAMPLE SELECTION AND SET ASSEMBLY

4.4. DETAILED GUIDE FOR SET ASSEMBLY

4.5. REVIEW AND APPROVAL OF TRAINING SETS

5. NAEP ANNOTATION WRITING GUIDELINES

5.1. INTRODUCTION

5.2. CHARACTERISTICS OF ANNOTATIONS

6. ITEM SCHEDULE AND NOMENCLATURE

6.1. DBA ITEM NUMBERS

3.2. TRADITIONAL PAPER ITEM NUMBERS

6.2. COMBINED ITEMS

7. TREND ITEM PREPARATION

8. NAEP ITEM TRAINING

8.1. ADULT LEARNING

8.2. NAEP TRAINER TECHNIQUES FOR ITEM TRAINING

8.2.1. Composition of NAEP Training Sets

8.2.2. Qualifying

8.2.3. Overview of Item Training

8.2.4. Training Trend

8.2.5. Use of Additional Training Notes for Training or Scoring

8.3. THE FIRST DAY OF THE SCORING WINDOW

9. RECORDKEEPING DURING TRAINING AND SCORING

9.1. REQUIRED FORMS FOR TRACKING TRAINING AND SCORING ACTIVITIES

9.2. DOCUMENTING SCORING DECISIONS THROUGHOUT THE TRAINING AND SCORING PROCESS

9.3. DOCUMENTING CHANGES TO TRAINING SETS DURING TRAINING, SCORING, OR POST-SCORING

10. MONITORING SCORING

10.1. INTRODUCTION

10.2. CURRENT YEAR INTER-RATER RELIABILITY (SECOND SCORING)

10.3. GETTING STARTED WITH MONITORING

10.4. MONITORING SCORING: PRIMARY MONITORING TASKS

10.4.1. Backreading

10.4.2. Current-Year Reports

10.5. MONITORING SCORING: ADDITIONAL DAILY MONITORING TASKS

10.6. REVIEW AND PULLED PAPERS QUEUES

10.7. PSYCHOMETRIC REVIEW OF SCORING DATA AND PEARSON-ETS COMMUNICATIONS

11. MONITORING TREND SCORING

11.1. CROSS-YEAR (TREND) INTER-RATER RELIABILITY

11.2. PRINCIPLES OF TREND MONITORING

11.3. TREND BACKREADING

11.4. TREND REPORTS

11.5. MONITORING THE SCORING OF OPERATIONAL TREND ITEMS

11.6. TRMC (TREND RELIABILITY AND MEAN COMPARISON)

11.7. SPREADSHEET REPORTS: TRMC/FLAG STATUS REPORT, COMPLETION REPORT

11.8. TREND FLAG CRITERIA

12. FEEDBACK DURING SCORING

12.1. FEEDBACK TO INDIVIDUAL SCORERS

12.2. CALIBRATION SETS

12.3. TRAINING TREND AS A CALIBRATION ACTIVITY

13. FINAL RECORDKEEPING REQUIREMENTS

13.1. DOCUMENTING ITEM PERFORMANCE

13.2. ARCHIVING REQUIREMENTS

13.3. ROA REQUIREMENTS FOR PILOT ITEMS

13.4. ROA REQUIREMENTS FOR NEW (FIRST-YEAR) OPERATIONAL ITEMS

13.5. ROA REQUIREMENTS FOR OPERATIONAL TREND ITEMS

APPENDIX A. MECHANICS OF ANNOTATION WRITING

APPENDIX B. GLOSSARY OF TERMS

APPENDIX C. COMMON ABBREVIATIONS AND ACRONYMS

APPENDIX D. CONDITION CODES

APPENDIX E. ALERTS PROCESS

APPENDIX F. BILINGUAL SCORING

APPENDIX G. MONITORING SCORING OVERVIEW

APPENDIX H. GUIDELINES FOR POST-PILOT CHANGES

APPENDIX I. FORMS

Tables

TABLE 3.1 2023 NAEP TARGET SAMPLE SIZES

TABLE 3.2 2023 NAEP ITEMS

TABLE 3.3 2023 NAEP TARGET NUMBER OF RESPONSES PER ITEM

TABLE 3.4 2023 NAEP SECOND SCORING PERCENTAGE

TABLE 3.5 2023 NAEP ESTIMATED TOTAL NUMBER OF RESPONSES TO BE SCORED

TABLE 6.1 2023 NAEP IRR FLAGGING THRESHOLDS

1 Introduction to NAEP Scoring

This Scoring Memorandum is a collaborative document prepared by Pearson to provide NCES with documentation of the NAEP 2023 scoring process. ETS was the primary author for the Trend Monitoring section of this document, with some input from Pearson. While primarily geared toward NCES, other potential users of the Scoring Memorandum include Pearson, ETS, and AIR.

Significant 2023 scoring dates are presented in the Scoring Calendar located on the inside cover of this Memorandum. Several appendices appear at the end of this Memorandum. Please refer to page 79 for a Glossary of Terms relevant to the NAEP scoring process.

This Scoring Memorandum is a companion document to the NAEP Trainer Handbook. The Trainer Handbook is tailored specifically to support NAEP Trainers by providing a useful reference document that presents the scoring information most directly relevant to the job of NAEP Trainer. NCES is the target audience for the Scoring Memorandum, a main purpose of which is to provide documentation of the scoring process, statistical monitoring criteria, and Quality Assurance procedures. Though the primary audiences differ, both NCES and NAEP Trainers have complete access to both documents. Both the Trainer Handbook and Scoring Memo will be provided and used for training purposes. Since both documents reference each other frequently, for ease of access the Trainer Handbook has been appended to this Scoring Memorandum.

1.1 The NAEP Program for 2023

The NAEP assessment for 2023 includes the following assessment components that will be scored by trained and qualified professional Scorers:

Operational LTT paper-based assessments (PBA) in Mathematics and Reading at age 13

Next-gen/Mixed-device Field Test assessments at grades 4, 8, and 12

1.1.1 NAEP Components

The NAEP 2023 scoring activity will include:

The NAEP Long-Term Trend (LTT) Operational assessment will be administered to a sample of age 13 students.

The NAEP 2023 Field Test assessment will be completely digital, and the Next-gen eNAEP delivery system will be used to administer the assessment on NAEP-provided devices. Technology-enhanced items as well as several scenarios-based tasks, from the 2022 (mathematics and reading) and 2019 (science) assessments will be administered in the 2023 Field Test.

1.1.2 Long-Term Trend

NAEP periodically administers a Long-Term Trend (LTT) assessment in mathematics and reading. For the NAEP LTT assessments, students are sampled by age, rather than by grade as for main NAEP assessments. The 2023 NAEP LTT assessment will be administered to students at age 13 (October through December, 2022).

Since its inception in 1969, NAEP has served the important function of measuring our nation’s educational progress by regularly administering various subject area assessments to nationally representative samples of students. The LTT reading assessment was first administered in 1971, and the LTT mathematics assessment in 1973. The LTT assessment measures student progress over time. In contrast to the main NAEP assessment that is updated regularly to reflect current educational content and assessment methodology, the LTT mathematics and reading assessments have remained essentially stable since first being administered.

Many items presented at different age levels are completely separate and unique, while in some cases the same items are presented to multiple age groups and are scored according to the same criteria. Some items are administered to students at ages 9 and 13, some items to students age 13 and 17, and some items to students in all three age groups.

In 2001 the National Assessment Governing Board reviewed LTT assessment content. At that time, the Governing Board suspended the LTT assessments in writing and science. Beginning in 2004, only reading and mathematics would continue to be assessed in Long-Term Trend. The Governing Board also recommended that outdated items be replaced. Currently, as items are gradually released, new items are developed to assess the same item types, using the same format at approximately the same level of difficulty as the released parent block at each age. As with the main NAEP assessment, all passages and items developed for LTT are reviewed by ETS assessment specialists, by editors, and by trained fairness reviewers to ensure that the materials are free from gender and racial-ethnic bias and inappropriate language stereotypes.

The NAEP LTT reading assessment blocks are 15 minutes in length, and each student completes a test booklet that contains three cognitive blocks and one block of background questions. The majority of items in the LTT reading assessment are four-option multiple-choice items, but some blocks include one extended constructed-response (ECR) item scored with either a 4- or 5-point rubric. These are the items scored at Pearson scoring sites during LTT. Texts are classified as Expository, Narrative, Document, or Other (for riddles, visuals, and sentence matches). Items are classified as Identify Main Idea, Locate Information Verbatim, Locate Information, or Inference.

Each LTT mathematics block is 15 minutes in length, and each student completes a test booklet with three blocks of mathematics questions, along with one block of background questions. Each block of mathematics questions contains both multiple-choice questions and short constructed-response (SCR) questions. These SCR questions are the items scored at Pearson scoring sites during LTT. Each constructed-response item requires the student to perform a computation or solve a problem and write the answer in the space provided. All items are scored dichotomously (i.e., correct or incorrect) to be consistent with the blocks in the current Long-Term Trend assessment.

1.2 Changes for 2023

Scoring of the 2023 NAEP assessments was scheduled to completed using the synchronous scoring from home model. NAEP scoring has used this model previously during 2020 and 2022 NAEP scoring.

NCES asked the Alliance to accelerate the reporting timeline for LTT 13. The Alliance was able to fulfill this request by moving 2023 LTT scoring (and certain other Alliance activities) forward on the calendar.

The 2023 Field Test was designed to assess the performance of the NextGen eNAEP system in the field. Since the FT was designed to test the eNAEP system (not the performance of students):

the total number of students assessed was very small. There will be no external reporting of 2023 FT results. The expected total of 500 (or fewer) student responses per item is well below standard NAEP scoring volumes. As a result, ETS-DAR has reduced the normal IRR targets for human scoring by 10% for Reading and Science and 5% for Math. Scoring reports in 2023 will retain the standard NAEP IRR targets, but there will be an understanding during daily scoring calls that these targets are to be reduced by 10% and 5% for the discussion purposes. In addition:

scoring teams will be smaller for FT than traditional NAEP scoring.

Certain Mathematics FT items were administered in Puerto Rico, but were not included in the mainland FT assessment. As a result: certain Mathematics items will have Spanish student responses to be scored, but no English responses to be scored. The absence of English student responses will be a deviation from standard NAEP bilingual procedures. For these items, scoring teams will be trained using the English NAEP training set them move directly to scoring the Spanish student responses. This unusual situation is expected to apply only to the 2023 FT (it is anticipated that in the future Mathematics PR items will also be administered to mainland students).

In addition, a handful of Mathematics items were included only in PR accommodation forms (and not in regular PR forms). For this handful of items: the total number of student responses to be scored is likely to be in the single digits.

Previous NAEP scoring from home relied primarily on MS Teams as the collaboration tool. For 2023, Pearson has integrated a new video conferencing tool within the ePEN scoring system.

This will largely replace the scoring teams use of MS Teams. It is hoped that using a single system rather than needing to switch back and forth between ePEN and MS Teams will lessen the technology burden for scorers.

Because all 2023 items were returning (there were no new items): training sets and rubrics were loaded into the ePEN scoring system. This allowed easier access for scoring staff during scoring activities.

There was no trend scoring for the 2023 Field Test.

1.3 Roles and Responsibilities in NAEP Scoring

This section addresses roles and responsibilities in NAEP Scoring (please refer to page 82).

The chart below shows NAEP’s onsite scoring staff.

Content Specialist

Plan and lead Operational projects

ETS

Content Lead

Develop and oversee content

Ensure staffing needs are met

Review and approve training sets

Oversee content Train Trainers and Supervisors

Trainers (Pearson or ETS)

Monitor content Backread and monitor reports Build training sets Provide feedback to Scorers Train and monitor Supervisors and Scorers

Supervisors (Pearson)

Backread and monitor reports Provide feedback to Scorers Maintain team records

Scorers

(Pearson) Score student responses

As the Materials, Distribution, Processing, and Scoring (MDPS) contractor, Pearson has the primary responsibility for NAEP constructed response scoring. Pearson receives all student test booklets from the field. Pearson scans and processes student booklets in preparation for loading the student constructed responses into ePEN. Pearson receives digitally based assessment (DBA) student response files from the field. Pearson extracts individual student responses to each DBA item from the larger student response file. Pearson prints a sample of student responses and delivers these to the various Pearson scoring sites and to ETS. From these samples, Pearson and ETS Trainers select example papers that will be used to build training sets. Trainers create annotations for the new training sets. Trainers train Scorers. Pearson and ETS staff at Pearson scoring centers complete all constructed response scoring using the ePEN system discussed in the next section. To facilitate efficient scoring and scoring monitoring, Pearson provides all scoring Supervisors and is responsible for training Scorers for all scoring of operational constructed response items. Pearson hires Scorers and monitors their performance. Pearson sends all data on scored constructed response items to ETS for analysis.

ETS staff members, trained in the various content areas, create scoring guides, and approve training sets created by Pearson Trainers. This is a collaborative review process where sets go back and forth between contractors. ETS may create some training sets for pilot items. ETS content personnel may train Scorers for the scoring of some pilot items. In addition, ETS staff members play a consulting role during the scoring of all constructed response items. ETS research scientists and data analysts are responsible for analyzing the data on all items. The ETS research scientists who direct NAEP analysis review scoring monitoring data on a daily basis during scoring. ETS planning and coordination staff facilitate daily scoring monitoring calls.

ETS consults with Pearson on content-related scoring decisions during scoring. ETS develops and updates all scoring guides; Pearson will not alter scoring guides. At the operational stage, Pearson and ETS discuss any implementation issues and ETS resolves those issues after considering implications for trend. Notes of clarification (“Trainer Notes”) may be added to supplement the scoring guides; however, scoring guides at the operational stage are not revised.

Given the complex interactions among contractors that are necessary for NAEP scoring to be successful, it is crucial that responsibilities and procedures be clear and explicit.

The American Institutes for Research (AIR) supports the National Center for Education Statistics (NCES) administration of the NAEP program, including assessment development and scoring.

Below are specific contractor responsibilities:

Pearson provides content and procedural training for Trainers provides training for Supervisors trains Scorers on all operational constructed response items builds training sets for new operational constructed response items provides Supervisors and Scorers assembles the Trend Pool (monitoring and training trend) develops and maintains the electronic scoring environment (ePEN) used for NAEP scans/processes multiple-choice items scans/processes constructed response items provides and equips the physical buildings for all scoring activities documents and maintains files for all scoring decisions made during the course of scoring generates reports and participates in daily scoring calls delivers a data file of all scores to ETS maintains electronic copies of all training sets, including item documentation forms provides electronic images of scored responses for use in reports provides Training Trend images to Trainers to assist in Trainer preparation creates operational scoring packets for the Report of Activities (ROA) archives ETS-provided pilot scoring packets for the ROA prepares Trainer Handbook, with input from ETS prepares Scoring Memorandum, with input from ETS plans and leads Scoring Debriefing and Scoring Planning meetings provides most Trainers and assembles sets for pilot items

Educational Testing Service (ETS) may train Scorers for some pilot constructed response items sets statistical specifications for valid and reliable scoring of NAEP items monitors scoring statistics and ensures integrity of scored data participates in daily scoring calls consults with Pearson Content Specialist on the scoring of constructed response items communicates to Pearson information and guidelines for building training sets builds training sets for pilot constructed response items reviews and approves all sets develops and updates all guides plans and leads Standing Committee meetings monitors scoring, and provides content decisions when needed reviews all scoring documentation provides Pearson with pilot ROA sets for ETS-led items tracks training set assembly and coordinates daily scoring calls participates in the Scoring Debriefing and Scoring Planning meetings

1.4 Other Resources

All of the following sources provide additional documentation that impacts the NAEP scoring effort.

2023 Trainer Handbook – NAEP’s Trainer Handbook and Scoring Memo are companion documents. The Trainer Handbook also documents the scoring process, but is written specifically for NAEP Trainers, while the Scoring Memo is written for NCES. It is also incorporated into this Scoring Memo. The Trainer Handbook folder address is https://www.naepims.org/sites/ims/S/TrainerHandbook/2023

Frameworks – NAEP utilizes separate Frameworks for each subject area in Main NAEP.

The NAEP Framework for all subjects are available on the National Assessment Governing Board’s website, https://www.nagb.gov/naep/frameworks-overview.html.

More Framework information is available in Section 3 of the Trainer Handbook.

IMS – NAEP’s Integrated Management System is a secure website used for communication and sharing of documents among various NAEP contractors, and with NCES. The main website is accessible at https://www.naepims.org. The 2023 Scoring Memo subfolder on the IMS contains both an electronic version of this Scoring Memo, as well as numerous related documents. Documents available in this folder include the most up-to-date versions of the Scoring Calendar and Open Ended (OE) Spreadsheet (please refer to page 38), and other working documents related to the scoring process. The Scoring Memo folder address is https://www.naepims.org/sites/ims/S/ScoringMemo/2023 and includes:

o Scoring Calendar o Scoring Memo (with appended Trainer Handbook) o OE Spreadsheet

Other scoring folders include:

o Daily calls checklist o Daily scoring reports o Conference call minutes o Training set tracking

1.5 Pearson Scoring System -- Overview

This section addresses the Pearson scoring system. Pearson utilizes the Electronic Performance Evaluation Network (ePEN) to score student responses.

Pearson’s scoring network couples technology and software with secure network communications to transmit electronic images or files of student responses to trained professional Scorers. Pearson scans student test booklets at Pearson’s Iowa City facility. The original student test booklets remain secure at Pearson’s facility. Student responses are converted into an electronic format, and secure network connections transfer the images or files to the appropriate Pearson site. ePEN distributes the electronic images or files to individual computer workstations of the trained and qualified Scorers. All scores assigned are automatically captured and available for review by additional qualified scoring personnel. Pearson’s scoring network integrates multiple processes (routing work, scoring responses, monitoring quality, and tracking progress and workflow) into a single, efficient system.

The Pearson scoring network electronically manages multiple aspects of the scoring process, including project setup, all scoring monitoring activities, and reporting of data. The ePEN project setup process allows customization of many different variables to accommodate project needs, such as trend for NAEP. In addition, pre-defined scoring rules describe scoring parameters and how responses are routed through the system. These rules dictate the required reliability rate and an appropriate response rate to Scorers. ePEN also provides a variety of processes and tools needed to manage the quality of scoring, as dictated by Pearson’s standards for quality. Internal quality assurance standards are set and met for every Pearson scoring project.

NAEP uses trend scoring to compare the consistency of scoring over time (i.e., cross-year interrater reliability). During trend scoring, the ePEN system allows for the presentation of a pool of trend responses to Scorers and offers the ability to generate reports to evaluate scoring consistency. The scoring network’s data management capabilities allow the capture and reporting of scores to occur accurately and quickly. These on-demand scoring reports include daily, cumulative, Scorer specific and team specific information regarding both the quality and quantity of scoring activities. The reports provide information that enables supervisory staff to monitor scoring continually and accurately (please refer to page 91).

The ePEN system delivers DBA and paper responses to Scorers in a similar fashion. DBA student responses are received from the field as text files or images rather than papers booklets.

However, just like with paper student booklets: individual item responses are extracted from the larger student record and presented to Scorers via the same ePEN system.

Beginning in 2023: the ePEN system incorporated a new video conferencing feature. This new feature was designed to replace the use of MS Teams and allow physically remote scoring staff to use a single system for both scoring and discussion.

1.5.1 Late Receipts

A decision must be made for how responses that arrive late from the field will be scored. The late responses could be routed to the team of Scorers to score, or they can be scored by Trainers or Supervisors. The benefit of a traditional team of Scorers is the alignment with the standardized NAEP scoring process, which is in place as a quality measure to ensure and protect trend.

However, given the timing of the late arriving responses, a new team could potentially have to be created and trained which is not practical for a small number of late responses. The benefit of having Trainers/Supervisors score the late responses is that they ideally assign the best and most accurate score given their expertise. The risk, however, of moving away from a standardized scoring process by having Trainers/Supervisors score the late responses is the possible introduction of bias, which threatens trend.

To balance the potential risks with the practical implications for scoring late responses, the following guideline should be followed: If the total number of late responses per item is greater than or equal to 50, then the responses must be routed to a full team of Scorers. If the total number of late responses per item is less than 50, then the Trainers or Supervisors may score the responses.

2 NAEP Scoring Quality Assurance

NAEP takes numerous steps to maximize scoring quality. These include procedural steps before and during the scoring effort as well as rigorous monitoring and review during and after scoring.

Debriefing and Planning Meetings NAEP Scoring Staff Qualifications and Hiring Placement Tools Monitoring Trend Training Sets NAEP Scoring Guides (Rubrics) Monitoring Scorer Accuracy

2.1 Debriefing and Planning Meetings

After the conclusion of each year’s scoring, NAEP conducts a Scoring Debriefing Meeting.

NCES and NAEP staff from all contractors involved with the scoring process meet to review lessons learned from the just-concluded scoring season and to evaluate how those lessons may best be applied to the improvement of future NAEP scoring. Agendas, minutes, and other related documents are available on IMS at https://www.naepims.org/sites/ims/S/ScoringDebriefing/

In preparation for the upcoming scoring, NAEP conducts a Scoring Planning meeting. NCES and staff from relevant NAEP contractors meet to plan the schedule and procedures for the next year’s scoring. The 2023 Scoring Planning meeting was held March 31st, 2023. Agendas, minutes, and other related documents are available on IMS at https://www.naepims.org/sites/ims/S/PlanningMeetings/2023

2.1.1 Lessons Learned

Some lessons learned from previous Scoring Debriefing / Scoring Planning meetings include:

The ongoing practice of including more Trainers than scoring teams continues to be very successful. This allows Trainers occasional time away from the scoring team to work on recordkeeping and/or prepare for their next items.

Scoring agreements during 2022 Operational scoring were slightly lower than in 2019.

Changes to NAEP scoring between 2019 and 2022 include the Covid-required shift from onsite scoring to scoring from home. Another major factor impacting agreement rates was the significant reduction in NAEP sample sizes. The number of responses per item to be scored in 2022 was approximately 19% lower than in 2019. This reduction has an impact on final agreement rates because teams have less time / practice scoring each item.

2.2 NAEP Scoring Staff Qualifications and Hiring

NAEP Trainers, Supervisors and Scorers must all possess college degrees. Additional qualifications are noted below.

General Qualifications for Trainers

A 4-year college degree is required, and a content-related degree is preferred.

Previous experience as a scoring Supervisor or extensive experience as a Scorer is preferred.

Teaching experience is desirable.

Scoring experience, teaching experience, and/or past training/leadership experience carry significant weight in the selection of potential Trainers.

Previous NAEP scoring experience is preferred.

The ability to work efficiently with the ePEN scoring system, including backreading, monitoring current year reliabilities, and administering trend, as needed, is preferred.

General Qualifications for Pearson Scoring Supervisors

A 4-year college degree is required, and a content-related degree is preferred.

Previous experience as a scoring Supervisor or extensive experience as a Scorer is preferred.

Teaching experience is desirable.

Previous NAEP scoring experience is preferred.

The ability to work efficiently with the ePEN scoring system, including backreading, monitoring current year reliabilities, and administering trend, as needed, is preferred.

General Qualifications for Bilingual Trainers, Supervisors, and Scorers

A 4-year college degree is required.

Scoring experience, including the use of the ePEN system, is preferred.

In addition to the other qualifications listed in this section, bilingual Trainers, Supervisors, and Scorers must demonstrate Spanish language proficiency via a bilingual Placement Tool.

Subject-Specific Qualifications for Scorers ETS content leads have developed written NAEP Scorer placement tools for most operational subject areas. The Pearson scoring center staff will maintain a database of Scorer demographics, experience, and qualifications. Information collected will help place Scorers most appropriately.

Qualifications for NAEP Mathematics Scoring

A 4-year college degree is required. (An education degree in elementary or secondary mathematics education or a mathematics-related discipline, such as physical sciences or statistics, is preferred.)

Scorers must possess sufficient knowledge of the content described in the NAEP mathematics Framework to permit scoring of assessment items.

Scoring experience, including use of the ePEN system, is preferred.

Scorers must meet all item-level qualifying standards.

Qualifications for NAEP Reading Scoring

A 4-year college degree is required. (An education degree with English language arts, literature, reading, or writing concentration is preferred.)

Scorers must possess sufficient knowledge of the content described in the NAEP reading Framework to permit scoring of assessment items.

Scoring experience, including the use of the ePEN system, is preferred.

Scorers must meet all item-level qualifying standards.

Qualifications for NAEP Science Scoring

A 4-year college degree is required. (An education degree with a concentration in the humanities is preferred.)

Scorers must possess sufficient knowledge of the content described in the NAEP science Framework to permit scoring of assessment items.

Scoring experience, including the use of the ePEN system, is preferred.

Scorers must meet all item-level qualifying standards.

2.3 Placement Tools

Pearson, as the contractor responsible for hiring Scorers, will administer placement tools developed by ETS for all operational subject areas to determine the best subject/grade match for each Supervisor and Scorer hired. These instruments must be practical to administer and score for a large group. The placement tools are available in the Scoring Memo folder on the IMS.

The NAEP placement tools are used to identify Supervisors and Scorers with an appropriate level of content knowledge, and are analyzed in conjunction with other qualifications (education/experience) so that individuals can be appropriately placed on teams (e.g., based on grade-level knowledge and/or performance). The placement tools provide an important piece of information to aid in these placement decisions.

For the purposes of hiring and placement, Pearson uses multiple measures; placement tools are one measure of potential Scorer performance. These evaluation instruments are one of many criteria used for assigning individuals to scoring teams within the constraints imposed by the need to hire a large number of people in a relatively short time. In addition, Scorers meet qualifying standards before scoring specific items (refer to page 48) and meet ongoing expectations for performance based on Supervisor backreading and monitoring.

2.4 Monitoring Trend

A defining feature of NAEP is its ability to provide student achievement data that can be used to track changes in achievement levels over time. For this to occur, assessment results must be placed on an equivalent scale every time the assessment is administered. Psychometric procedures are applied to create this equivalency, with these procedures depending on a subset of assessment items being repeated across years. Because constructed response items are scored by human beings rather than machines (as multiple-choice items are), the scoring of these items is monitored carefully to ensure that scoring remains consistent across administrations.

Procedures for scoring trend items have improved over time. In past years, prior-year, or “trend” papers, were clustered together in a group. Scoring trend papers was consequently an activity distinct from scoring current-year papers, and Trainers, Supervisors, and Scorers could not help but be aware that they were engaged in an activity that differed from scoring current-year papers.

It was difficult to ensure that scoring practices did not shift in some ways when teams shifted from current-year papers to trend papers. The interspersed trend scoring model implemented in 2006 has improved the monitoring system by interspersing trend responses among the current-year responses rendering the prior-year responses indistinguishable from current-year responses to Scorers (please refer to page 39). This change has greatly improved the validity of the measure of cross-year scoring consistency.

2.5 NAEP Scoring Guides (Rubrics)

Scoring guides are the starting point for training sets. NAEP uses customized scoring guides to govern the scoring of constructed response items. The scoring guides are designed to ensure that Scorers follow a single standard and that scores are assigned consistently and fairly.

Scoring guides for constructed response items are developed with the items and undergo the same rigorous review process as the items. Only by keeping the two synchronized can item developers know that both item and scoring guide are measuring the intended objective. A typical scoring guide consists of a list of the different score categories for a particular item with descriptions of the criteria responses must meet to receive a particular score. NAEP also uses diagnostic codes to track response types within score levels for some items. Depending on the subject and the item, scoring guides may also contain a sample correct response or a list of credited responses if there is more than one possible answer.

Scoring guides receive external reviews just as items do. The subject area committee reviews are important because committee members are expert in both content and student cognitive development. Thus they contribute to the accuracy of the credited responses and scoring categories. Additional important comments emerge from the state item review and reviews by both NCES and the Governing Board. Another critical review occurs after the pilot test administration but prior to scoring, when the Standing Committee convenes to review the scoring guides and training materials in light of actual student responses. Here, committee members use their expertise to help developers make essential distinctions among the sample responses and fine tune final rubrics.

During the review of sample student responses obtained for new operational Items, Pearson and ETS work collaboratively, with Pearson Trainers initially sorting student responses into score categories based on original versions of the scoring guides and note any issues and observations for discussion with ETS staff. In consultation with ETS staff and members of the Standing Committees, scoring guides will be refined and approved by NCES. Training sets will be assembled and then given final approval by ETS item developers. ETS item developers will approve any revisions to scoring guides and training materials.

Additional subject-specific scoring guide information is available in Section 3 of each NAEP 2023 Trainer Handbook.

2.6 Training Sets

Individual training sets are created for every NAEP constructed response item. Training sets are first developed prior to the first operational scoring of each new item. Once created, the same training set is used from year to year throughout the item’s operational life. However, in some cases, new and variant student responses may be appended to the original training set to help further define and clarify the intent of the scoring guide. Training Scorers with the same training set each time an item is used is an integral part of NAEP’s process for maintaining consistent cross-year scoring quality. NAEP training sets go through a significant review and approval process to assure they are of the highest quality. New operational sets are assembled by Pearson

Trainers and reviewed by ETS item developers in a back-and-forth process designed to maximize the quality of new sets (please refer to page 35).

2.7 Monitoring Scorer Accuracy

Careful monitoring of scoring is a hallmark of NAEP. Monitoring is conducted at various levels:

by Trainers and Supervisors at the team level, by ETS psychometricians, by Pearson and ETS content staff, by Pearson program staff, and by NCES and AIR. Both backreading and second scoring are important monitoring tools (please refer to page 46).

3 Overview of 2023 Scoring

3.1 Student Sample Sizes

The total number of constructed responses to be scored is a function of the target sample size for each assessment. Presented below is a breakdown of the sample sizes by subject and grade, according to the NAEP 2023 Design Memo available here:

https://www.naepims.org/sites/ims/DA/Design/AnnualDesignPlans

Table 3.1 2023 NAEP Target Sample Sizes

Component Students

Total Students Assessed: 5,810 22,480 4,400 32,690

NAEP LTT

Age 9 Age 13 Age 17 Total Mathematics 8,345 Operational - 8,320 - 8,320 Bilingual - 25 - 25

Reading 8,320 Operational - 8,320 - 8,320

NextGen Field Test

Grade 4 Grade 8 Grade 12 Total Mathematics 8,770 Field Test 2,350 2,000 2,800 7,150 Bilingual 10 10 - 20 Puerto Rico 800 800 - 1,600

Reading 6,000 Field Test 2,650 1,750 1,600 6,000

Science 1,255 Field Test - 1,250 - 1,250 Bilingual - 5 - 5

3.2 Number and Types of Items

Tables 3.2 and 3.5 present information about the number of distinct ePEN items to be scored in 2023.

In ePEN scoring, distinct items are those which require a Professional Scorer to assign a distinct score, and for which separate statistics are reported. Individual parts of multi-part items, cross-grade items, and bilingual items, as well as Spanish-language items used in Puerto Rico, are all examples of item extensions that are considered distinct items within ePEN.

The number of distinct items from this scoring perspective may differ from the number of items as considered from an Item Development or a Data Analysis perspective, since these various extensions may, or may not, be tallied as separate items during development or analysis.

Each ePEN item (please refer to page 152Error! Bookmark not defined.) is identified as a separate row on both the OE Spreadsheet and the daily Completion Report, as illustrated in the hypothetical item numbers below.

Single item:

Subject Grade Item ID

Item Description Score Point Levels

Reading 8 23171R5_03 Big Cheese 1,2,3

Cross-grade items:

Reading 8 231923R6G08_04 Red Sweater 1,2,3,4 Reading 12 231923R6G12_04 Red Sweater 1,2,3,4

Multi-part items:

Subject Grade Item ID Part Item Description Score Point Levels Math 4 23191M11_16 Part A Number of seats at table 1,2 Math 4 23191M11_16 Part B Number of seats at table 1,2B,2A,3

Bilingual / Spanish items:

Math 4 23171M6_03 Town council members 1,2,3 Math 4 23171M6_03EN Town council members 1,2,3 Math 4 23171M6_03SP Town council members 1,2,3 Math 4 23171M6_03S Town council members 1,2,3

Table 3.2 presents the number of items broken down by multiple choice vs. constructed-response and score levels for constructed-response items. It also presents the numbers of multi-part items and numbers of items utilizing expanded scoring guides. Expanded scoring guides track different response types within score levels (please refer to Section 3 of the Trainer Handbook).

Table 3.2 2023 NAEP Items

NAEP Component

Se le ct ed

R es po ns e Constructed-Response

Items Total

CR

Items

Multi-part

Items

Ex pa nd ed

Ite m s

Total Items

Number of Score

Levels 2 3 4 5

Grand Totals: 1,101 204 145 34 10 393 51 124 1,494

LTT Mathematics Operational Age 13 117 35 - - - 35 - - 152 Bilingual Age 13 62 40 - - - 40 - - 102

Total 179 75 0 0 0 75 0 0 254

LTT Reading Operational Age 13 89 - - 4 2 6 - - 95

Total 89 0 0 4 2 6 0 0 95

Field Test Mathematics

FT

Grade 4 118 22 15 - - 37 8 22 155 Grade 8 105 23 15 1 - 39 14 16 144 Grade 12 99 13 13 1 2 29 6 15 128

FT Bilingual Grade 4 33 5 3 - - 8 - 6 41 Grade 8 31 2 6 - - 8 - 6 39

FT Puerto Rico Grade 4 108 25 17 - - 42 9 25 150 Grade 8 106 22 16 - - 38 9 15 144

Total 600 112 85 2 2 201 46 105 801

Field Test Reading

FT

Grade 4 53 8 17 8 - 33 - - 86 Grade 8 54 6 13 8 - 27 - - 81 Grade 12 36 3 10 4 - 17 - - 53

Total 143 17 40 20 0 77 0 0 220

Field Test Science FT Grade 8 67 - 15 5 3 23 5 14 90

FT Bilingual Grade 8 23 - 5 3 3 11 - 5 34 Total 90 0 20 8 6 34 5 19 124

3.3 Number of Student Responses

Table 3.3 presents the expected number of 2023 NAEP student responses per item based upon the target sample sizes. The number of student responses per item is determined by dividing the student sample size by the number of student book types and multiplying by the number of times each block appears.

Table 3.3 2023 NAEP Target Number of Responses per Item1

Component Responses

Age 9 Age 13 Age 17 Mathematics Operational - 4,160 - Bilingual - 45 -

Reading Operational - 2,800 -

Grade 4 Grade 8 Grade 12 Mathematics Field Test 500 500 500 Bilingual 10 10 - Puerto Rico 200 200 -

Field Test 500 500 500

Science Field Test - 500 - Bilingual - 5 -

1 These figures represent the target sample sizes. Actual receipts from the field are usually slightly higher than these targets. DBA estimates are very speculative.

Table 3.4 presents the second scoring percentage for 2023 NAEP scoring. Second scoring refers to scoring of the same student response by two different Scorers for the purpose of monitoring scoring consistency.

Table 3.4 2023 NAEP Second Scoring Percentage

Component Percentage

Age 9 Age 13 Age 17 Mathematics Operational - 25% - Bilingual - 100% -

Operational - 25% -

Grade 4 Grade 8 Grade 12 Mathematics Field Test 25% 25% 25% Bilingual 100% 100% - Puerto Rico 25% 25% -

Field Test 25% 25% 25%

Science Field Test - 25% - Bilingual - 100% -

Table 3.5 presents the estimated number of NAEP Constructed Responses to be scored in 2023.

Table 3.5 2023 NAEP Estimated Total Number of Responses to be Scored

Projected

2nd Scores

Projected trend scores

Total Scores

Component Responses

Projected Responses: 54,583 174,045 0 228,628

79,356 40,821 433,995

Age 9 Age 13 Age 17 Total Mathematics 156,241 Operational - 155,321 - 155,321 38,835 34,730 228,886 Bilingual - 920 - 920 920 - 1,840

Reading 17,804 Operational - 17,804 - 17,804 4,453 6,091 28,348

Field Test

Grade 4 Grade 8 Grade

12 Total

Mathematics 89,663 Field Test 18,500 18,500 14,500 51,500 12,875 - 64,375 Bilingual 80 80 - 160 160 - 320 Puerto Rico 19,503 18,500 - 38,003 9,503 - 47,506

Reading 38,500 Field Test 16,500 13,500 8,500 38,500 9,625 - 48,125

Science 11,610 Field Test - 11,500 - 11,500 2,875 - 14,375 Bilingual - 110 - 110 110 - 220

4 Training Set Assembly and Approval

This section describes how NAEP training sets are created, from the initial arrival of sample student responses through final training set approval.

4.1 Training Set Design

NAEP training sets may vary slightly among subject areas. Training set guidelines have also evolved somewhat over time so that slight variations exist between older and newer training sets.

For example, annotations are included in current NAEP training sets, but not necessarily in older training sets.

NAEP training sets generally include:

Cover sheet identifying the item Copy of the stimulus materials Copy of the item Item-specific scoring rubric, including Framework classification Anchor Set Practice Sets (generally two) Qualification Sets (generally two, if applicable)

Anchor sets include papers that show clear examples of each score category. The anchor papers will be identified as A-1, A-2, A-3, etc., and the paper’s score (e.g., 1, 2, 3). Papers within an anchor set will be ordered from the highest to the lowest score category.

Practice sets generally consist of one or two sets of 10 to 12 papers (some older trend items only have one practice set which may be a larger set if the item is more complex). The practice papers will be identified as P1-1, P1-2, P1-3, etc. in the first practice set and P2-1, P2-2, P2-3, etc. in the second practice set. Scores are not included on practice papers. Each set should cover (a) the range of scores on the scoring guide, (b) different levels of responses within a score level (i.e., high “2” and low “2”), (c) a mix of clear/solid and borderline papers and (d) examples of common student responses. Practice set papers are randomly ordered within the sets. Usually, the first practice set includes examples that may be more straightforward, while more challenging papers often appear in the second practice set.

Qualification sets (please refer to page 48) are two sets of 10 papers that cover (a) the range of scores on the scoring guide, (b) different levels of responses within a score for some content areas (e.g., high “2” and low “2”), (c) examples that reflect clear/solid examples of score categories and (d) examples of common student responses. The qualification papers will be identified as Q1-1, Q1-2, Q1-3, etc. in the first qualification set and Q2-1, Q2-2, Q2-3, etc. in the second qualification set. Trainers randomly order papers within each set during set assembly.

Annotations are explanatory comments that provide a clear reason, or reasons, why a response received a particular score. All initial training papers, including qualifying papers, must be annotated. These annotations provide crucial information to Trainers in preparing to use the materials and serve as support in defining scoring decisions both in the year they are written and in future years.

NAEP does not include condition code responses in training sets, because Scorers do not independently score non-blank condition codes. Separate sets of condition code samples may be used to train Trainers and Supervisors. Some complex items may include some additional background information provided by the Trainers to the Scorers, if applicable. More detailed information about training set assembly is available in Section 4 of the Trainer Handbook.

The 2023 NAEP assessment consisted entirely of returning items. Therefore, no new training sets are planned for 2023. As in any year, training sets may be updated as needed based upon specific item requirements.

4.2 Sample Selection

Sample Selection refers to the printing of sample current year student responses to be used for creating new training sets.

In 2005, Pearson began printing scanned images for sample responses (rather than photocopying student test books prior to scanning). Student responses are printed on high-speed industrial printers located at Pearson’s facility in Cedar Rapids, Iowa. Printing of these responses is random (based upon when student booklets arrive to Iowa City).