Attachment Q-2022 Scoring Memo 12.1.22.pdf

N A E P 2 0 2 2 S c o r i n g M e m o r a n d u m

NAEP 2022

Scoring Memorandum

Constructed Response Scoring of

2022 NAEP

Operational and Long-Term Trend Assessments

Prepared by Pearson for NCES

12-01-22 version

2022 NAEP Scoring Calendar

Activity Math Reading Math Reading Social Sciences

Operational LTT 9 Operational

Test Administration 1/24 - 3/31

1/10 - 3/18

1/24 - 3/31

Pearson Scoring Center

Pearson / ETS Train the Trainer I

2/24 - 2/25 3/1 - 3/2

N/A

2/24 - 2/25

ETS delivers rubrics to Pearson

Samples delivered for Training Set creation

Training set Assembly, Review, Approval

Standing Committee Reviews

N/A N/A

Train the Trainer II

4/5 - 4/6

4/5 - 4/6

Scoring Starts 4/11 3/28 3/28 6/27

Scoring Ends 5/13 5/20 4/1 3/31 7/29

MDPS Delivers Scored Data to DAR

5/20 5/27

4/8 4/7

8/5

ID delivers ROA sets to

MDPS

Table of Contents

2022 NAEP SCORING CALENDAR

1 INTRODUCTION TO NAEP SCORING

1.1 THE NAEP PROGRAM FOR 2022

1.1.1 NAEP Components

1.1.2 Long-Term Trend

1.2 CHANGES FOR 2022

1.3 ROLES AND RESPONSIBILITIES IN NAEP SCORING

1.4 OTHER RESOURCES

1.5 PEARSON SCORING SYSTEM -- OVERVIEW

1.5.1 Late Receipts

2 NAEP SCORING QUALITY ASSURANCE

2.1 DEBRIEFING AND PLANNING MEETINGS

2.1.1 Lessons Learned

2.2 NAEP SCORING STAFF QUALIFICATIONS AND HIRING

2.3 PLACEMENT TOOLS

2.4 MONITORING TREND

2.5 NAEP SCORING GUIDES (RUBRICS)

2.6 TRAINING SETS

2.7 MONITORING SCORER ACCURACY

3 OVERVIEW OF 2022 SCORING

3.1 STUDENT SAMPLE SIZES

3.2 NUMBER AND TYPES OF ITEMS

3.3 NUMBER OF STUDENT RESPONSES

4 TRAINING SET ASSEMBLY AND APPROVAL

4.1 TRAINING SET DESIGN

4.2 SAMPLE SELECTION

4.3 TRAINING SET ASSEMBLY

4.3.1 Pilot Training Set Assembly

4.3.2 New Operational Training Set Assembly

4.4 STANDING COMMITTEE REVIEW

5 PRELIMINARY SCORING ACTIVITIES

5.1 OE SPREADSHEET

5.1.1 SRDS (Student Response Definitions and Specifications)

5.2 TREND POOLS

5.2.1 Training Trend Pool

5.2.2 Monitoring Trend Pool

5.2.3 Creation of the Trend Pools

5.3 ITEM-LEVEL SCHEDULING

5.4 TRAIN THE TRAINER I

5.5 OPERATIONAL TREND ITEM PREPARATION

5.6 NEW OPERATIONAL ITEM PREPARATION

5.7 PILOT ITEM PREPARATION

5.8 TRAIN THE TRAINER II

6 SCORING PROCEDURES AND MONITORING

6.1 GENERAL SCORER TRAINING PROCEDURES

6.1.1 Scorer Orientation

6.1.2 Scorer Content Training

6.1.3 Scorer Qualification

6.1.4 Training Trend

6.2 TRAINER AND SUPERVISOR ROLES IN SCORING MONITORING

6.2.1 Backreading procedures

6.2.2 Backreading scores

6.3 WITHIN-YEAR INTERRATER RELIABILITY (SECOND SCORING)

6.4 CALIBRATION

6.5 SCORING PROCEDURES – PILOT ITEMS

6.6 SCORING PROCEDURES – NEW OPERATIONAL ITEMS

6.7 SCORING PROCEDURES – OPERATIONAL TREND ITEMS

6.8 SCORING PROCEDURES – BILINGUAL ITEMS

6.9 REPORTS USED TO MONITOR SCORING

6.10 GUIDELINES FOR ESTABLISHING MONITORING STATISTICS

6.11 TREND MONITORING

6.11.1 T-test

6.11.2 Cross-year percentage exact agreement (trend year reliability index)

6.11.3 Unweighted Kappa

6.11.4 Conditions for trend flagging

6.11.5 Binomial Test

6.11.6 Rescoring guidelines

6.12 CONDITION CODES IN MONITORING

6.13 DAILY CONFERENCE CALLS

6.14 A COMPREHENSIVE REVIEW OF KAPPA

7 POST-SCORING ACTIVITIES

7.1 DATA DELIVERY

7.2 PEARSON ROA

7.3 ETS MEMORANDA

APPENDIX A: GLOSSARY OF TERMS

APPENDIX B: FACILITIES AND STAFFING

APPENDIX C: HISTORY OF NAEP SCORING

APPENDIX D: EPEN REPORT DEFINITIONS

APPENDIX E: CONDITION CODES

APPENDIX F: UNIVERSAL PLACEHOLDERS

APPENDIX G: FEEDBACK LOOP

APPENDIX H: PROCESS FOR MACHINE SCORED ITEMS

2022 SCORING MEMO ADDENDUM

TRAINER HANDBOOK

2022 NAEP SCORING CALENDAR

1. OVERVIEW OF NAEP

1.1. BACKGROUND OF NAEP

1.2. NAEP ORGANIZATIONAL STRUCTURE

1.3. THE NAEP PROGRAM FOR 2022

1.3.1. 2022 Scoring Changes

1.4. LONG-TERM TREND

1.5. SECURITY AND CONFIDENTIALITY

1.5.1. Scoring from Home Security Considerations

1.6. NAEP WEB SITE

2. ROLES AND RESPONSIBILITIES

2.1. ETS CONTENT LEAD ROLES & RESPONSIBILITIES

2.2. PEARSON CONTENT SPECIALIST ROLES & RESPONSIBILITIES

2.3. TRAINER ROLES & RESPONSIBILITIES

2.3.1. NAEP Trainer

2.4. SUPERVISOR ROLES & RESPONSIBILITIES

2.4.1. NAEP Scoring Supervisor

2.5. SCORER ROLES & RESPONSIBILITIES

2.6. ADDITIONAL PEARSON STAFF

3. NAEP MATHEMATICS PROJECT OVERVIEW

3.1. OVERVIEW OF FRAMEWORK AND ASSESSMENT DESIGN

3.2. NAEP 2022 MATHEMATICS SCORING GUIDE OVERVIEW

GENERAL GUIDELINES FOR THE SCORING OF CONSTRUCTED RESPONSE QUESTIONS

3. NAEP READING PROJECT OVERVIEW

3.1. OVERVIEW OF FRAMEWORK AND ASSESSMENT DESIGN

3.2. NAEP 2022 READING SCORING GUIDE OVERVIEW

GENERAL GUIDELINES FOR THE SCORING OF CONSTRUCTED RESPONSE QUESTIONS

3. NAEP SOCIAL SCIENCES PROJECT OVERVIEW

3.1. OVERVIEW OF FRAMEWORK AND ASSESSMENT DESIGN

3.1.1. Civics

3.1.2. U.S. History

3.2. NAEP 2022 SOCIAL SCIENCES SCORING GUIDE OVERVIEW

GENERAL GUIDELINES FOR THE SCORING OF CONSTRUCTED RESPONSE QUESTIONS

4. TRAINING SET ASSEMBLY (PILOT & NEW OPERATIONAL ITEMS)

4.1. GUIDELINES FOR PRODUCING TRAINING SETS

4.2. TRAINING SET OVERVIEW AND MATERIALS

3.3. OVERVIEW OF SAMPLE SELECTION AND SET ASSEMBLY

4.4. DETAILED GUIDE FOR SET ASSEMBLY

4.5. REVIEW AND APPROVAL OF TRAINING SETS

5. NAEP ANNOTATION WRITING GUIDELINES

5.1. INTRODUCTION

5.2. CHARACTERISTICS OF ANNOTATIONS

6. ITEM SCHEDULE AND NOMENCLATURE

6.1. DBA ITEM NUMBERS

3.2. TRADITIONAL PAPER ITEM NUMBERS

6.2. COMBINED ITEMS

7. TREND ITEM PREPARATION

8. NAEP ITEM TRAINING

8.1. ADULT LEARNING

8.2. NAEP TRAINER TECHNIQUES FOR ITEM TRAINING

8.2.1. Composition of NAEP Training Sets

8.2.2. Qualifying

8.2.3. Overview of Item Training

8.2.4. Training Trend

8.2.5. Use of Flip Charts for Training or Scoring

8.3. THE FIRST DAY OF THE SCORING WINDOW

9. RECORDKEEPING DURING TRAINING AND SCORING

9.1. REQUIRED FORMS FOR TRACKING TRAINING AND SCORING ACTIVITIES

9.2. DOCUMENTING SCORING DECISIONS THROUGHOUT THE TRAINING AND SCORING PROCESS

9.3. DOCUMENTING CHANGES TO TRAINING SETS DURING TRAINING, SCORING, OR POST-SCORING

10. MONITORING SCORING

10.1. INTRODUCTION

10.2. CURRENT YEAR INTER-RATER RELIABILITY (SECOND SCORING)

10.3. GETTING STARTED WITH MONITORING

10.4. MONITORING SCORING: PRIMARY MONITORING TASKS

10.4.1. Backreading

10.4.2. Current-Year Reports

10.5. MONITORING SCORING: ADDITIONAL DAILY MONITORING TASKS

10.6. REVIEW AND PULLED PAPERS QUEUES

10.7. PSYCHOMETRIC REVIEW OF SCORING DATA AND PEARSON-ETS COMMUNICATIONS

11. MONITORING TREND SCORING

11.1. CROSS-YEAR (TREND) INTER-RATER RELIABILITY

11.2. PRINCIPLES OF TREND MONITORING

11.3. TREND BACKREADING

11.4. TREND REPORTS

11.5. MONITORING THE SCORING OF OPERATIONAL TREND ITEMS

11.6. TRMC (TREND RELIABILITY AND MEAN COMPARISON)

11.7. SPREADSHEET REPORTS: TRMC/FLAG STATUS REPORT, COMPLETION REPORT

11.8. TREND FLAG CRITERIA

12. FEEDBACK DURING SCORING

12.1. FEEDBACK TO INDIVIDUAL SCORERS

12.2. CALIBRATION SETS

12.3. TRAINING TREND AS A CALIBRATION ACTIVITY

13. FINAL RECORDKEEPING REQUIREMENTS

13.1. DOCUMENTING ITEM PERFORMANCE

13.2. ARCHIVING REQUIREMENTS

13.3. ROA REQUIREMENTS FOR PILOT ITEMS

13.4. ROA REQUIREMENTS FOR NEW (FIRST-YEAR) OPERATIONAL ITEMS

13.5. ROA REQUIREMENTS FOR OPERATIONAL TREND ITEMS

APPENDIX A. MECHANICS OF ANNOTATION WRITING

APPENDIX B. GLOSSARY OF TERMS

APPENDIX C. COMMON ABBREVIATIONS AND ACRONYMS

APPENDIX D. CONDITION CODES

APPENDIX E. ALERTS PROCESS

APPENDIX F. BILINGUAL SCORING

APPENDIX G. MONITORING SCORING OVERVIEW

APPENDIX H. GUIDELINES FOR POST-PILOT CHANGES

APPENDIX I. FORMS

Tables

TABLE 3.1 2022 NAEP TARGET SAMPLE SIZES

TABLE 3.2 2022 NAEP ITEMS

TABLE 3.3 2022 NAEP TARGET NUMBER OF RESPONSES PER ITEM

TABLE 3.4 2022 NAEP SECOND SCORING PERCENTAGE

TABLE 3.5 2022 NAEP ESTIMATED TOTAL NUMBER OF RESPONSES TO BE SCORED

TABLE 6.1 2022 NAEP IRR FLAGGING THRESHOLDS

1 Introduction to NAEP Scoring

This Scoring Memorandum is a collaborative document prepared by Pearson to provide NCES with documentation of the NAEP 2022 scoring process. ETS was the primary author for the Trend Monitoring section of this document, with some input from Pearson. While primarily geared toward NCES, other potential users of the Scoring Memorandum include Pearson, ETS, and AIR.

Significant 2022 scoring dates are presented in the Scoring Calendar located on the inside cover of this Memorandum. Several appendices appear at the end of this Memorandum. Please refer to page 78 for a Glossary of Terms relevant to the NAEP scoring process.

This Scoring Memorandum is a companion document to the NAEP Trainer Handbook. The Trainer Handbook is tailored specifically to support NAEP Trainers by providing a useful reference document that presents the scoring information most directly relevant to the job of NAEP Trainer. NCES is the target audience for the Scoring Memorandum, a main purpose of which is to provide documentation of the scoring process, statistical monitoring criteria, and Quality Assurance procedures. Though the primary audiences differ, both NCES and NAEP Trainers have complete access to both documents. Both the Trainer Handbook and Scoring Memo will be provided and used for training purposes. Since both documents reference each other frequently, for ease of access the Trainer Handbook has been appended to this Scoring Memorandum.

1.1 The NAEP Program for 2022

The NAEP assessment for 2022 includes the following assessment components that will be scored by trained and qualified professional Scorers:

Operational digitally-based assessments (DBA) in Mathematics and Reading at grades 4 and 8

Operational digitally-based assessments (DBA) in Civics and US History at grade 8

Operational LTT paper-based assessments (PBA) in Mathematics and Reading at age 9

1.1.1 NAEP Components

The NAEP 2022 scoring activity will include:

All main NAEP assessments will be digital, and the eNAEP delivery system will be used to administer the assessment on tablets. Technology-enhanced items (e.g., matching, in-line choice, fill-in the blank, zones, multiple selection multiple choice, grid/table selection, and items using interactive item components (IICs)), from the 2019 (mathematics, reading) and 2018 (civics, U.S. history) assessments will continue to be administered in the 2022 main NAEP assessments.

The NAEP Long-Term Trend (LTT) assessment will be administered to a sample of age

9 students. The age 17 assessment was originally scheduled to be administered in 2020, along with the corresponding assessments for ages 9 and 13. However, the school closures in response to Covid-19 made the postponement of the age 17 assessment necessary. In August 2021 it was decided to administer age 9 in 2022 in lieu of age 17.

1.1.2 Long-Term Trend

NAEP periodically administers a Long-Term Trend (LTT) assessment in mathematics and reading. For the NAEP LTT assessments, students are sampled by age, rather than by grade as for main NAEP assessments. The 2022 NAEP LTT assessment will be administered to students at age 9 (January through March, 2022).

Since its inception in 1969, NAEP has served the important function of measuring our nation’s educational progress by regularly administering various subject area assessments to nationally representative samples of students. The LTT reading assessment was first administered in 1971, and the LTT mathematics assessment in 1973. The LTT assessment measures student progress over time. In contrast to the main NAEP assessment that is updated regularly to reflect current educational content and assessment methodology, the LTT mathematics and reading assessments have remained essentially stable since first being administered.

Many items presented at different age levels are completely separate and unique, while in some cases the same items are presented to multiple age groups and are scored according to the same criteria. Some items are administered to students at ages 9 and 13, some items to students age 13 and 17, and some items to students in all three age groups.

In 2001 the National Assessment Governing Board reviewed LTT assessment content. At that time, the Governing Board suspended the LTT assessments in writing and science. Beginning in 2004, only reading and mathematics would continue to be assessed in Long-Term Trend. The Governing Board also recommended that outdated items be replaced. Currently, as items are gradually released, new items are developed to assess the same item types, using the same format at approximately the same level of difficulty as the released parent block at each age. As with the main NAEP assessment, all passages and items developed for LTT are reviewed by ETS assessment specialists, by editors, and by trained fairness reviewers to ensure that the materials are free from gender and racial-ethnic bias and inappropriate language stereotypes.

The NAEP LTT reading assessment blocks are 15 minutes in length, and each student completes a test booklet that contains three cognitive blocks and one block of background questions. The majority of items in the LTT reading assessment are four-option multiple-choice items, but some blocks include one extended constructed-response (ECR) item scored with either a 4- or 5-point rubric. These are the items scored at Pearson scoring sites during LTT. Texts are classified as Expository, Narrative, Document, or Other (for riddles, visuals, and sentence matches). Items are classified as Identify Main Idea, Locate Information Verbatim, Locate Information, or Inference.

Each LTT mathematics block is 15 minutes in length, and each student completes a test booklet with three blocks of mathematics questions, along with one block of background questions. Each block of mathematics questions contains both multiple-choice questions and short constructed-response (SCR) questions. These SCR questions are the items scored at Pearson scoring sites during LTT. Each constructed-response item requires the student to perform a computation or solve a problem and write the answer in the space provided. All items are scored dichotomously (i.e., correct or incorrect) to be consistent with the blocks in the current Long-Term Trend assessment.

1.2 Changes for 2022

Covid-19 presented challenges across multiple aspects of the 2022 NAEP assessment. Two impacts to scoring included extending the field assessment window and scoring from home.

Scoring of the 2022 NAEP assessments was originally intended to be done onsite at Pearson scoring centers. Traditional NAEP scoring sites include Mesa, AZ, Norfolk, VA, and Iowa City, IA. The ongoing challenges of Covid-19 required shifting the 2022 NAEP scoring activities to the synchronous scoring from home model. NAEP scoring in 2020 was completed successfully via the synchronous scoring from home model.

To ameliorate the uncertainties of student attendance in schools, NAEP’s 2022 field assessment window was extend until March 31st. The field administration historically ended the first week in March. This change required pushing the scoring window back approximately three weeks. The later scoring window created more staffing conflicts than usual with other projects.

The later window also separated LTT and operational scoring into two different windows. Per NCES’ request to complete LTT scoring as quickly as possible, LTT scoring in 2022 was scheduled as its own scoring activity two weeks before the main operational scoring window.

For 2022 scoring, NAEP training sets have been loaded to the ePEN scoring system. Practice and Qualification tally sheets previously completed on paper will instead be recorded electronically within the ePEN system.

1.3 Roles and Responsibilities in NAEP Scoring

This section addresses roles and responsibilities in NAEP Scoring (please refer to page 81).

The chart below shows NAEP’s onsite scoring staff.

Content Specialist

Plan and lead Operational projects

ETS

Content Lead

Develop and oversee content

Ensure staffing needs are met

Review and approve training sets

Oversee content Train Trainers and Supervisors

Trainers (Pearson or ETS)

Monitor content Backread and monitor reports Build training sets Provide feedback to Scorers Train and monitor Supervisors and Scorers

Supervisors (Pearson)

Backread and monitor reports Provide feedback to Scorers Maintain team records

Scorers

(Pearson) Score student responses

As the Materials, Distribution, Processing, and Scoring (MDPS) contractor, Pearson has the primary responsibility for NAEP constructed response scoring. Pearson receives all student test booklets from the field. Pearson scans and processes student booklets in preparation for loading the student constructed responses into ePEN. Pearson receives digitally based assessment (DBA) student response files from the field. Pearson extracts individual student responses to each DBA item from the larger student response file. Pearson prints a sample of student responses and delivers these to the various Pearson scoring sites and to ETS. From these samples, Pearson and ETS Trainers select example papers that will be used to build training sets. Trainers create annotations for the new training sets. Trainers train Scorers. Pearson and ETS staff at Pearson scoring centers complete all constructed response scoring using the ePEN system discussed in the next section. To facilitate efficient scoring and scoring monitoring, Pearson provides all scoring Supervisors and is responsible for training Scorers for all scoring of operational constructed response items. Pearson hires Scorers and monitors their performance. Pearson sends all data on scored constructed response items to ETS for analysis.

ETS staff members, trained in the various content areas, create scoring guides, and approve training sets created by Pearson Trainers. This is a collaborative review process where sets go back and forth between contractors. ETS may create some training sets for pilot items. ETS content personnel may train Scorers for the scoring of some pilot items. In addition, ETS staff members play a consulting role during the scoring of all constructed response items. ETS research scientists and data analysts are responsible for analyzing the data on all items. The ETS research scientists who direct NAEP analysis review scoring monitoring data on a daily basis during scoring. ETS planning and coordination staff facilitate daily scoring monitoring calls.

ETS consults with Pearson on content-related scoring decisions during scoring. ETS develops and updates all scoring guides; Pearson will not alter scoring guides. At the operational stage, Pearson and ETS discuss any implementation issues and ETS resolves those issues after considering implications for trend. Notes of clarification (“Trainer Notes”) may be added to supplement the scoring guides; however, scoring guides at the operational stage are not revised.

Given the complex interactions among contractors that are necessary for NAEP scoring to be successful, it is crucial that responsibilities and procedures be clear and explicit.

The American Institutes for Research (AIR) supports the National Center for Education Statistics (NCES) administration of the NAEP program, including assessment development and scoring.

Below are specific contractor responsibilities:

Pearson provides content and procedural training for Trainers provides training for Supervisors trains Scorers on all operational constructed response items builds training sets for new operational constructed response items provides Supervisors and Scorers assembles the Trend Pool (monitoring and training trend) develops and maintains the electronic scoring environment (ePEN) used for NAEP scans/processes multiple-choice items scans/processes constructed response items provides and equips the physical buildings for all scoring activities documents and maintains files for all scoring decisions made during the course of scoring generates reports and participates in daily scoring calls delivers a data file of all scores to ETS maintains electronic copies of all training sets, including item documentation forms provides electronic images of scored responses for use in reports provides Training Trend images to Trainers to assist in Trainer preparation creates operational scoring packets for the Report of Activities (ROA) archives ETS-provided pilot scoring packets for the ROA prepares Trainer Handbook, with input from ETS prepares Scoring Memorandum, with input from ETS plans and leads Scoring Debriefing and Scoring Planning meetings provides most Trainers and assembles sets for pilot items

Educational Testing Service (ETS) may train Scorers for some pilot constructed response items sets statistical specifications for valid and reliable scoring of NAEP items monitors scoring statistics and ensures integrity of scored data participates in daily scoring calls consults with Pearson Content Specialist on the scoring of constructed response items communicates to Pearson information and guidelines for building training sets builds training sets for pilot constructed response items reviews and approves all sets develops and updates all guides plans and leads Standing Committee meetings monitors scoring, and provides content decisions when needed reviews all scoring documentation provides Pearson with pilot ROA sets for ETS-led items tracks training set assembly and coordinates daily scoring calls participates in the Scoring Debriefing and Scoring Planning meetings

1.4 Other Resources

All of the following sources provide additional documentation that impacts the NAEP scoring effort.

2022 Trainer Handbook – NAEP’s Trainer Handbook and Scoring Memo are companion documents. The Trainer Handbook also documents the scoring process, but is written specifically for NAEP Trainers, while the Scoring Memo is written for NCES. It is also incorporated into this Scoring Memo. The Trainer Handbook folder address is https://www.naepims.org/sites/ims/S/TrainerHandbook/2022

Frameworks – NAEP utilizes separate Frameworks for each subject area in Main NAEP.

The NAEP Framework for all subjects are available on the National Assessment Governing Board’s website, https://www.nagb.gov/naep/frameworks-overview.html.

More Framework information is available in Section 3 of the Trainer Handbook.

IMS – NAEP’s Integrated Management System is a secure website used for communication and sharing of documents among various NAEP contractors, and with NCES. The main website is accessible at https://www.naepims.org. The 2022 Scoring Memo subfolder on the IMS contains both an electronic version of this Scoring Memo, as well as numerous related documents. Documents available in this folder include the most up-to-date versions of the Scoring Calendar and Open Ended (OE) Spreadsheet (please refer to page 37), and other working documents related to the scoring process. The Scoring Memo folder address is https://www.naepims.org/sites/ims/S/ScoringMemo/2022 and includes:

o Scoring Calendar o Scoring Memo (with appended Trainer Handbook) o OE Spreadsheet

Other scoring folders include:

o Daily calls checklist o Daily scoring reports o Conference call minutes o Training set tracking

1.5 Pearson Scoring System -- Overview

This section addresses the Pearson scoring system. Pearson utilizes the Electronic Performance Evaluation Network (ePEN) to score student responses.

Pearson’s scoring network couples technology and software with secure network communications to transmit electronic images or files of student responses to trained professional Scorers. Pearson scans student test booklets at Pearson’s Iowa City facility. The original student test booklets remain secure at Pearson’s facility. Student responses are converted into an electronic format, and secure network connections transfer the images or files to the appropriate Pearson site. ePEN distributes the electronic images or files to individual computer workstations of the trained and qualified Scorers. All scores assigned are automatically captured and available for review by additional qualified scoring personnel. Pearson’s scoring network integrates multiple processes (routing work, scoring responses, monitoring quality, and tracking progress and workflow) into a single, efficient system.

The Pearson scoring network electronically manages multiple aspects of the scoring process, including project setup, all scoring monitoring activities, and reporting of data. The ePEN project setup process allows customization of many different variables to accommodate project needs, such as trend for NAEP. In addition, pre-defined scoring rules describe scoring parameters and how responses are routed through the system. These rules dictate the required reliability rate and an appropriate response rate to Scorers. ePEN also provides a variety of processes and tools needed to manage the quality of scoring, as dictated by Pearson’s standards for quality. Internal quality assurance standards are set and met for every Pearson scoring project.

NAEP uses trend scoring to compare the consistency of scoring over time (i.e., cross-year interrater reliability). During trend scoring, the ePEN system allows for the presentation of a pool of trend responses to Scorers and offers the ability to generate reports to evaluate scoring consistency. The scoring network’s data management capabilities allow the capture and reporting of scores to occur accurately and quickly. These on-demand scoring reports include daily, cumulative, Scorer specific and team specific information regarding both the quality and quantity of scoring activities. The reports provide information that enables supervisory staff to monitor scoring continually and accurately (please refer to page 89).

The ePEN system delivers DBA and paper responses to Scorers in a similar fashion. DBA student responses are received from the field as text files or images rather than papers booklets.

However, just like with paper student booklets: individual item responses are extracted from the larger student record and presented to Scorers via the same ePEN system.

1.5.1 Late Receipts

A decision must be made for how responses that arrive late from the field will be scored. The late responses could be routed to the team of Scorers to score, or they can be scored by Trainers or Supervisors. The benefit of a traditional team of Scorers is the alignment with the standardized NAEP scoring process, which is in place as a quality measure to ensure and protect trend.

However, given the timing of the late arriving responses, a new team could potentially have to be created and trained which is not practical for a small number of late responses. The benefit of having Trainers/Supervisors score the late responses is that they ideally assign the best and most accurate score given their expertise. The risk, however, of moving away from a standardized scoring process by having Trainers/Supervisors score the late responses is the possible introduction of bias, which threatens trend.

To balance the potential risks with the practical implications for scoring late responses, the following guideline should be followed: If the total number of late responses per item is greater than or equal to 50, then the responses must be routed to a full team of Scorers. If the total number of late responses per item is less than 50, then the Trainers or Supervisors may score the responses.

2 NAEP Scoring Quality Assurance

NAEP takes numerous steps to maximize scoring quality. These include procedural steps before and during the scoring effort as well as rigorous monitoring and review during and after scoring.

Debriefing and Planning Meetings NAEP Scoring Staff Qualifications and Hiring Placement Tools Monitoring Trend Training Sets NAEP Scoring Guides (Rubrics) Monitoring Scorer Accuracy

2.1 Debriefing and Planning Meetings

After the conclusion of each year’s scoring, NAEP conducts a Scoring Debriefing Meeting.

NCES and NAEP staff from all contractors involved with the scoring process meet to review lessons learned from the just-concluded scoring season and to evaluate how those lessons may best be applied to the improvement of future NAEP scoring. Agendas, minutes, and other related documents are available on IMS at https://www.naepims.org/sites/ims/S/ScoringDebriefing/

In preparation for the upcoming scoring, NAEP conducts a Scoring Planning meeting. NCES and staff from relevant NAEP contractors meet to plan the schedule and procedures for the next year’s scoring. The 2022 Scoring Planning meeting was held December 14th, 2021. Agendas, minutes, and other related documents are available on IMS at https://www.naepims.org/sites/ims/S/PlanningMeetings/2022

2.1.1 Lessons Learned

Some lessons learned from previous Scoring Debriefing / Scoring Planning meetings include:

Scoring staffing should include more Trainers than scoring teams. This allows Trainers occasional time away from the scoring team to work on recordkeeping and/or prepare for their next items.

MS Teams sub-rooms may be used for 1-on-1 or smaller group discussions.

2.2 NAEP Scoring Staff Qualifications and Hiring

NAEP Trainers, Supervisors and Scorers must all possess college degrees. Additional qualifications are noted below.

General Qualifications for Trainers

A 4-year college degree is required, and a content-related degree is preferred.

Previous experience as a scoring Supervisor or extensive experience as a Scorer is preferred.

Teaching experience is desirable.

Scoring experience, teaching experience, and/or past training/leadership experience carry significant weight in the selection of potential Trainers.

Previous NAEP scoring experience is preferred.

The ability to work efficiently with the ePEN scoring system, including backreading, monitoring current year reliabilities, and administering trend, as needed, is preferred.

General Qualifications for Pearson Scoring Supervisors

A 4-year college degree is required, and a content-related degree is preferred.

Previous experience as a scoring Supervisor or extensive experience as a Scorer is preferred.

Teaching experience is desirable.

Previous NAEP scoring experience is preferred.

The ability to work efficiently with the ePEN scoring system, including backreading, monitoring current year reliabilities, and administering trend, as needed, is preferred.

General Qualifications for Bilingual Trainers, Supervisors, and Scorers

A 4-year college degree is required.

Scoring experience, including the use of the ePEN system, is preferred.

In addition to the other qualifications listed in this section, bilingual Trainers, Supervisors, and Scorers must demonstrate Spanish language proficiency via a bilingual Placement Tool.

Subject-Specific Qualifications for Scorers ETS content leads have developed written NAEP Scorer placement tools for most operational subject areas. The Pearson scoring center staff will maintain a database of Scorer demographics, experience, and qualifications. Information collected will help place Scorers most appropriately.

Qualifications for NAEP Mathematics Scoring

A 4-year college degree is required. (An education degree in elementary or secondary mathematics education or a mathematics-related discipline, such as physical sciences or statistics, is preferred.)

Scorers must possess sufficient knowledge of the content described in the NAEP mathematics Framework to permit scoring of assessment items.

Scoring experience, including use of the ePEN system, is preferred.

Scorers must meet all item-level qualifying standards.

Qualifications for NAEP Reading Scoring

A 4-year college degree is required. (An education degree with English language arts, literature, reading, or writing concentration is preferred.)

Scorers must possess sufficient knowledge of the content described in the NAEP reading Framework to permit scoring of assessment items.

Scoring experience, including the use of the ePEN system, is preferred.

Scorers must meet all item-level qualifying standards.

Qualifications for NAEP Civics and US History Scoring

A 4-year college degree is required. (An education degree with a concentration in the humanities is preferred.)

Scorers must possess sufficient knowledge of the content described in the NAEP civics and US history Frameworks to permit scoring of assessment items.

Scoring experience, including the use of the ePEN system, is preferred.

Scorers must meet all item-level qualifying standards.

2.3 Placement Tools

Pearson, as the contractor responsible for hiring Scorers, will administer placement tools developed by ETS for all operational subject areas to determine the best subject/grade match for each Supervisor and Scorer hired. These instruments must be practical to administer and score for a large group. The placement tools are available in the Scoring Memo folder on the IMS.

The NAEP placement tools are used to identify Supervisors and Scorers with an appropriate level of content knowledge, and are analyzed in conjunction with other qualifications (education/experience) so that individuals can be appropriately placed on teams (e.g., based on grade-level knowledge and/or performance). The placement tools provide an important piece of information to aid in these placement decisions.

For the purposes of hiring and placement, Pearson uses multiple measures; placement tools are one measure of potential Scorer performance. These evaluation instruments are one of many criteria used for assigning individuals to scoring teams within the constraints imposed by the need to hire a large number of people in a relatively short time. In addition, Scorers meet qualifying standards before scoring specific items (refer to page 47) and meet ongoing expectations for performance based on Supervisor backreading and monitoring.

2.4 Monitoring Trend

A defining feature of NAEP is its ability to provide student achievement data that can be used to track changes in achievement levels over time. For this to occur, assessment results must be placed on an equivalent scale every time the assessment is administered. Psychometric procedures are applied to create this equivalency, with these procedures depending on a subset of assessment items being repeated across years. Because constructed response items are scored by human beings rather than machines (as multiple-choice items are), the scoring of these items is monitored carefully to ensure that scoring remains consistent across administrations.

Procedures for scoring trend items have improved over time. In past years, prior-year, or “trend” papers, were clustered together in a group. Scoring trend papers was consequently an activity distinct from scoring current-year papers, and Trainers, Supervisors, and Scorers could not help but be aware that they were engaged in an activity that differed from scoring current-year papers.

It was difficult to ensure that scoring practices did not shift in some ways when teams shifted from current-year papers to trend papers. The interspersed trend scoring model implemented in 2006 has improved the monitoring system by interspersing trend responses among the current-year responses rendering the prior-year responses indistinguishable from current-year responses to Scorers (please refer to page 38). This change has greatly improved the validity of the measure of cross-year scoring consistency.

2.5 NAEP Scoring Guides (Rubrics)

Scoring guides are the starting point for training sets. NAEP uses customized scoring guides to govern the scoring of constructed response items. The scoring guides are designed to ensure that Scorers follow a single standard and that scores are assigned consistently and fairly.

Scoring guides for constructed response items are developed with the items and undergo the same rigorous review process as the items. Only by keeping the two synchronized can item developers know that both item and scoring guide are measuring the intended objective. A typical scoring guide consists of a list of the different score categories for a particular item with descriptions of the criteria responses must meet to receive a particular score. NAEP also uses diagnostic codes to track response types within score levels for some items. Depending on the subject and the item, scoring guides may also contain a sample correct response or a list of credited responses if there is more than one possible answer.

Scoring guides receive external reviews just as items do. The subject area committee reviews are important because committee members are expert in both content and student cognitive development. Thus they contribute to the accuracy of the credited responses and scoring categories. Additional important comments emerge from the state item review and reviews by both NCES and the Governing Board. Another critical review occurs after the pilot test administration but prior to scoring, when the Standing Committee convenes to review the scoring guides and training materials in light of actual student responses. Here, committee members use their expertise to help developers make essential distinctions among the sample responses and fine tune final rubrics.

During the review of sample student responses obtained for new operational Items, Pearson and ETS work collaboratively, with Pearson Trainers initially sorting student responses into score categories based on original versions of the scoring guides and note any issues and observations for discussion with ETS staff. In consultation with ETS staff and members of the Standing Committees, scoring guides will be refined and approved by NCES. Training sets will be assembled and then given final approval by ETS item developers. ETS item developers will approve any revisions to scoring guides and training materials.

Additional subject-specific scoring guide information is available in Section 3 of each NAEP 2022 Trainer Handbook.

2.6 Training Sets

Individual training sets are created for every NAEP constructed response item. Training sets are first developed prior to the first operational scoring of each new item. Once created, the same training set is used from year to year throughout the item’s operational life. However, in some cases, new and variant student responses may be appended to the original training set to help further define and clarify the intent of the scoring guide. Training Scorers with the same training set each time an item is used is an integral part of NAEP’s process for maintaining consistent cross-year scoring quality. NAEP training sets go through a significant review and approval process to assure they are of the highest quality. New operational sets are assembled by Pearson

Trainers and reviewed by ETS item developers in a back-and-forth process designed to maximize the quality of new sets (please refer to page 34).

2.7 Monitoring Scorer Accuracy

Careful monitoring of scoring is a hallmark of NAEP. Monitoring is conducted at various levels:

by Trainers and Supervisors at the team level, by ETS psychometricians, by Pearson and ETS content staff, by Pearson program staff, and by NCES and AIR. Both backreading and second scoring are important monitoring tools (please refer to page 45).

3 Overview of 2022 Scoring

3.1 Student Sample Sizes

The total number of constructed responses to be scored is a function of the target sample size for each assessment. Presented below is a breakdown of the sample sizes by subject and grade, according to the NAEP 2022 Design Memo available here:

https://www.naepims.org/sites/ims/DA/Design/AnnualDesignPlans

Table 3.1 2022 NAEP Target Sample Sizes

Component Students

Total Students Assessed: 253,485 252,760 0 506,245

Main NAEP

Grade 4 Grade 8 Grade 12 Total Mathematics 239,850 Operational 116,850 116,850 - 233,700 Bilingual 100 50 - 150 Puerto Rico 3,000 3,000 - 6,000

Reading 233,700 Operational 116,850 116,850 - 233,700

Social Sciences 16,010 Civics - 8,000 - 8,000 Civics Bilingual - 5 - 5 U.S. History - 8,000 - 8,000 U.S. History Bilingual - 5 - 5

NAEP LTT

Age 9 Age 13 Age 17 Total Mathematics 8,365 Operational 8,320 - - 8,320 Bilingual 45 - - 45

Reading 8,320 Operational 8,320 - - 8,320

3.2 Number and Types of Items

Tables 3.2 and 3.5 present information about the number of distinct ePEN items to be scored in 2022.

In ePEN scoring, distinct items are those which require a Professional Scorer to assign a distinct score, and for which separate statistics are reported. Individual parts of multi-part items, cross-grade items, and bilingual items, as well as Spanish-language items used in Puerto Rico, are all examples of item extensions that are considered distinct items within ePEN.

The number of distinct items from this scoring perspective may differ from the number of items as considered from an Item Development or a Data Analysis perspective, since these various extensions may, or may not, be tallied as separate items during development or analysis.

Each ePEN item (please refer to page 158) is identified as a separate row on both the OE Spreadsheet and the daily Completion Report, as illustrated in the hypothetical item numbers below.

Single item:

Subject Grade Item ID

Item Description Score Point Levels

Reading 8 22171R5_03 Big Cheese 1,2,3

Cross-grade items:

Reading 8 221923R6G08_04 Red Sweater 1,2,3,4 Reading 12 221923R6G12_04 Red Sweater 1,2,3,4

Multi-part items:

Subject Grade Item ID Part Item Description Score Point Levels Math 4 22191M11_16 Part A Number of seats at table 1,2 Math 4 22191M11_16 Part B Number of seats at table 1,2B,2A,3

Bilingual / Spanish items:

Math 4 22171M6_03 Town council members 1,2,3 Math 4 22171M6_03EN Town council members 1,2,3 Math 4 22171M6_03SP Town council members 1,2,3 Math 4 22171M6_03S Town council members 1,2,3

Table 3.2 presents the number of items broken down by multiple choice vs. constructed-response and score levels for constructed-response items. It also presents the numbers of multi-part items and numbers of items utilizing expanded scoring guides. Expanded scoring guides track different response types within score levels (please refer to Section 3 of the Trainer Handbook).

Table 3.2 2022 NAEP Items

NAEP Component

Se le ct ed

R es po ns e Constructed-Response

Items Total

CR

Items

Multi-part

Items Ex pa nd ed

Ite m s

Total Items

Number of Score Levels 2 3 4 5

Grand Totals: 1,562 240 281 44 1 566 82 159 2,128

Mathematics

Operational Grade 4 188 36 32 - - 68 19 41 256 Grade 8 189 33 30 1 - 64 19 28 253

Bilingual Grade 4 57 11 7 - - 18 6 11 75 Grade 8 55 7 11 - - 18 4 13 73

Puerto Rico Grade 4 172 33 24 - - 57 13 35 229 Grade 8 173 30 26 1 - 57 17 24 230

Total 834 150 130 2 0 282 78 152 1,116

Reading

Operational Grade 4 73 5 22 9 - 36 - - 109 Grade 8 84 7 37 13 - 57 - - 141

Total 157 12 59 22 0 93 0 0 250

Social Sciences Civics Grade 8 131 4 31 5 - 40 4 5 171

Civics Bilingual Grade 8 43 - 11 - - 11 - 1 54 U.S. History Grade 8 143 2 40 11 - 53 - 1 196 U.S. History

Bilingual Grade 8 31 - 10 2 - 12 - - 43 Total 348 6 92 18 0 116 4 7 464

LTT Mathematics

Operational Age 9 99 36 - - - 36 - - 135 Bilingual Age 9 49 36 - - - 36 - - 85

Total 148 72 0 0 0 72 0 0 220

LTT Reading Operational Age 9 75 - - 2 1 3 - - 78

Total 75 0 0 2 1 3 0 0 78

3.3 Number of Student Responses

Table 3.3 presents the expected number of 2022 NAEP student responses per item based upon the target sample sizes. The number of student responses per item is determined by dividing the student sample size by the number of student book types and multiplying by the number of times each block appears.

Table 3.3 2022 NAEP Target Number of Responses per Item1

Component Responses

Grade 4 Grade 8 Grade 12 Mathematics Operational 16,693 16,693 - Bilingual 100 50 - Puerto Rico 462 462 -

Operational 21,245 16,693 -

Civics - 2,286 - Civics Bilingual - 5 - U.S. History - 1,778 - U.S. History Bilingual - 5 -

Age 9 Age 13 Age 17 Mathematics Operational 4,160 - - Bilingual 45 - -

Operational 2,800 - -

1 These figures represent the target sample sizes. Actual receipts from the field are usually slightly higher than these targets. DBA estimates are very speculative.

Table 3.4 presents the second scoring percentage for 2022 NAEP scoring. Second scoring refers to scoring of the same student response by two different Scorers for the purpose of monitoring scoring consistency.

Table 3.4 2022 NAEP Second Scoring Percentage

Component Percentage

Grade 4 Grade 8 Grade 12 Mathematics Operational 10% 10% - Bilingual 100% 100% - Puerto Rico 25% 25% -

Operational 10% 10% -

Civics - 25% - Civics Bilingual - 100% - U.S. History - 25% - U.S. History Bilingual - 100% -

Age 9 Age 13 Age 17 Mathematics Operational 25% - - Bilingual 100% - -

Operational 25% - -

Table 3.5 presents the estimated number of NAEP Constructed Responses to be scored in 2022.

Table 3.5 2022 NAEP Estimated Total Number of Responses to be Scored

Projected

2nd Scores

Projected trend scores

Total Scores

Component Responses

Projected Responses: 2,087,858 2,232,876 0 4,320,734

495,605 507,600 5,323,939

Grade 4 Grade 8 Grade 12 Total Mathematics 2,258,844 Operational 1,135,124 1,068,352 - 2,203,476 220,308 211,200 2,634,984 Bilingual 1,800 900 - 2,700 2,700 - 5,400 Puerto Rico 26,334 26,334 - 52,668 13,224 22,800 88,692

Reading 1,716,321 Operational 764,820 951,501 - 1,716,321 171,633 148,800 2,036,754

Social Sciences 185,789 Civics - 91,440 - 91,440 22,880 40,000 154,320 Civics Bilingual - 55 - 55 55 - 110 U.S. History - 94,234 - 94,234 23,585 53,000 170,819 U.S. History Bilingual - 60 - 60

60 - 120

Age 9 Age 13 Age 17 Total Mathematics 151,380 Operational 149,760 - - 149,760 37,440 28,800 216,000 Bilingual 1,620 - - 1,620 1,620 - 3,240

Reading 8,400 Operational 8,400 - - 8,400 2,100 3,000 13,500

4 Training Set Assembly and Approval

This section describes how NAEP training sets are created, from the initial arrival of sample student responses through final training set approval.

4.1 Training Set Design

NAEP training sets may vary slightly among subject areas. Training set guidelines have also evolved somewhat over time so that slight variations exist between older and newer training sets.

For example, annotations are included in current NAEP training sets, but not necessarily in older training sets.

NAEP training sets generally include:

Cover sheet identifying the item Copy of the stimulus materials Copy of the item Item-specific scoring rubric, including Framework classification Anchor Set Practice Sets (generally two) Qualification Sets (generally two, if applicable)

Anchor sets include papers that show clear examples of each score category. The anchor papers will be identified as A-1, A-2, A-3, etc., and the paper’s score (e.g., 1, 2, 3). Papers within an anchor set will be ordered from the highest to the lowest score category.

Practice sets generally consist of one or two sets of 10 to 12 papers (some older trend items only have one practice set which may be a larger set if the item is more complex). The practice papers will be identified as P1-1, P1-2, P1-3, etc. in the first practice set and P2-1, P2-2, P2-3, etc. in the second practice set. Scores are not included on practice papers. Each set should cover (a) the range of scores on the scoring guide, (b) different levels of responses within a score level (i.e., high “2” and low “2”), (c) a mix of clear/solid and borderline papers and (d) examples of common student responses. Practice set papers are randomly ordered within the sets. Usually, the first practice set includes examples that may be more straightforward, while more challenging papers often appear in the second practice set.

Qualification sets (please refer to page 47) are two sets of 10 papers that cover (a) the range of scores on the scoring guide, (b) different levels of responses within a score for some content areas (e.g., high “2” and low “2”), (c) examples that reflect clear/solid examples of score categories and (d) examples of common student responses. The qualification papers will be identified as Q1-1, Q1-2, Q1-3, etc. in the first qualification set and Q2-1, Q2-2, Q2-3, etc. in the second qualification set. Trainers randomly order papers within each set during set assembly.

Annotations are explanatory comments that provide a clear reason, or reasons, why a response received a particular score. All initial training papers, including qualifying papers, must be annotated. These annotations provide crucial information to Trainers in preparing to use the materials and serve as support in defining scoring decisions both in the year they are written and in future years.

NAEP does not include condition code responses in training sets, because Scorers do not independently score non-blank condition codes. Separate sets of condition code samples may be used to train Trainers and Supervisors. Some complex items may include some additional background information provided by the Trainers to the Scorers, if applicable. More detailed information about training set assembly is available in Section 4 of the Trainer Handbook.

The 2022 NAEP assessment consisted entirely of returning items. Therefore, no new training sets are planned for 2022. As in any year, training sets may be updated as needed based upon specific item requirements.

4.2 Sample Selection

Sample Selection refers to the printing of sample current year student responses to be used for creating new training sets.

In 2005, Pearson began printing scanned images for sample responses (rather than photocopying student test books prior to scanning). Student responses are printed on high-speed industrial printers located at Pearson’s facility in Cedar Rapids, Iowa. Printing of these responses is random (based upon when student booklets arrive to Iowa City). Responses sent to the Trainers should be representative of each score level (excluding blanks). Two hundred non-blank sample responses are requested for each constructed response item. In situations when all score levels are not represented in the initial pull, additional responses are pulled. An attempt is made to find responses that represent all possible score levels for an item. On rare occasions it is not possible to find sample responses at every score level (this may occur when items have an unusually large number of different score categories or an extremely small n-count). Pearson provides sample student responses to trainers in pdf format. Sample responses are provided prior to the start of Train the Trainer.

After receiving the initial 200 responses per item, Trainers may occasionally want to request additional responses for use in building training sets. When requested, these additional responses are provided to the Trainers electronically (in pdf format). The Trainers print only those specific responses needed.

The DBA items do not require scanning. Student responses for DBA items are printed and…