Attachment I - Task 5 Scoring QASP.pdf

Attachment I - Task 5 Scoring QASP

Quality Assurance Surveillance Plan Evaluation Plan

National Assessment of Educational Progress

NAEP Scoring

Introduction

This performance-based Quality Assurance Surveillance Plan (QASP) Evaluation Plan has been developed in accordance with Federal Acquisition Regulation (FAR) 12.208 and pursuant to the requirements identified in the Statement of Objectives (SOO). The QASP sets forth procedures and guidelines that the Department of Education (ED) will use in evaluating the technical performance of the contractor. A copy of this plan will be forwarded to the contractor so that the contractor shall be aware of the methods that the

Government will employ in evaluating performance on this contract, and to address any concerns that the contractor may have prior to initiating work.

Purpose of the QASP

The QASP is intended to accomplish the following:

• Define the roles and responsibilities of participating Government officials and outside experts.

• Define the key deliverables that will be assessed.

• Describe the rating elements and standards of performance against which the contractor’s performance will be assessed for each key deliverable.

• Describe the process of quality assurance assessment; and

• Provide copies of the quality assurance monitoring forms that will be used by the

Government in documenting and evaluating the contractor’s performance.

Roles and Responsibilities of Participating Government Officials and Experts

The following Government officials and/or experts will participate in assessing the quality of the contractor’s performance. Their roles and responsibilities are described as follows:

The Contracting Officer’s Representative (COR) is responsible for monitoring, assessing, recording, and reporting on the technical performance of the contractor on a day-to-day basis. The COR also is responsible for documentation, inspection, and evaluation of the contractor’s work performance.

Additional ED staff with technical knowledge and experience related to the planning, operations, and coordination of the NAEP Program may provide input and review deliverables. However, ED reserves the right to have only the COR assess each deliverable.

The Contract Specialist (CS) is responsible for the day-to-day monitoring of the contractor’s performance in the areas of contract compliance, contract administration, cost control, property control; reviewing the COR’s assessment of the contractor’s performance; and resolving all differences between the COR’s version and the contractor’s version. The CS may call upon the expertise of other Government individuals as required.

The Contracting Officer (CO) has the following procurement authorities:

• Sole authority for any decisions that produce an increase or decrease in the contract’s scope.

• Sole authority for any decision to be rendered under the “Disputes” clause.

• Sole authority to approve the substitution or replacement of the project manager or other key personnel.

• Sole authority to monitor and enforce Department of Labor promulgated labor requirements.

• Authority to arrange for and supervise quality assurance activities under this contract.

• Sole authority to approve the contractor’s quality control program.

• Signatory authority for the issuance of all modifications to the contract.

Annual Past Performance Evaluation

The Contractor’s overall performance shall be documented and measured through the annual past performance evaluation through the Contractor Performance

Assessment Reporting System (CPARS). Performance will be evaluated in the areas of:

• Quality of Product or Service

• Schedule

• Cost Control (if applicable)

• Business Relations

• Management of Key Personnel

• Utilization of Small Business

Quality Assurance Review Process

Even though the Government, through its COR, will monitor the contractor’s performance on a continuing basis, the volume of tasks performed by the contractor makes technical inspection of every task and step impractical. Therefore, ED will use an annual quality assurance review process to monitor the contractor’s performance under the award. Specifically, the COR will assess the contractor’s performance on the following four key deliverables:

1. Annual NAEP Report Card Submission

2. 5.1 Human Scoring and Monitoring

3. 5.2 Automated Scoring and Monitoring

4. 5.3 Conduct Targeted Special Studies

The contractor’s performance will be evaluated annually by the COR using the following adjectival ratings: Exceptional, Very Good, Satisfactory, Marginal, and Unsatisfactory.

Descriptions of these adjectival ratings are provided below:

CPARS Adjectival

Rating

Description

Exceptional Performance meets contractual requirements and exceeds many to the

Government's benefit. The element being assessed was accomplished with few minor problems for which corrective actions taken by the contractor were highly effective.

Very Good Very Good: Performance meets contractual requirements and exceeds some to the Government's benefit. The element being assessed was accomplished with some minor problems for which corrective actions taken by the contractor were effective.

Satisfactory Satisfactory: Performance meets contractual requirements. The element being assessed contains some minor problems for which corrective actions taken by the contractor appear or were satisfactory.

Marginal Marginal: Performance does not meet some contractual requirements.

The element being assessed reflects a serious problem for which the contractor has not yet identified corrective actions.

Unsatisfactory Unsatisfactory: Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The element being assessed contains a serious problem(s) for which the contractor's corrective actions appear or were ineffective.

Each key task/subtask shall be separately evaluated using the above scale.

ED will rate performance separately for each Key Outcome. For each evaluation period.

The contractor shall revise any key deliverable determined to be “unacceptable” at no additional cost to the Government. That is, the contractor shall not charge the Government for the additional work to revise unacceptable products, such as reports that do not meet

NCES Statistical Standards.

Each year ED will evaluate the contractor's performance by assessing the performance on key deliverables within the task. The work will be evaluated in terms of how well the contract requirements are satisfied; the extent to which the work performed follows the approach found in the contractor’s technical proposal; quality and clarity of documentation; and timeliness of key deliverables.

While quality is closely tied to the performance standards for activities and /or deliverable content, timeliness and cost are also important considerations in the assessment of contractor performance.

In the event of an excusable delay (as defined in FAR 52.249-14, Excusable Delays and

EDAR 3452.242-71, Notice to the Government of Delays), the Department and the contractor shall work together to modify the contract with regard to the due dates of the deliverables. If such an event were to occur that would require a modification to the due dates of the deliverables, the contractor's performance shall be measured by the date agreed upon in the modification.

Below are the evaluation periods (Dates may vary based on actual date of award, but will align with the period of performance):

1. Base Year: November 1, 2024 – October 31, 2025

2. Option Period I: November 1, 2025 – October 31, 2026

3. Option Period II: November 1, 2026 – October 31, 2027

4. Option Period III: November 1, 2027 – October 31, 2028

5. Option Period IV: November 1, 2028 – October 31, 2029

The specific task performance indicators, and primary surveillance method are described below:

Key Task/Subtask #1: – Annual NAEP Report Card

Performance Indicators: The Annual NAEP Report Card is the culmination of NAEP activities from item development to the operational administration of the assessment to students. Findings from the NAEP assessment are made available to the public through the release of the Nations Report Card. NCES is required to release findings from the mathematics and reading report cards six months after the end of the data collection; 12-

15 months are the average release times for all other NAEP reports. Each activity completed by the Coalition is critical to ensure NCES’s ability to successfully meet its obligations to report on student performance, culminating with the release of the NAEP reports. The contractors shall, as a Coalition, ensure the established deadlines are made for release of all NAEP reports in the defined contract period. Details of NAEP report card requirements for each contract year time period are included in Chart 1 below. The contractors shall provide weekly reports with issues/problems and requirements stated in the reporting task.

Primary Surveillance Method: CORs review of Nations Report Card

Evaluation Criteria:

Acceptable Performance

1. Materials prepared by the contractor (i.e., draft, and final publications) are of high quality, require no more than minimal revisions, and are consistent with ED’s guidance on content, accuracy, completeness, formatting, and publishing requirements.

2. Materials prepared by the contractor are complete and delivered on time to the

COR.

Key Task/Subtask #2 5.1 Human Scoring and Monitoring

Scoring activities are central to NAEP’s quality of measurement, ability to maintain trend, and to report reliably on what the Nation’s students know and can do. During the

2024 – 2029 contract, NCES will employ a combination of human and automated scoring practices. The testing schedule (subjects, grades and years assessed) is determined by the

National Assessment Governing Board (NAGB). The schedule for the use of human and the rate of implementation for automated scoring will be determined, by NCES and the scoring contractor. The initial expectation is that Reading will be the first subject scored using a combination of human and automate scoring, beginning in 2025. Scoring human scoring is the foundation of NAEP scoring and has been part of NAEP’s scoring and reporting systems since 1969. Scoring activities include, but are not limited to:

recruitment, hiring, training and qualifying of scoring staff, maintenance of a computer-based scoring platform, and systems to support distributed and/or in-person scoring, acquisition, preparation and presentation of student responses for scoring; creation, validation and maintenance of item-level training sets; preparing for and completion of scoring for all pilot, operational and special study assessments; real-time monitoring of scorer agreement and quality and other measures of distribution, central tendency, and progress; daily updates; data QC, preparation and delivery for analyses, scoring and reporting; data management; pre-and post-scoring meetings and attendance at meetings where scoring is part of the agenda.

Performance Indicators:

− Scoring debriefing and planning – completed by 11/1 the previous year.

− Scoring and training Handbooks/materials – on time, accurate, per reviews with

NCES and collaborating contractors, available, on time for training and scoring

− Training set preparation – selections, annotations and review with collaborating contractors are completed, on time, are accurate, and support on-time scoring.

− Computer-based scoring platform/system preparation, QC, and maintenance

− Distributed scoring working platform. If distributed scoring is used, the platform

(Teams, Zoom, Web-ex, etc.) is secure, and functions reliably, and does not impact staff and scorer ability to perform their work. Data is captured and stored accurately, and no data is lost.

− Trainer and Scorer preparation – if scoring is distributed, training includes adequate preparation and practice with the secure platform. If scoring is in-person, scoring sites are safe, secure, and support staffs’ ability to work comfortably and successfully and to maintain all materials, securely, at all times.

− Scoring “site” preparation, in-person or distributed – site supports scoring

(professional, sufficient space, quiet, supports technology, and communication

− Training scoring trainers and managers– completed, per agreed upon schedule, reflects best practices in Scoring and Training handbooks/materials.

− Hiring of scorers – meets contracted criteria, completed on time, site is 100% staffed with qualified, diverse groups of scorers and trainers, per staffing criteria and hiring patterns and trends.

− Scorer training – reflects best practices in Scoring and Training materials.

− Preparation and presentation of students’ responses for scoring – images are consistent with student’s responses, images and responses are transferred and presented with 100% accuracy, scoring system is confirmed, functioning accurately 6 weeks prior to the start of the first subject scoring, system functions accurately and does not inhibit scorer performance or capture of scores.

− Accurate capture of scorer’s scores – 100% of scorer’s entries are captured accurately.

− Real-time monitoring of scoring is conducted online, in the same systems as score capturing, and provides accurate reports of agreement, pacing, completion, score distribution, individual and group performance.

− Scorer agreement – Scorer agreement, per score point and subject agreement, meet or exceed correlations, Kappa levels, and match to trend sets, as specified in the 2024 Scoring Memo.

− Monitor and track scorer trends, at the item and subject levels, to the previous administration and over time.

− Daily scoring calls – Are conducted, as needed, based on item-level agreement, pacing, staffing, or other issues that arise that threaten score quality or completion.

− Completion, on time – Scoring is completed, on time, per the published scoring schedule agreed upon by NCES and collaborating contractors.

− Training set updates – Training sets are updated, by trainers, at the end of scoring, per the schedule set by NCES.

− Materials security – materials’ security is maintained, at all times, and there are no breeches in security during preparation or conduct of scoring.

Primary Surveillance Method: Activity completion is tracked, based on e-mail notification, scoring completing and accuracy are tracked by monitoring scoring reports and daily calls and logs.

Evaluation Criteria – Deadlines for completion are either specified on the contract, or are negotiated with NCES and other participating contractors. Quality indicators including levels of agreement, and quality for scorers are stipulated, based on the previous year’s

Scoring Memo.

1. Deadlines for materials production (handbooks and annotated training sets, site selection, platform and software preparation and contracting, hiring, training, scoring, and scoring wrap-up are met.

2. Scoring quality, agreement, (current year and trend) as well as Kappa meet or exceed levels detailed in the most recent administration year’s Scoring Handbook.

3. Materials prepared by the contractor (i.e., draft, and final publications) are of high quality, require no more than minimal revisions, and are consistent with ED’s guidance on content, formatting, and publishing requirements.

4. Materials prepared by the contractor are complete and delivered on time to the

Key Task/Subtask #3: - 5.2 Automated Scoring and Monitoring

NCES will use a system of scoring that combines human and AI-enabled automated scoring to score students’ responses. NCES will continue it transition to Automated

Scoring (AS) with its use of advanced natural language processing methods to score student responses with levels of reliability that are similar to human scoring, in certain subjects. As part of this work, the contractor shall build, train and test, and implement item-specific, predictive models of human scoring.

NCES places a premium on the maintenance of scoring trends and avoiding “drift”, as well as AI’s potential to inform improvements in item development, and contribute to improved reporting. NCES will work with the contractor to design a schedule that will roll out automated scoring, across the life of the contract, one subject at a time, beginning with Reading. The order and rate of this implementation will be informed by the contractor’s expertise and experience and will be shaped by the assessment schedule, budgets, and priorities.

NCES’s use of AI to score NAEP will be characterized by the following:

• The scoring system aligns with established standards and criteria.

• Over the course of the contract, the scoring system improves the efficiency and accuracy of the assessment process over and above human scoring, especially in the area of bias.

• Over the course of the contract, the system enhances the accessibility and fairness of scoring for diverse student populations.

• The AI systems integrate with existing assessment platform and technologies.

• Systems ensure the security of the data, students’ PII and NAEP’s assessments, and

• Meets and adheres to schedules for handoffs, quality checks and deliverables, program wide.

Performance Indicators:

• A kick-off meeting, at the beginning of each round of scoring, per subject, to provide documentation of methods and criteria for item selection, and model training. This documentation should be as transparent and interpretable by

NCES as possible and should not include thoroughly “black box” models.

• For each subject to be scored, successful selection and written justification of effective scoring engine(s) best suited to individual subjects and trained on corpora that minimize bias and maximize fairness. (include recent research literature as well as other applications of the engine)

• For each subject to be scored, prior to beginning scoring, a written description of textual feature extraction as well as the Large Language Model (LLM) or other Natural Language Processing (NLP) system that are being proposed to process raw written responses. The contractor must use a local version of an

LLM, if being used, that does not potentially comprise this sensitive student data, as opposed to an API that potentially sends examinee data to a third party.

• Detailed descriptions of model development processes for the entire scoring pipeline from text processing to final score prediction.

• Secure and accurate data retrieval and or transfer process via a secure pipeline to protect sensitive student data.

• Report analyses to demonstrate that models are, as far as possible, free of bias.

At the bare minimum demonstrate that the models are no more biased than the human scoring for each demographic group by demonstrating that their

Standardized Mean Difference (SMD), or other empirically justified metric chosen, is no more than 0.10 different than the human scores for that same demographic group. Demonstrate that the engine is unbiased across all major demographic groups.

• Following scoring, provide post hoc metrics to detail the accuracy and stability of the scores for each engine, by subject. This should include the use of such agreement metrics between the current engine predictions and the extant human scores: as Quadratic Weighted Kappa (QWK), exact and adjacent agreement, and the correlation between the new scores and the human gold standard scores.

Describe which items were not scoreable, why they were not scorable, and what was done instead.

• A post hoc interpretability review for each item such as the Local Interpretable

Model-Agnostic Explanation (LIME) analysis or Shapley Additive Explanations

(SHAP) analysis.

• Demonstrate that the engine is scoring consistently across the scoring period and there is no “drift”.

• Meets all deliverable deadlines for detailed plans, reports and analyses.

Primary Surveillance Method: The COR reviews the deliverables including agenda and meeting minutes from the Committee meetings, data on agreement and fairness from scoring, deliverable quality and deadlines, and data inclusion (items and student responses, including 3 categories for condition code papers).

1. Deadlines for all scoring-related activities, deliverables and handoffs are met.

2. Scoring quality, agreement meets or exceeds agreed upon levels. At subject and grade levels, within subject,

- Score agreement is within .1 of human-to-human agreement (at least .75 - .8)

(Williamson)

- For fairness, fewer than 5% more were flagged for dif than were flagged in Human-to-

Human comparison.

3.

4. Materials prepared by the contractor (i.e., draft, and final publications) are of high quality, require no more than minimal revisions, and are consistent with ED’s guidance on content, formatting, and publishing requirements.

5. Materials prepared by the contractor are complete and delivered on time to the

Key Task/Subtask #4: 5.3 Conduct Targeted Special Studies

In support of NCES’s commitment to ensuring that each part of the NAEP program works to maintain NAEP as the gold standard in large scale assessment and to support innovation in large scale assessment, the scoring contractor will develop a plan for a single special study or a series of special studies that target one of the contract’s primary areas of responsibility. For example, mitigating bias in automated scoring, or increasing the transparency and interpretability of AI decision making in automated scoring. The scoring contractor will also participate in any studies that are required of the NAEP

Coalition contractors.

Performance Indicators: The contractor will propose studies that advance the goals of

NAEP, are responsive to NAEP’s development, scoring and reporting needs and priorities, and that any timeline or budgetary constraints specified by NCES. Study proposals may be requested by NCES or be suggested by the contractor. Following the approval of the proposal, the contractor shall create performance work statements, timelines, detailed research study designs and analyses plans. Once these have been approved, the contractor will conduct and report on the study, as specified in performance work statement and timelines. The contractor will update the COR and any advisory groups, throughout the work, as stipulated.

Primary Surveillance Method: The COR reviews all interim documents, data collection plans, instruments and deliverables as agreed upon in the Performance Work Statement, study design and analyses plans.

Evaluation Criteria:

Successful proposals will demonstrate how NCES’s goals for NAEP administration, development, reporting and innovation will be met and will include clear guidelines for quality and evaluation of interim and final work products and reports.

Research design, instrument design, data collection, analyses and reporting will be psychometrically sound and will meet the criteria approved by NCES in the proposal design.

Work products and reports should be submitted on time and free of errors.

1. Materials prepared by the contractor, are complete, responsive and reflect the study design agreed upon by the contractor and NCES. (i.e., draft, and final publications and associated presentations) are accurate, of high quality, require no more than minimal revisions, and are consistent with ED’s guidance on content, formatting, and publishing requirements.

2. Materials prepared by the contractor are complete and delivered on time to the

Process of Quality Assurance Assessment/Award Fee Evaluation

In the event of an excusable delay (as defined in FAR 52.249-14), the Department and the contractor shall work together to modify the contract regarding the due dates of deliverables. If an event occurs that would require modifying deliverable due dates, the contractor’s performance, where applicable in this QASP, shall be measured by the date agreed upon in the modification.

In narrative form, the COR will substantiate all adjectival ratings other than Acceptable.

Performance at the “Acceptable” level is expected from the contractor. The COR will forward a report of average scores to the CS.