Attachment H - Task 4 ACD QASP.pdf

Attachment H - Task 4 ACD QASP

Quality Assurance Surveillance Plan/ Evaluation Plan

National Assessment of Educational Progress

Assessment Content Development (ACD)

Introduction

This performance-based Quality Assurance Surveillance Plan (QASP) Evaluation Plan has been developed in accordance with Federal Acquisition Regulation (FAR) 12.208 and pursuant to the requirements identified in the Statement of Objectives (SOO). The QASP sets forth procedures and guidelines that the Department of Education (ED) will use in evaluating the technical performance of the contractor. A copy of this plan will be forwarded to the contractor so that the contractor shall be aware of the methods that the Government will employ in evaluating performance on this contract, and to address any concerns that the contractor may have prior to initiating work.

Purpose of the QASP

The QASP is intended to accomplish the following:

• Define the roles and responsibilities of participating Government officials and outside experts;

• Define the key deliverables that will be assessed;

• Describe the rating elements and standards of performance against which the contractor’s performance will be assessed for each key deliverable;

• Describe the process of quality assurance assessment; and

• Provide copies of the quality assurance monitoring forms that will be used by the

Government in documenting and evaluating the contractor’s performance.

Roles and Responsibilities of Participating Government Officials and Experts

The following Government officials and/or experts will participate in assessing the quality of the contractor’s performance. Their roles and responsibilities are described as follows:

The Contracting Officer’s Representative (COR) is responsible for monitoring, assessing, recording, and reporting on the technical performance of the contractor on a day-to-day basis. The

COR also is responsible for documentation, inspection and evaluation of the contractor’s work performance.

Additional ED staff with technical knowledge and experience related to the development of assessment items/tasks and survey instruments may provide input and review deliverables.

However, ED reserves the right to have only the COR assess each deliverable.

The Contract Specialist (CS) is responsible for the day-to-day monitoring of the contractor’s performance in the areas of contract compliance, contract administration, cost control, property control; reviewing the COR’s assessment of the contractor’s performance; and resolving all differences between the COR’s version and the contractor’s version. The CS may call upon the expertise of other Government individuals as required.

The Contracting Officer (CO) has the following procurement authorities:

• Sole authority for any decisions that produce an increase or decrease in the contract’s scope;

• Sole authority for any decision to be rendered under the “Disputes” clause;

• Sole authority to approve the substitution or replacement of the project manager or other key personnel;

• Sole authority to monitor and enforce Department of Labor promulgated labor requirements;

• Authority to arrange for and supervise quality assurance activities under this contract;

• Sole authority to approve the contractor’s quality control program;

• Signatory authority for the issuance of all modifications to the contract.

Annual Past Performance Evaluation

The Contractor’s overall performance shall be documented and measured through the annual past performance evaluation through the Contractor Performance Assessment Reporting System

(CPARS). Performance will be evaluated in the areas of:

• Quality of Product or Service

• Schedule

• Cost Control (if applicable)

• Business Relations

• Management of Key Personnel

• Utilization of Small Business

Quality Assurance Review Process

The NAEP administers the core grade 4 and 8 (reading and mathematics) assessments every two years and the grade 12 core (reading and mathematics) assessments every four years while the other content area assessments (e.g., science, writing, civics, geography, U.S history, etc.) every four years. Even though the Government, through the ACD COR, will monitor the contractor’s performance on a continuing basis, the volume of tasks performed by the contractor makes technical inspection of every task and step impractical. Therefore, ED will use an annual quality assurance review process to monitor the contractor’s performance under the award. Specifically, the COR will assess the contractor’s performance on the deliverables from the following key tasks:

1. Manage Assessment Content Development

2. Develop and Maintain Content for the Cognitive Assessments

3. Develop and Maintain Content for the Survey Questionnaires

4. Prepare for the Assessment Administration

5. Convene External Reviews

6. Conduct Research to Improve Assessment Development

7. Support IT Business Processes

The contractor’s performance will be evaluated annually by the COR using the following adjectival ratings: Excellent, Very Good, Satisfactory, Marginal, and Unsatisfactory. Descriptions of these adjectival ratings are provided below:

CPARS Adjectival

Rating

Description

Exceptional Performance meets contractual requirements and exceeds many to the

Government's benefit. The element being assessed was accomplished with few minor problems for which corrective actions taken by the contractor were highly effective.

Very Good Very Good: Performance meets contractual requirements and exceeds some to the Government's benefit. The element being assessed was accomplished with some minor problems for which corrective actions taken by the contractor were effective.

Satisfactory Satisfactory: Performance meets contractual requirements. The element being assessed contains some minor problems for which corrective actions taken by the contractor appear or were satisfactory.

Marginal Marginal: Performance does not meet some contractual requirements.

The element being assessed reflects a serious problem for which the contractor has not yet identified corrective actions.

Unsatisfactory Unsatisfactory: Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The element being assessed contains a serious problem(s) for which the contractor's corrective actions appear or were ineffective.

Each key task/subtask shall be separately evaluated using the above scale.

ED will rate performance separately for each Key Outcome.

The contractor shall revise any key deliverable determined to be “unacceptable” at no additional cost to the Government. That is, the contractor shall not charge the Government for the additional work to revise unacceptable products, such as reports that do not meet NCES Statistical Standards.

Each year ED will evaluate the contractor's performance by assessing the performance on key deliverables within the task. The work will be evaluated in terms of how well the contract requirements are satisfied; the extent to which the work performed follows the approach found in the contractor’s technical proposal; quality and clarity of documentation; and timeliness of key deliverables.

While quality is closely tied to the performance standards for activities and /or deliverable content, timeliness and cost are also important considerations in the assessment of contractor performance.

In the event of an excusable delay (as defined in FAR 52.249-14, Excusable Delays and EDAR

3452.242-71, Notice to the Government of Delays), the Department and the contractor shall work together to modify the contract with regard to the due dates of the deliverables. If such an event were to occur that would require a modification to the due dates of the deliverables, the contractor's performance shall be measured by the date agreed upon in the modification.

Below are the evaluation periods (Dates may vary based on actual date of award, but will align with the period of performance):

1. Base Year: November 1, 2024 – October 31, 2025

2. Option Period I: November 1, 2025 – October 31, 2026

3. Option Period II: November 1, 2026 – October 31, 2027

4. Option Period III: November 1, 2027 – October 31, 2028

5. Option Period IV: November 1, 2028 – October 31, 2029

The specific task/subtask, performance indicators, and primary surveillance method are described below:

Key Task/Subtask #1: – Annual NAEP Report Card

Performance Indicators: The Annual NAEP Report Card is the culmination of NAEP activities from item development to the operational administration of the assessment to students. Findings from the NAEP assessment are made available to the public through the release of the Nations

Report Card. NCES is required to release findings from the mathematics and reading report cards six months after the end of the data collection; 12-15 months are the average release times for all other NAEP reports. Each activity completed by the Coalition is critical to ensure NCES’s ability to successfully meet its obligations to report on student performance, culminating with the release of the NAEP reports. The contractors shall, as a Coalition, ensure the established deadlines are made for release of all NAEP reports in the defined contract period. Details of NAEP report card requirements for each contract year time period are included in Chart 1 below. The contractors shall provide weekly reports with issues/problems and requirements stated in the reporting task.

Primary Surveillance Method: CORs review of Nations Report Card.

Evaluation Criteria:

Acceptable Performance

1. Materials prepared by the contractor (i.e., draft and final publications) are of high quality, require no more than minimal revisions, and are consistent with ED’s guidance on content, formatting and publishing requirements.

2. Materials prepared by the contractor are complete and delivered on time to the COR.

Key Task/Subtask #2: (Task 4.1) – Manage Assessment Content Development

Performance Indicators: The purpose of this task is to establish and maintain protocols for how the contractor will execute, monitor, and control assessment content development activities and processes under this contract to ensure materials developed meet validity, reliability, accessibility, and equitability standards. The contractor shall submit a comprehensive Project Management Plan and Project Schedule. The contractor shall be responsible for tracking and monitoring contract deliverables according to the project schedule, Primary Surveillance Method: The COR reviews the Project Management Plan and Project

Schedule deliverables and evaluates overall management of the contract activities.

formatting and publishing requirements.

2. Materials prepared by the contractor are complete and delivered on time to the COR.

3. Established assessment content development activities and processes are followed with minimal error.

Key Task/Subtask #3: (Subtask 4.2.2) – Develop the Cognitive Assessment Content

Development Plan

Performance Indicators: The purpose of these development plans is to guide the development of new items and tasks to replace and fill gaps in the item pools. The contractor shall submit a comprehensive Cognitive Assessment Content Development Plan for each subject area that outlines the approach and procedures, development targets, and project schedules for new content development.

Primary Surveillance Method: The COR reviews the deliverables submitted under this task for adherence to quality and schedule.

1. Materials prepared by the contractor (i.e., draft and final publications) are of high quality

(i.e., align with NAEP Frameworks, meet platform requirements, comply with accessibility standards, and follow technical standards and best practices for content development), require no more than minimal revisions, and are consistent with ED’s guidance on content, Key Task/Subtask #4: (Subtask 4.2.3) – Develop New Content and Scoring Rubrics

Performance Indicators: The purpose of this task is to develop new content and associated rubrics for NAEP assessments in accordance with the Cognitive Assessment Content Development Plan

(task 4.2.2). The contractor shall develop new discrete items and scenario-based tasks (SBTs), associated scoring rubrics, and other supporting materials that are of high quality (i.e., align with

NAEP Frameworks, meet platform requirements, comply with accessibility and equity standards, and follow technical standards and best practices for content development). The contractor submits a sufficient number of quality items to fulfill the design specifications for each assessment and the associated supporting materials and documentation.

standards, and follow technical standards and best practices for content development), require no more than minimal revisions, and are consistent with ED’s guidance on content, Key Task/Subtask #5: (Subtask 4.2.1 - 4.2.11) – Randomly selected deliverables from the tasks associated with developing and maintaining cognitive assessment content for each pilot and operational assessment in accordance with scheduled assessments.

Performance Indicators: The contractor shall conduct the variety of tasks and submit associated deliverables in accordance with the plans and schedule for developing items/tasks for each pilot and operational assessment.

Primary Surveillance Method: The COR randomly selects deliverables from these subtasks for an award. The COR reviews the deliverables for adherence to the quality and schedule.

Evaluation Criteria formatting and publishing requirements.

2. Materials prepared by the contractor are complete and delivered on time to the COR.

3. New development meets requirements in the approved Cognitive Assessment Content

Development Plans (task 4.2.2).

Key Task/Subtask #6: (Subtask 4.3.1) – Evaluate existing question pool

Performance Indicators: The purpose of this task is to identify and understand key constructs of interest to support NAEP contextual reporting that need to be addressed for the next development cycle. The contractor shall develop an issues paper and evaluate the existing questionnaire item pool. The contractor shall submit an evaluation report of the quality of specific question clusters under consideration including an update that incorporates lessons learned regarding the question pool with recommended revisions.

Primary Surveillance Method: The COR reviews the deliverables and the summary evaluations of the question clusters for adherence to quality and schedule.

require no more than minimal revisions, and are consistent with ED’s guidance on content, Key Task/Subtask #7: (Subtask 4.3.3) – Develop new survey questions

Performance Indicators: The purpose of this task is to develop new survey questionnaires for

NAEP assessments in accordance with the Survey Questionnaire Development Plans (task 4.3.2).

The contractor shall develop new survey questions and indices that are of high quality (i.e., align with NAEP Frameworks, meet platform requirements, comply with accessibility and equity standards, and follow technical standards and best practices for content development). The contractor shall submit a sufficient number of new or updated survey questions to address findings from the survey question quality reviews, issues papers, and meet reporting goals for NAEP.

and equity standards, and follow technical standards and best practices for content development), meet reporting goals for NAEP, require no more than minimal revisions, and are consistent with ED’s guidance on content, formatting and publishing requirements.

2. Materials prepared by the contractor are complete and delivered on time to the COR.

3. New survey items and indices developed meet requirements in the Survey Questionnaire

Development Plans (task 4.3.2).

Key Task/Subtask #8: (Subtask 4.3.1 - 4.3.8) – Randomly selected deliverables from the tasks associated with developing and maintaining survey questionnaires content for each pilot and operational assessment in accordance with scheduled assessments.

Performance Indicators: The contractor shall conduct the variety of tasks and submit associated deliverables in accordance with the plans and schedule for developing items/tasks for each pilot and operational assessment.

Primary Surveillance Method: The COR randomly selects deliverables from these subtasks for an award. The COR reviews the deliverables for adherence to the quality and schedule.

4. Materials prepared by the contractor (i.e., draft and final publications) are of high quality,

5. Materials prepared by the contractor are complete and delivered on time to the COR.

Key Task/Subtask #9: (Subtask 4.4.1) – Develop supporting content for assessment delivery.

Performance Indicators: The purpose of this task is to develop supporting content and ancillary materials (e.g., directions, read aloud) prior to each assessment administration. The contractor shall be responsible for developing field-ready supporting assessment content materials prior to each

NAEP data collection.

Primary Surveillance Method: The COR reviews deliverables under this task for adherence to quality and schedule.

Key Task/Subtask #10: (Subtask 4.4.5) – Conduct content certification prior to assessment administration

Performance Indicators: The purpose of this task is to review and certify that all paper and digital assessment content, whether prepared and implemented within NAEP’s digital platforms or prepared for print, are field-ready prior to the NAEP administration. The contractor shall be responsible for certifying items, tasks, directions, blocks, booklets, and any supporting ancillary materials ahead of each NAEP administration.

Primary Surveillance Method: The COR reviews digital and paper content materials for adherence to quality and schedule.

(i.e., free from error and function as expected), require no more than minimal revisions, and are consistent with ED’s guidance on content, formatting and publishing requirements.

Key Task/Subtask #11: (Subtask 4.5.3) – National Assessment Governing Board reviews

Performance Indicators: The purpose of this task is to support the National Assessment Governing

Board’s review of NAEP assessment content at key stages in the development cycle. The contractor shall develop and implement an efficient review process and associated materials that facilitates the Board’s review of NAEP assessment content.

Primary Surveillance Method: The COR reviews the deliverables submitted under this task.

Process of Quality Assurance Assessment/Award Fee Evaluation

In the event of an excusable delay (as defined in FAR 52.249-14), the Department and the contractor shall work together to modify the contract regarding the due dates of deliverables. If an event occurs that would require modifying deliverable due dates, the contractor’s performance, where applicable in this QASP, shall be measured by the date agreed upon in the modification.

In narrative form, the COR will substantiate all adjectival ratings other than Satisfactory.

Performance at the “Satisfactory” level is expected from the contractor. The COR will forward a report of average scores to the CS.