Attachment J - Task 3 DAR QASP.pdf

Attachment J

Quality Assurance Surveillance Plan (QASP) National Assessment of Educational Progress

Design, Analysis and Reporting

Introduction This performance-based Quality Assurance Surveillance Plan (QASP) Evaluation Plan has been developed in accordance with Federal Acquisition Regulation (FAR) 16.401(e)(3) and pursuant to the requirements identified in the Statement of Objectives (SOO). The QASP sets forth procedures and guidelines that the Department of Education (ED) will use in evaluating the technical performance of the contractor. A copy of this plan will be forwarded to the contractor so that the contractor shall be aware of the methods that the Government will employ in evaluating performance on this contract, and to address any concerns that the contractor may have prior to initiating work.

Purpose of the QASP

The QASP is intended to accomplish the following:

• Define the roles and responsibilities of participating Government officials and outside experts;

• Define the key deliverables that will be assessed;

• Describe the rating elements and standards of performance against which the contractor’s performance will be assessed for each key deliverable;

• Describe the process of quality assurance assessment; and

• Provide copies of the quality assurance monitoring forms that will be used by the

Government in documenting and evaluating the contractor’s performance.

Roles and Responsibilities of Participating Government Officials and Experts The following Government officials and/or experts will participate in assessing the quality of the contractor’s performance. Their roles and responsibilities are described as follows:

The Contracting Officer’s Representative (COR) is responsible for monitoring, assessing, recording, and reporting on the technical performance of the contractor on a day-to-day basis. The COR also is responsible for documentation, inspection and evaluation of the contractor’s work performance.

Additional ED staff with technical knowledge and experience related to NAEP design, analysis, and reporting may provide input and review deliverables. However, ED reserves the right to have only the COR assess each deliverable.

The Contract Specialist (CS) is responsible for the day-to-day monitoring of the contractor’s performance in the areas of contract compliance, contract administration, cost control, property control; reviewing the COR’s assessment of the contractor’s performance; and resolving all differences between the COR’s version and the contractor’s version. The CS may call upon the expertise of other Government individuals as required.

The Contracting Officer (CO) has the following procurement authorities:

• Sole authority for any decisions that produce an increase or decrease in the contract’s scope;

• Sole authority for any decision to be rendered under the “Disputes” clause;

• Sole authority to approve the substitution or replacement of the project manager or other key personnel;

• Sole authority to monitor and enforce Department of Labor promulgated labor requirements;

• Authority to arrange for and supervise quality assurance activities under this contract;

• Sole authority to approve the contractor’s quality control program;

• Signatory authority for the issuance of all modifications to the contract.

Annual Past Performance Evaluation The Contractor’s overall performance shall be documented and measured through the annual past performance evaluation through the Contractor Performance Assessment Reporting System (CPARS). Performance will be evaluated in the areas of:

• Quality of Product or Service

• Schedule

• Cost Control (if applicable)

• Business Relations

• Management of Key Personnel

• Utilization of Small Business

Quality Assurance Review Process Even though the Government, through its COR, will monitor the contractor’s performance on a continuing basis, the volume of tasks performed by the contractor makes technical inspection of every task and step impractical. Therefore, ED will use an annual quality assurance review process to monitor the contractor’s performance under the award. Specifically, the COR will assess the contractor’s performance on the following key deliverables:

1. Team award: Annual NAEP Report Card Submission

2. Item Analysis and Item Development Support – Subtask 3.2.2

3. Process Data Analysis for Operational Support and Report Cards – Subtask

3.2.4

4. Analyses for Report Cards– Subtask 3.2.5

5. Developing Report Cards – Subtask 3.3.2

6. Ad Hoc Technical Support – Subtask 3.3.10

7. Research and Development – Subtask 3.4

The contractor’s performance will be evaluated annually by the COR using the following adjectival ratings: Exceptional, Very Good, Satisfactory, Marginal, and Unsatisfactory.

Descriptions of these adjectival ratings are provided below:

CPARS Adjectival Rating

Description

Exceptional Performance meets contractual requirements and exceeds many to the Government's benefit. The element being assessed was accomplished with few minor problems for which corrective actions taken by the contractor were highly effective.

Very Good Very Good: Performance meets contractual requirements and exceeds some to the Government's benefit. The element being assessed was accomplished with some minor problems for which corrective actions taken by the contractor were effective.

Satisfactory Satisfactory: Performance meets contractual requirements. The element being assessed contains some minor problems for which corrective actions taken by the contractor appear or were satisfactory.

Marginal Marginal: Performance does not meet some contractual requirements.

The element being assessed reflects a serious problem for which the contractor has not yet identified corrective actions.

Unsatisfactory Unsatisfactory: Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The element being assessed contains a serious problem(s) for which the contractor's corrective actions appear or were ineffective.

Each key task/subtask shall be separately evaluated using the above scale.

ED will rate performance separately for each Key Outcome.

The contractor shall revise any key deliverable determined to be “unacceptable” at no additional cost to the Government. That is, the contractor shall not charge the Government for the additional work to revise unacceptable products, such as reports that do not meet NCES Statistical Standards.

Each year ED will evaluate the contractor's performance by assessing the performance on key deliverables within the task. The work will be evaluated in terms of how well the contract requirements are satisfied; the extent to which the work performed follows the approach found in the contractor’s technical proposal; quality and clarity of documentation; and timeliness of key deliverables.

While quality is closely tied to the performance standards for activities and /or deliverable content, timeliness and cost are also important considerations in the assessment of contractor performance.

In the event of an excusable delay (as defined in FAR 52.249-14, Excusable Delays and EDAR 3452.242-71, Notice to the Government of Delays), the Department and the contractor shall work together to modify the contract with regard to the due dates of the deliverables. If such an event were to occur that would require a modification to the due dates of the deliverables, the contractor's performance shall be measured by the date agreed upon in the modification.

Below are the evaluation periods (Dates may vary based on actual date of award, but will align with the period of performance):

1. Base Year – Base Year – October 15, 2024 – September 30, 2025

2. Option Period I – October 1, 2025 – September 30, 2026

3. Option Period II – October 1, 2026 – September 30, 2027

4. Option Period III – October 1, 2027 – September 30, 2028

5. Option Period IV – October 1, 2028 – September 30, 2029

The specific task/subtask, performance indicators, and primary surveillance method are described below:

Key Task/Subtask #1: – Annual NAEP Report Card

Performance Indicators: The Annual NAEP Report Card is the culmination of NAEP activities from item development to the operational administration of the assessment to students. Findings from the NAEP assessment are made available to the public through the release of the Nations Report Card. NCES is required to release findings from the mathematics and reading report cards six months after the end of the data collection; 12- 15 months are the average release time for all other NAEP reports. Each activity completed by the Coalition is critical to ensure NCES’s ability to successfully meet its obligations to report on student performance, culminating with the release of the NAEP reports. The contractors shall, as a Coalition, ensure the established deadlines are made for release of all NAEP reports in the defined contract period. Details of NAEP report card requirements for each contract year time period are included in Chart 1 below. The contractors shall complete tasks in the PWS in a timely manner to ensure the release of Report Cards on time.

Primary Surveillance Method: Individual COR’s review of activities leading to Nations Report Card development and release.

Evaluation Criteria:

Acceptable Performance

1. The Report Cards adhere to the release schedule approved by NCES.

Key Task/Subtask #2: (Subtask 3.2.2) – Item Analysis and Item Development Support

Performance Indicators: The DAR contractor shall provide psychometric support to the item development contractor for all phases of item development, including (1) determining the number of items and blocks to be developed, (2) evaluating the performance of piloted items, and (3) developing automated block assembly algorithm to support MST and linear assessment block assembly.

Primary Surveillance Method: The COR reviews handoffs from the DAR contractor to the Assessment Content Development contractor.

Evaluation Criteria

Acceptable Performance

1. The number of items and blocks to be developed is recommended to the Assessment Content Development contractor and approved by NCES to guide item development.

2. The pilot analysis results and memorandum are shared with the ID contractor in a timely manner to inform item selection for the operational assessment and the future item development planning.

3. Automated block assembly algorithm is developed or commercial product is recommended and implemented for the subjects assessed in the contract cycle.

Key Task/Subtask #3: (Subtask 3.2.4) – Process Data Analysis for Operational Support and Report Cards

Performance Indicators: The DAR contractor shall conduct analyses of the process data as part of the routine analysis activity. The process data analysis shall have at least two purposes: (1) operational support, and (2) Report Card development support. The DAR contractor shall develop a framework to identify the approaches to analyzing the process data for operational and reporting purposes.

Primary Surveillance Method: The COR reviews process data analysis findings for operational support and Report Card development support.

Evaluation Criteria

Acceptable Performance

1. The framework is delivered on time to the COR.

2. Process data analyses are conducted in a timely manner to support operational activities such as data validation and item development.

3. Process data analyses are conducted in a timely manner to support Report Card development.

Key Task/Subtask #4: (Subtask 3.2.5) – Analyses for Report Cards

Performance Indicators: The contractor shall submit electronic version of summary data tables, parameter estimates, standard errors and significance test results, and hold a meeting with NCES to present the results of the analyses for Report Cards.

Primary Surveillance Method: The COR reviews the data summary and the meeting materials.

Evaluation Criteria

Acceptable Performance

1. The data summary reflects sampling and design features and incorporates appropriate standard errors and tests of significance.

2. Data summarized and meeting held with NCES in timely fashion for reporting deadlines to be met.

Key Task/Subtask #5: (Subtask 3.3.2) – Developing Report Cards

Performance Indicators: The contractor shall develop all report cards for the contract period for each assessed subject showing (at minimum) results in terms of achievement levels and scale scores, results for states where applicable, results for major subgroups of interest, a sample of released content items for each grade, process or observable data from student interactions with the assessment, and results from contextual questions in surveys of students, teachers, and schools. Report cards also include discussion of results, as well as such technical information required to make the report understandable to the public.

Primary Surveillance Method: The COR reviews the Report Cards.

Evaluation Criteria

Acceptable Performance

1. The Report Cards on 6-month schedule are approved in time to meet 6-month reporting requirement. Other Report Cards are approved in time to meet 12-month or other reporting timeline as required.

Key Task/Subtask #6: (Subtask 3.3.11) – Ad Hoc Technical Support

Performance Indicators: The contractor shall provide technical support on an ad hoc basis for quick turnaround analyses of NAEP data requested from media, policy makers, and the general public. The contractor shall ensure the analyses are error-free and shall provide clear documentation of the methods used and results. Turnaround shall be within 24 hours or as otherwise agreed with NCES.

Primary Surveillance Method: The COR reviews the deliverables.

Evaluation Criteria:

1. Materials and analyses provided by the contractor are error-free and in a timely manner.

Key Task/Subtask #7: (Subtask 3.4) – Research and Development

Performance Indicators: The contractor shall develop research proposals to solve psychometric issues and advance NAEP design, data analysis and reporting methodologies..

Primary Surveillance Method: The COR reviews research proposals and study findings.

Evaluation Criteria:

1. Research proposal and study plan are submitted to the COR in a timely manner when psychometric issues are identified.

Process of Quality Assurance Assessment

In the event of an excusable delay (as defined in FAR 52.249-14), the Department and the contractor shall work together to modify the contract regarding the due dates of deliverables. If an event occurs that would require modifying deliverable due dates, the contractor’s performance, where applicable in this QASP, shall be measured by the date agreed upon in the modification.

In narrative form, the COR will substantiate all adjectival ratings other than Acceptable.

Performance at the “Acceptable” level is expected from the contractor. The COR will forward a report of average scores to the CS.