Attachment J - Task 6 NPD QASP.pdf

Attachment J - Task 6 NPD QASP

Quality Assurance Surveillance Plan Evaluation Plan

National Assessment of Educational Progress

NAEP Platform Development (NPD)

Introduction

This performance-based Quality Assurance Surveillance Plan (QASP) Evaluation Plan has been developed in accordance with Federal Acquisition Regulation (FAR) 12.208 and pursuant to the requirements identified in the Statement of Objectives (SOO). The QASP sets forth procedures and guidelines that the Department of Education (ED) will use in evaluating the technical performance of the contractor. A copy of this plan will be forwarded to the contractor so that the contractor shall be aware of the methods that the

Government will employ in evaluating performance on this contract, and to address any concerns that the contractor may have prior to initiating work.

Purpose of the QASP

The QASP is intended to accomplish the following:

• Define the roles and responsibilities of participating Government officials and outside experts.

• Define the key deliverables that will be assessed.

• Describe the rating elements and standards of performance against which the contractor’s performance will be assessed for each key deliverable.

• Describe the process of quality assurance assessment; and

• Provide copies of the quality assurance monitoring forms that will be used by the

Government in documenting and evaluating the contractor’s performance.

Roles and Responsibilities of Participating Government Officials and Experts

The following Government officials and/or experts will participate in assessing the quality of the contractor’s performance. Their roles and responsibilities are described as follows:

The Contracting Officer’s Representative (COR) is responsible for monitoring, assessing, recording, and reporting on the technical performance of the contractor on a day-to-day basis. The COR also is responsible for documentation, inspection and evaluation of the contractor’s work performance.

Additional ED staff with technical knowledge and experience related to the development of the platform and systems for the administration of NAEP assessments may provide input and review deliverables. However, ED reserves the right to have only the COR assess each deliverable.

The Contract Specialist (CS) is responsible for the day-to-day monitoring of the contractor’s performance in the areas of contract compliance, contract administration, cost control, property control; reviewing the COR’s assessment of the contractor’s performance; and resolving all differences between the COR’s version and the contractor’s version. The CS may call upon the expertise of other Government individuals as required.

The Contracting Officer (CO) has the following procurement authorities:

• Sole authority for any decisions that produce an increase or decrease in the contract’s scope.

• Sole authority for any decision to be rendered under the “Disputes” clause.

• Sole authority to approve the substitution or replacement of the project manager or other key personnel.

• Sole authority to monitor and enforce Department of Labor promulgated labor requirements.

• Authority to arrange for and supervise quality assurance activities under this contract.

• Sole authority to approve the contractor’s quality control program.

• Signatory authority for the issuance of all modifications to the contract.

Annual Past Performance Evaluation

The Contractor’s overall performance shall be documented and measured through the annual past performance evaluation through the Contractor Performance

Assessment Reporting System (CPARS). Performance will be evaluated in the areas of:

• Quality of Product or Service

• Schedule

• Cost Control (if applicable)

• Business Relations

• Management of Key Personnel

• Utilization of Small Business

Quality Assurance Review Process

Even though the Government, through its COR, will monitor the contractor’s performance on a continuing basis, the volume of tasks performed by the contractor makes technical inspection of every task and step impractical. Therefore, ED will use an annual quality assurance review process to monitor the contractor’s performance under the award. Specifically, the COR will assess the contractor’s performance on the following key deliverables:

1. Annual NAEP Report Card Submission

2. Develop and Implement Platform – Subtask 6.1.4

3. Develop and Implement Assessment Management System – Subtask 6.1.6

4. Develop and Implement Data Capture, Storage, and Transfer Protocols –

Subtask 6.1.7

5. Develop Platform Tutorial and Instructional Design – Subtask 6.1.8

6. Develop and Implement Online NAEP Teacher and School Administrator

Questionnaire – Subtask 6.1.9

7. Support Platform Deployment and Logistics – Subtask 6.3.1

8. Support Assessment Pre-Planning and Readiness Checks – Subtask 6.3.2

9. Provide Training and Demonstrations – Subtask 6.3.3

10. Provide Technical Assistance and Help Desk Support for the Platform and

Supporting Systems– Subtask 6.3.4

11. Monitor Platform Performance in the Field – Subtask 6.3.5

12. Conduct Studies and R&D for the Platform and Supporting Systems – Task 6

The contractor’s performance will be evaluated annually by the COR using the following adjectival ratings: Excellent, Very Good, Satisfactory and Unsatisfactory. Descriptions of these adjectival ratings are provided below:

CPARS Adjectival

Rating

Description

Exceptional Performance meets contractual requirements and exceeds many to the

Government's benefit. The element being assessed was accomplished with few minor problems for which corrective actions taken by the contractor were highly effective.

Very Good Very Good: Performance meets contractual requirements and exceeds some to the Government's benefit. The element being assessed was accomplished with some minor problems for which corrective actions taken by the contractor were effective.

Satisfactory Satisfactory: Performance meets contractual requirements. The element being assessed contains some minor problems for which corrective actions taken by the contractor appear or were satisfactory.

Marginal Marginal: Performance does not meet some contractual requirements.

The element being assessed reflects a serious problem for which the contractor has not yet identified corrective actions.

Unsatisfactory Unsatisfactory: Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The element being assessed contains a serious problem(s) for which the contractor's corrective actions appear or were ineffective.

Each key task/subtask shall be separately evaluated using the above scale.

ED will rate performance separately for each Key Outcome.

In order to award payment for Key Outcomes within a given evaluation period, the AFRB must rate the contractor’s performance on the key deliverables as "Outstanding," or

"Superior." No award payment will be made for performance that is rated as “Acceptable” or

"Unacceptable." The contractor shall revise any key deliverable determined to be

“unacceptable” at no additional cost to the Government. That is, the contractor shall not charge the Government for the additional work to revise unacceptable products, such as reports that do not meet NCES Statistical Standards.

Each year ED will evaluate the contractor's performance by assessing the performance on key deliverables within the task. The work will be evaluated in terms of how well the contract requirements are satisfied; the extent to which the work performed follows the approach found in the contractor’s technical proposal; quality and clarity of documentation; and timeliness of key deliverables.

While quality is closely tied to the performance standards for activities and /or deliverable content, timeliness and cost are also important considerations in the assessment of contractor performance.

In the event of an excusable delay (as defined in FAR 52.249-14, Excusable Delays and

EDAR 3452.242-71, Notice to the Government of Delays), the Department and the contractor shall work together to modify the contract with regard to the due dates of the deliverables. If such an event were to occur that would require a modification to the due dates of the deliverables, the contractor's performance shall be measured by the date agreed upon in the modification.

Below are the evaluation periods (Dates may vary based on actual date of award, but will align with the period of performance):

1. Base Year – November 1, 2024 – October 31, 2025

2. Option Period I – November 1, 2025 – October 31, 2026

3. Option Period II – November 1, 2026 – October 31, 2027

4. Option Period III – November 1, 2027 – October 31, 2028

5. Option Period IV – November 1, 2028 – October 31, 2029

The specific task/subtask, performance indicators, and primary surveillance method are described below:

Key Task/Subtask #1: – Annual NAEP Report Card

Performance Indicators: The Annual NAEP Report Card is the culmination of NAEP activities from item development to the operational administration of the assessment to students. Findings from the NAEP assessment are made available to the public through the release of the Nations Report Card. NCES is required to release findings from the mathematics and reading report cards six months after the end of the data collection; 12-

15 months are the average release times for all other NAEP reports. Each activity completed by the Coalition is critical to ensure NCES’ ability to successfully meet its obligations to report on student performance, culminating with the release of the NAEP reports. The contractors shall, as a Coalition, ensure the established deadlines are made for release of all NAEP reports in the defined contract period. Details of NAEP report card requirements for each contract year time period are included in Chart 1 below. The contractors shall provide weekly reports with issues/problems and requirements stated in the reporting task.

Primary Surveillance Method: COR’s review of Nations Report Card.

Evaluation Criteria:

Acceptable Performance

1. Materials prepared by the contractor (i.e., draft and final publications) are of high quality, require no more than minimal revisions, and are consistent with ED’s guidance on content, formatting and publishing requirements.

2. Materials prepared by the contractor are complete and delivered on time to the

COR.

Key Task/Subtask #2: (Subtask 6.1.4) - Develop and Implement Platform

Performance Indicators: In support of NAEP’s goal of serving as a model of excellence and innovation in large-scale assessment, to facilitate operational administration of the assessment to students, and to ensure a positive and engaging student experience the contractor shall develop and deliver annually a state-of-the-art NAEP DBA Platform. The

Platform shall be of high-quality and employ a modern, age-appropriate UI/UX that is consistent, intuitive, and accessible across all relevant subjects and grade levels. The

Contractor shall also ensure that the Platform is device agnostic and operable on school-owned devices and networks, and fully supports developed content and item components from the NAEP ACD Contractor and other third-party vendors (as appropriate). In addition, the Contractors shall, as a Coalition, ensure that the Platform and all integrated assessment content are fully 508 and WCAG compliant; and that the Platform incorporates the agreed upon technical accommodations and universal design features.

All activities shall be completed in alignment with the documented Project Management

Plan, NAEP DBA Platform Development Plan and Schedule, and multi-cycle process model (Subtask 9.1.1); and the Contractors shall, as a Coalition, ensure the established deadlines are met for deployment of the final, field-ready Platform by the defined administration window. The Contractor shall provide weekly reports to the COR with issues/problems and requirements as stated in Task # 1 in the PWS.

Primary Surveillance Method: COR’s review of the Platform and associated Task 1 deliverables including the Project Management Plan, Schedule, and requirements and design specifications; volume and severity of known and/or identified issues in each iteration of the Platform delivered to NCES for review, and prior to field deployment;

and performance of the Platform during operational field administration.

3. Each iteration of the Platform developed by the Contractor and delivered to NCES for review is of high quality, has minimal defects or known issues, and is consistent with NCES expectations for the defined stage of review.

4. Materials prepared by the Contractor, and each iteration of the Platform developed by the Contractor, is complete and delivered on time to the COR in alignment with the Project Management Plan, Platform Development Schedule, and NAEP Contractor deadlines for field deployment and administration.

5. The final field-ready Platform developed by the Contractor with integrated assessment content is of high quality, has minimal defects or known issues, and is consistent with NCES expectations and documented Platform requirement and design specifications.

6. Smooth operational administration of the assessment to students with minimal technical errors or student-facing issues.

Key Task/Subtask #3: (Subtask 6.1.6) – Develop and Implement Assessment

Management System

Performance Indicators: The Contractor, in collaboration with the NDC, SW, and NSSC

Contractors, shall develop an Assessment Management System (AMS). This system shall be of high-quality and support NCES, NAEP Contractor staff, NAEP State and TUDA coordinators, school and district staff, and data collection staff throughout the country with critical tasks related to recruitment and sampling, preassessment activities with schools, assessment day operations, and special studies, and with ongoing support for the administration of the NAEP assessment on school-owned devices and networks. The

AMS shall employ a state-of-the-art UI/UX with a modern user-based design that provides a positive and efficient user experience across all of its wide-range of user groups and applications. In addition, the system shall be fully integrated with the DBA

Platform and other critical NCES contractor applications and workflows, where applicable. The system shall also be fully 508 and WCAG compliant, and operable in both the mainland United States (English and Spanish) and Puerto Rico (Spanish). All activities shall be completed in alignment with the documented Project Management

Plan, AMS Development Plan and Schedule (Subtask 9.1.1), and AMS Integration Plan;

and the Contractors shall, as a Coalition, ensure that the established deadlines are met for deployment of the final, field-ready AMS system by the defined administration window.

Primary Surveillance Method: COR’s review of the AMS System and associated Task 1 deliverables including the requirements and design specifications; volume and severity of known and/or identified issues in each iteration of the AMS delivered to NCES for review, and prior to field deployment; and performance of the AMS during pre-assessment activities and the operational field administration.

1. Each iteration of the AMS developed by the Contractor and delivered to NCES for review is of high quality, has minimal defects or known issues, and is consistent with NCES expectations for the defined stage of review.

2. Materials prepared by the Contractor, and each iteration of the AMS developed by the Contractor, is complete and delivered on time to the COR in alignment with the Project Management Plan, AMS Development Schedule, and NAEP

Contractor deadlines for field deployment and administration.

3. The final field-ready AMS developed by the Contractor is of high quality, has minimal defects or known issues, and is consistent with NCES and NDC, SW, and NSSC Contractor expectations and documented requirement and design specifications.

4. Smooth operational use by the AMS’ wide-range of user groups and stakeholders with minimal technical errors or user-facing issues.

Key Task/Subtask #4: (Subtask 6.1.7) – Develop and Implement Data Capture, Storage, and Transfer Protocols

Performance Indicators: Efficient, accurate, and reliable data capture, storage, and transfer are critical to NAEP operations and a wide range of NAEP activities. The

Contractors shall, as a Coalition, develop and implement protocols that ensure efficient and effective capture, storage, and transfer of Platform (both student response and observable) and supporting systems data securely, accurately, and reliably across NAEP contractors; and in a format that fulfills the NAEP contractors various needs for sampling, data collection, administration, scoring, analysis, and reporting. The contractor shall accurately capture and document/inventory the requirements of NCES and the

NAEP Contractors in terms of data capture and develop and implement the necessary technical solutions and protocols. All activities shall be completed in alignment with the documented Project Management Plan, Platform and AMS Development Plans and

Schedules (Subtask 9.1.1), and Data Capture, Storage, and Transfer Plan; and the contractors shall, as a Coalition, ensure the established deadlines are met for deployment of the final, field-ready Platform by the defined administration window.

The contractor shall provide at least weekly reports with issues/problems and requirements as stated in Task # 1 in the PWS.

Primary Surveillance Method: COR’s review of deliverables; volume and severity of known issues prior to field deployment; documented results of NAEP Contractor Data

Capture, Storage, and Transfer QC activities; and results from operational administration and subsequent scoring and analysis activities.

1. Data capture, storage, and transfer protocols developed by the Contractor are efficient and effective, with minimal data loss, high fidelity, and high usability across all NAEP contractors and their various needs, and in alignment with NCES and NAEP Contractor expectations and documented requirements.

2. The technical solutions and protocols developed by the Contractor are complete and implemented on time in alignment with the Project Management Plan, Platform and AMS Development Plans and Schedules, Data Capture, Storage, and

Transfer Plan, and NAEP Contractor deadlines for field deployment and administration.

Key Task/Subtask #5: (Subtask 6.1.8) – Develop Platform Tutorial and

Instructional Design

Performance Indicators: The Contractor shall develop and implement an interactive tutorial for the NAEP DBA Platform. As the gateway to a positive assessment experience, and valid measurement of student achievement, the Platform tutorial developed by the Contractor shall be engaging, state-of-the art, and employ a modern, age-appropriate UI/UX that is consistent, intuitive, and accessible across all relevant subjects and grade levels. The Contractor shall ensure that the tutorial is fully 508 and

WCAG compliant; and that it incorporates the agreed upon technical accommodations and universal design features consistent with the Platform itself. Of critical importance is that the tutorial be effective in terms of acclimating students to the Platform and component tools. The Contractors shall, as a Coalition, develop a strategy for evaluating and assessing tutorial efficacy and implementing necessary improvements to the tutorial model and content based on findings. All activities shall be completed in alignment with the documented Instructional Design and Tutorial Development Plans and Tutorial

Development Schedule; and the contractors shall, as a Coalition, ensure the established deadlines are met for integration and deployment of the final, field-ready Platform

Tutorial by the defined administration window.

The contractor shall provide at least monthly reports with issues/problems and requirements stated in Task # 1 in the PWS.

Primary Surveillance Method: COR’s review of the Tutorial and associated subtask deliverables, including the Instructional Design and Tutorial Development Plans and associated efficacy studies and continuous improvement activities; volume and severity of known and/or identified issues in each iteration of the Platform delivered to NCES for review, and prior to field deployment; and performance of the Tutorial during operational field administration.

Evaluation Criteria

1. Each iteration of the Tutorial developed by the Contractor and delivered to NCES for review is of high quality, has minimal defects or known issues, and is consistent with NCES expectations for the defined stage of review.

2. Materials prepared by the Contractor, and each iteration of the Tutorial developed by the Contractor, are complete and delivered on time to the COR (or other relevant stakeholder such as the Tutorial Standing Committee) in alignment with the Tutorial Development Plan and Schedule, and NAEP Contractor deadlines for field deployment and administration.

3. The final field-ready Tutorial developed by the Contractor is of high quality, has minimal defects or known issues, and is consistent with NCES expectations and documented requirements and specifications.

5. The contractor has documented a strategy for evaluating tutorial efficacy and identifying improvements and enhancements to the model and content; and is executing these activities in alignment with the Tutorial Development Plan.

6. Smooth operational administration of the Tutorial to students with minimal technical errors or student-facing issues.

Key Task/Subtask #6: (Subtask 6.1.9) – Develop and Implement Online NAEP

Teacher and School Administrator Questionnaire

Performance Indicators: The Contractor shall develop and implement a state-of-the-art online NAEP Teacher and School Administrator Questionnaire (NAEPq) platform. The developed solution shall be of high-quality and employ a modern UI/UX and associated

Help and tutorial content to ensure a high-level of usability and accessibility for the targeted user base (i.e., teachers and school administrators). The Contractor shall ensure that the Platform fully supports item content and components from the NAEP ACD

Contractor; and smooth, error-free administration of the questionnaires and associated data capture and transfer. In addition, the Contractor shall ensure that the Platform and all integrated content are fully 508 and WCAG compliant. All activities shall be completed in alignment with the documented Questionnaire Platform Development Plan and Schedule; and the contractors shall, as a Coalition, ensure the established deadlines are met for deployment of the final, field-ready Platform by the defined administration window.

The contractor shall provide at least weekly reports with issues/problems and requirements as stated in Task # 1 in the PWS.

Primary Surveillance Method: COR’s review of the Platform and associated Task 1 deliverables, including the Questionnaire Platform Development Plan and requirements and design specifications relevant to the Questionnaire Platform; volume and severity of known and/or identified issues in each iteration of the Platform delivered to NCES for review, and prior to final deployment; and performance of the Platform during operational administration.

1. Each iteration of the Questionnaire Platform developed by the Contractor and delivered to NCES for review is of high quality, has minimal defects or known issues, and is consistent with NCES expectations for the defined stage of review.

2. Materials prepared by the Contractor, and each iteration of the Questionnaire

Platform developed by the Contractor, is complete and delivered on time to the

COR in alignment with the Questionnaire Platform Development Plan and

Schedule; and NAEP Contractor deadlines for deployment and administration.

3. The final Questionnaire Platform developed by the Contractor with integrated item content is of high quality, has minimal defects or known issues, and is consistent with NCES expectations and documented requirement and design specifications.

4. Smooth administration of the questionnaires to teachers and school administrators with minimal technical errors or user-facing issues.

Key Task/Subtask #7: (Subtask 6.3.1) – Support Platform Deployment and Logistics

Performance Indicators: The Contractors shall, as a Coalition, coordinate deployment of the final system to the field in a format accessible to school and district staff for easy installation and operation on school devices. The developed solution shall also have built in provisions for simple and timely validation of successful installation and configuration, and confirmation of assessment readiness prior to and on the day of the assessment. In collaboration with the NDC Contractor, and other NAEP contractors, as appropriate, the

Contractor shall develop a Platform Deployment Plan for operationalizing the Platform in the field. The plan shall identify the proposed method of deployment, and the Contractor shall prepare all necessary materials, applications, and resources required to support the technical deployment of the assessment. In addition, the Contractors shall, as a Coalition, develop and coordinate comprehensive procedures for testing the planned deployment process and verifying the completeness and correctness of the deployed system in the field. All activities shall be completed in alignment with the documented Platform

Deployment Plan; and the contractors shall, as a Coalition, ensure the established deadlines are met for deployment of the final, field-ready Platform by the defined administration window.

Primary Surveillance Method: COR’s review of deliverables, namely the Platform

Deployment Plan; documented results of Platform deployment pre-testing, QC, and validation activities; and validity of deployment for operational field administration.

1. The Platform Deployment Plan and protocols developed by the Contractor are efficient and effective, with successful and smooth deployment of the final

Platform to the field for student administration on time in alignment with NAEP

Contractor deadlines for field deployment and administration.

2. Pre-testing and validation of the deployment process are completed on time in alignment with the Platform Deployment Plan and NAEP Contractor deadlines for field deployment and administration, and necessary changes implemented to correct identified defects and ensure successful deployment.

Key Task/Subtask #8: (Subtask 6.3.2) – Support Assessment Pre-Planning and

Readiness Checks of Schools

Performance Indicators: Validation of school network, device, and application readiness for assessment administration is critical with the shift to administering NAEP on school-owned devices. The Contractor shall develop and implement robust and efficient solutions for validating school networks, devices, and setup/installation prior to the assessment day. The developed solution(s) shall be easily accessible to school and district staff supporting assessment preparations, and simple to use/highly user-friendly to ensure low burden. The Contractors shall, as a Coalition, prepare all necessary materials and solutions to support school pre-planning and preparedness leading up to the assessment date. This includes documentation and clear communication of minimum requirements for supporting the assessment (both network and device) along with the necessary diagnostic and readiness checks to validate eligibility and readiness. All planned materials and communications for assessment pre-planning and readiness shall be documented and packaged for NCES review and approval prior to being disseminated to schools each year. In addition, the Contractors shall, as a Coalition, develop and coordinate comprehensive procedures for testing the planned protocols. All activities shall be completed in alignment with the documented Project Management Plan and

Platform and AMS Development Plans and Schedules (Subtask 9.1.1); and the contractors shall, as a Coalition, ensure the established deadlines are met for deployment of the final, field-ready solution(s) by the defined administration window.

Primary Surveillance Method: COR’s review of deliverables, namely the materials, communications, and developed solutions for validating school networks, devices, and readiness (setup/installation) for assessment day; documented results of pre-testing, QC, and validation activities; and performance and efficacy of the developed solutions during pre-assessment activities and operational field administration.

1. The materials and solutions developed by the Contractor are of high quality, efficient and effective, with minimal defects or known issues, and consistent with

NCES and NAEP Contractor expectations and documented requirement and design specifications.

2. Each iteration of the package developed by the Contractor and delivered to NCES for review is of high quality, has minimal defects or known issues, and is consistent with NCES expectations for the defined stage of review.

3. Materials and solutions prepared by the Contractor across each iteration and/or defined stage review are complete and delivered on time to the COR in alignment with the Project Management Plan, Platform and AMS Development Schedules, and NAEP Contractor deadlines for field deployment and administration.

4. The final field-ready solutions developed by the Contractor are of high quality, have minimal defects or known issues, and are consistent with NCES and NAEP

Contractor expectations and documented requirements and design specifications.

5. Smooth operational use by school and district staff and other stakeholders supporting pre-assessment activities with minimal technical errors or user-facing issues.

Key Task/Subtask #9: (Subtask 6.3.3) – Provide Training and Demonstrations

Performance Indicators: Training is a major NCES activity, particularly for NAEP data collection staff and State and TUDA Coordinators. It is critical to NAEP operations that these key user groups be efficiently on-boarded to the AMS. In addition, the first line of defense in supporting the DBA Platform in the field is the proper training of data collection staff administering the assessment so that they can address as many issues as possible immediately; on-site. In collaboration with the NDC Contractor, the Contractor shall provide personnel, materials, and equipment; and fully support training of field staff and key user groups of the AMS system in the use of the Platform and AMS. In addition, the Contractor shall develop demonstration and training environments of both the

Platform and AMS that are continuously available, and faithfully replicate the most up-to-date version of both systems in support of training and demonstrations. The Contractor shall be available on an ad hoc basis to provide and/or support demonstrations of the

Platform and supporting systems to NCES staff, NCES contractors, the NAEP Governing

Board, or other stakeholders at the request of the COR. All activities shall be completed in alignment with the documented Training and Demonstration Plan; and the contractors shall, as a Coalition, ensure the established deadlines are met for scheduled trainings, demonstrations, and associated materials.

Primary Surveillance Method: COR’s review of deliverables, including the Training and

Demonstration Plan and demonstration and training environment; and participant feedback from trainings and demonstrations conducted.

1. Materials prepared by the Contractor and technical solutions developed are of high quality with minimal defects and are consistent with NCES and NDC contractor needs and expectations for training and demonstration.

2. Materials prepared by the Contractor and technical solutions developed are complete and delivered on time in alignment with the Training and Demonstration

Plan and Coalition deadlines for field staff training and related activities.

3. Support provided to NCES and the Coalition for training and demonstration of the

Platform is comprehensive and robust.

Key Task/Subtask #10: (Subtask 6.3.4) – Provide Technical Assistance and Help

Desk Support for the Platform and Supporting Systems

Performance Indicators: The Contractor shall provide full-time, continuous technical support for field deployment of the DBA Platform and supporting systems—including all pre-assessment data entry and validation; school technical pre-planning and readiness checks; installation/configuration of the assessment application on school devices and for operation on school networks; confirmation of full assessment readiness; and any necessary troubleshooting on or before the assessment day at each assessed school. This applies to all NAEP assessment and data collection operations (including operational studies) that utilize the DBA Platform and supporting systems; and is inclusive of user support for the Platform, AMS, and NAEPq. In coordination with the NDC Service Desk, the Contractor shall manage Tier 2 technical support, and make available dedicated and knowledgeable personnel to respond to technical problems and answer questions from the field. The Contractor shall dedicate resources to this simultaneously with and separate from ongoing development and maintenance of the DBA Platform or supporting systems for subsequent assessment cycles and any operational studies planned or underway. All activities shall be completed in alignment with the documented Help Desk Management

Plan; and the Contractors shall, as a Coalition, ensure the established deadlines are met for operational support.

Primary Surveillance Method: COR’s review of deliverables; help desk performance statistics and metrics; and quality of materials developed and support provided for operational field administrations.

1. Materials prepared by the Contractor and technical support provided are of high quality and delivered on time in alignment with the Help Desk Management Plan and NCES and NDC contractor needs and expectations.

2. All Tier 2 Help Desk tickets are resolved in a timely fashion with no undue backlog of issues or non-response.

3. Tier 2 Help Desk staff are knowledgeable; and resource allocation does not impact ongoing development and maintenance of the Platform, supporting systems, or operational studies in any way.

Key Task/Subtask #11: (Subtask 6.3.5) – Monitor Platform Performance in the

Field

Performance Indicators: Following the production release of the DBA Platform and the opening of the AMS and NAEPq, the contractor shall continuously monitor system performance to ensure correct operation in the field and for early identification and resolution of critical issues. The necessary constructs shall be developed, and resources allocated, to facilitate dynamic and comprehensive monitoring and analysis of system performance and health; and to develop and deploy any software updates or other technical solutions required to address critical issues in an efficient and timely fashion. In addition, the Contractors, as a Coalition, shall develop and implement a robust digital dashboard accessible to NCES and the NAEP Contractors to enable real-time systems and field operations monitoring. All activities shall be completed in alignment with the documented Performance Monitoring and Dashboard Development Plans; and the contractors shall, as a Coalition, ensure the established deadlines are met for operational administration monitoring and support.

Primary Surveillance Method: CORs review of deliverables, including the digital dashboard and performance monitoring and software update protocols; performance of

Dashboard and developed protocols during operational field administration.

1. The Performance Monitoring Plan, digital Dashboard, and protocols developed by the Contractor are efficient and effective, with successful monitoring of deployed systems and early identification and analysis of issues.

2. Materials prepared by the Contractor and technical solutions developed are complete and delivered on time in alignment with the Performance Monitoring

Plan, Dashboard Development Plan and Schedule, and Coalition deadlines for operational field administration monitoring and support.

3. Final field-ready Dashboard developed by the contractor is of high quality, has minimal defects or known issues, and is consistent with NCES expectations and documented requirements and specifications.

4. Protocols developed (monitoring and software update deployment) are thoroughly pre-tested and work smoothly, as expected, during operational field administration (if applicable).

Key Task/Subtask #12: (Task 6.4) – Conduct Studies and R&D for the Platform and

Supporting Systems

Performance Indicators: The contractors shall, as a Coalition, work toward the continuous refinement and innovation of the DBA Platform and supporting technologies. All improvements and enhancements to the Platform must be firmly grounded in research and analysis—of both the available NAEP performance data (student response and process data) and trends and best practices at the cutting edge of large-scale assessment and education technology. The Contractor shall develop and execute a comprehensive research plan in support of Platform innovation and improvement that encompasses both of these key areas of study; and report findings and recommendations to NCES quarterly.

In addition, the Contractor shall develop and execute a Usability Study Plan for frequent and robust usability study of the Platform, AMS, and NAEPq and their respective audiences; and for reporting and follow-through on findings with NCES and the

Coalition. Any planned changes, innovations, or enhancements shall undergo rigorous usability study prior to implementation. The culmination of these activities shall be the development and continuous maintenance of a NAEP technology roadmap for the innovation and improvement of the Platform and supporting systems.

In addition, the Contractor shall fully support NAEP operational studies and R&D activities, as needed, and at the request of the COR such that studies shall be capable of being integrated and delivered within the context of the DBA Platform and the Contractor shall support technical development, implementation, and administration of all such studies. Resource allocation in support of operational studies and R&D activities shall not impact ongoing development and maintenance of the mainstream NAEP assessment, Platforms, or supporting systems.

Primary Surveillance Method: COR’s review of Task 4 deliverables; and review and performance of the Platform or other technical solutions developed in support of NAEP operational studies and R&D activities.

1. The Annual Research Plan and activities conducted by the Contractor are informative and constructive, encompassing both the industry at-large (large-scale assessment and education technology) and NAEP and Platform performance data, and effectively inform future Platform innovation and improvement.

2. Findings and recommendations from studies and R&D activities are documented and tracked, with thorough reporting and follow-through with NCES and the

NAEP Contractors.

3. Materials prepared and technical solutions developed by the Contractor are of high quality with minimal defects, and are consistent with NCES requirements and expectations for NAEP studies and R&D activities.

4. Materials prepared by the Contractor are complete and delivered on time to the

COR.

5. Usability studies conducted are comprehensive, of high quality, and age/user appropriate depending on which system is being studied; and the Contractor demonstrates thorough and consistent reporting and follow-through with NCES and the NAEP Contractors around findings.

6. Smooth administration of NAEP operational studies and R&D activities with minimal technical errors or student-facing issues.

Process of Quality Assurance Assessment Evaluation

In the event of an excusable delay (as defined in FAR 52.249-14), the Department and the contractor shall work together to modify the contract regarding the due dates of deliverables. If an event occurs that would require modifying deliverable due dates, the contractor’s performance, where applicable in this QASP, shall be measured by the date agreed upon in the modification.

In narrative form, the COR will substantiate all adjectival ratings other than Satisfactory.

Performance at the “Satisfactory” level is expected from the contractor. The COR will forward a report of average scores to the CS.