Attachment 0002 - QASP - ADA IR.pdf

QUALITY ASSURANCE SURVEILLANCE PLAN

(QASP)

1. Contract or Task Order Title:

AI Data Accelerator, Incident Response

2. Purpose:

This QASP provides a systematic method to evaluate performance for the stated contract. This QASP explains the following:

• What will be monitored.

• How monitoring will take place.

• Who will conduct the monitoring.

• How monitoring efforts and results will be documented.

This QASP does not detail how the Contractor accomplishes the work. Rather, the QASP is created with the prem-ise that the Contractor is responsible for management and quality control actions to meet the terms of the contract. It is the Government’s responsibility to be objective, fair, and consistent in evaluating performance. In addition, the QASP should recognize that unforeseen and uncontrollable situations may occur.

This QASP is a “living document” and the Government may review and revise it on a regular basis. Updates shall ensure that the QASP remains a valid, useful, and enforceable document. Copies of the original QASP and revisions shall be provided to the Contractor and Government officials implementing surveillance activities.

3. Roles and Responsibilities:

The following personnel shall oversee and coordinate surveillance activities:

Program/Project Manager (PM) – The PM provides program oversight and supports the Contract Officer’s Repre-sentative (COR) performance assessment activities.

Contracting Officer (KO) – The KO shall ensure performance of all necessary actions for effective contracting, ensure compliance with the contract terms, and shall safeguard the interests of the United States in the contractual relationship. The KO shall also ensure that the Contractor receives impartial, fair, and equitable treatment under this contract. The KO will determine the final assessment of the Contractor’s performance.

COR – The COR is responsible for providing continuous technical oversight of the Contractor’s performance. The COR uses the QASP to conduct the oversight/surveillance process. The COR shall keep a quality assurance file that accurately documents the Contractor’s actual performance. The purpose is to ensure that the Contractor meets the performance standards contained in the contract. The COR is responsible for reporting early identification of perfor-mance problems to the KO. The COR is required to provide an annual performance assessment to the KO which will be used in documenting Past Performance. The QASP is the primary tool for documenting Contractor perfor-mance. The COR is not empowered to make any contractual commitments or to authorize any contractual change on the Government’s behalf.

4. Methods of Surveillance:

100 Percent Inspection. This is usually only the most appropriate method for infrequent tasks or tasks with stringent performance requirements, e.g., where safety or health is a concern. With this method, per-formance is inspected/evaluated at each occurrence. One hundred percent inspection is too expensive to be used in most cases.

Random Sampling. This is usually the most appropriate method for recurring tasks. With random sam-pling, services are sampled to determine if the level of performance is acceptable. Random sampling works best when the number of instances of the services being performed is very large and a statistically valid sample can be obtained. Computer programs may be available to assist in establishing sampling proce-dures.

Periodic Inspection: These services are monitored weekly, monthly, quarterly, semiannually, and annu-ally. Periodic types of activities are perfect for periodic inspection because not only are they infrequent, but there is normally a predetermined, specified timeframe within which the tasks must be accomplished.

Therefore, you know exactly when to conduct the evaluations. Periodic inspections automatically become 100 percent evaluations or "100 percent checks." Inspections should be divided and scheduled by fre-quency: annual, semiannual, quarterly, monthly, weekly, and as required. Sometimes services are required for which the time or frequency cannot be predicted, such as accident investigations, one-time special task-ing by higher headquarters, etc. These would be labeled "as required inspections." Others are known and predictable such as the Quarterly Status Report or the Monthly Travel Report currently included in some DISA service contracts.

Customer Feedback. Although usually not a primary method, this is a valuable supplement to more sys-tematic methods. For example, in a case where random sampling indicates unsatisfactory service, customer complaints can be used as substantiating evidence. In certain situations where customers can be relied upon to complain consistently when the quality of performance is poor, e.g., dining facilities, building services, then customer surveys and customer complaints may be a primary surveillance method, and customer satis-faction an appropriate performance standard. In all cases, complaints should be documented, preferably on a standard form.

4.1 Surveillance Matrix

The Surveillance Matrix is the list of performance objectives and standards that must be performed by the Contrac-tor. This matrix details the method of surveillance the COR will use to validate and inspect these performance ele-ments. Inspection of each element will be documented in the COR file.

Performance objectives define the desired outcomes. Performance standards define the level of service required un-der the contract to successfully meet the performance objective. The inspection methodology defines how, when, and what will be assessed in measuring performance. The Government shall use performance surveillance, using this QASP, to determine the quality of the Contractor’s performance as it relates to the performance element stand-ards. The Acceptable Quality Level (AQL) performance Standard Table should be used to form the foundation of the COR’s inspection checklist.

Service Category Method & Frequency of Surveillance

Performance Metrics & Standards

AQL Results/Ratings

Task 1 Program Manage-ment

- Periodic Inspection

- Random Sampling

- Customer Satisfaction (all monthly)

Relevant Performance Standards Section 5 Item #1

See values specified in Performance Standards

CS >= 90% / Exceptional 90% > CS >= 80% / Very Good 80% > CS >= 70% / Satisfactory 70% > CS >= 60% / Marginal CS < 60% / Unsatisfactory

Task 2 Provide Program Integration, Train-ing, and End User Support

- Periodic Inspection

- Random Sampling

- Customer Satisfaction (all monthly)

Relevant Performance Standards Section 5 Item #2

See values specified in Performance Standards

CS >= 90% / Exceptional 90% > CS >= 80% / Very Good 80% > CS >= 70% / Satisfactory

Task 3 Data Labeling and Pipeline Manage-ment

- Periodic Inspection

- Random Sampling

- Customer Satisfaction (all monthly)

Relevant Performance Standards Section 5 Item #3

See values specified in Performance Standards

CS >= 90% / Exceptional 90% > CS >= 80% / Very Good 80% > CS >= 70% / Satisfactory

Service Category Method & Frequency of Surveillance

Performance Metrics & Standards

AQL Results/Ratings

Task 4 AI Engineering and Integration Support

- Customer Satisfaction (all monthly)

Relevant Performance Standards Section 5 Item #4

See values specified in Performance Standards

CS >= 90% / Exceptional 90% > CS >= 80% / Very Good 80% > CS >= 70% / Satisfactory

Task 5 Acceleration of Emerging AI Technologies Re-search and Integra-tion

- Customer Satisfaction (all monthly)

Relevant Performance Standards Section 5 Item #5

See values specified in Performance Standards

CS >= 90% / Exceptional 90% > CS >= 80% / Very Good 80% > CS >= 70% / Satisfactory

5. Performance Standards:

Item Performance Standard PWS Acceptable Quality Level (AQL) Method of Surveillance

All subtasks outlined in the PWS section 3.0 – Program Management

3.0.0 thru 3.0.13

The Contractor's work products are suitable to support the full range of analysis as specified in the PWS

The Contractor's personnel are qualified and adept at presenting clear, concise, objective, and factual reports free from conclusions.

Editorial and typographical errors should be few.

Communication is frequent, concise, and effective when working both with the government sponsors and supported organizations.

Work across the entirety of the portfolio to ensure consistent meth-odologies are employed.

No major complaints from customers. Embrace course correction when/where applicable and communicate effectively across the en-tire portfolio.

Refer to section

4.1 Surveillance

Matrix

All subtasks outlined in the PWS section 3.1 - Provide Pro-gram Integration, Training, and End User Support

3.1.0 thru 3.1.10

Provide continuous process improvement from industry best prac-tices.

Communicate observed government shortcomings in a professional manner and provide suggested solutions commensurate with indus-try standards.

Continuously seek customer feedback and make continuous im-provement to drive customer satisfaction ratings higher.

Work across the entirety of the portfolio to ensure consistent meth-odologies are employed.

All subtasks outlined in the PWS section 3.2 - Data Label-ing and Pipeline Management

3.2.0 thru 3.2.6

Document and recover 100% of developed source code and cus-tomized applications.

Ensure thorough and complete documentation is maintained in gov-ernment controlled locations sufficient to support transition to mis-sion partners without the requirement for on the job training or knowledge transfer.

Ensure that performance monitoring infrastructure paces with fea-ture improvement.

Item Performance Standard PWS Acceptable Quality Level (AQL) Method of Surveillance

All subtasks outlined in the PWS section 3.3 AI Engineer-ing and Integration Support

3.3.0 thru 3.3.17

Document and recover 100% of developed source code and cus-tomized applications.

Ensure thorough and complete documentation is maintained in gov-ernment controlled locations sufficient to support transition to mis-sion partners without the requirement for on the job training or knowledge transfer.

Ensure that performance monitoring infrastructure paces with fea-ture improvement.

Work across the entirety of the portfolio to ensure consistent meth-odologies are employed.

All subtasks outlined in the PWS section 3.4 Acceleration of Emerging AI Technologies Research and Integration

3.4.0 thru 3.4.4

Continuously find new and innovative technologies, methods, in-dustry best practices that can be leveraged by the government.

Assist in making recommendations that can help to set strategic courses of action pertaining to the adoption of technologies, meth-ods and industry best practices.

Work across the entirety of the portfolio to ensure consistent meth-odologies are employed.

6. Documenting Performance:

Acceptable Performance.

The Government shall document positive performance. A Performance Assessment Report (PAR) template is at-tached (Appendix 2). Any report may become part of the supporting documentation for fixed price payments, award/incentive fee payments, Contractor Performance Assessment Reporting System (CPARS) entries, or other actions.

Unacceptable performance.

Unacceptable Contractor performance can be identified by a Customer Complaint Record (Appendix 3) or upon schedule surveillance evaluations. When unacceptable performance occurs, the COR shall inform the Contractor.

This will normally be in writing unless circumstances necessitate verbal communication. The COR shall document the discussion and place it in the COR file and provide a copy to the KO.

When the COR determines that formal written communication is required, the COR shall prepare a Corrective Ac-tion Report (CAR). The CAR is forwarded to the KO for dissemination to the Contractor. A CAR template is at-tached to this QASP (Appendix 4).

The Contractor shall acknowledge receipt of the CAR in writing. The CAR will specify if the Contractor is required to prepare a Corrective Action Plan to document how the Contractor shall correct the unacceptable performance and avoid a recurrence. The CAR will also state how long after receipt the Contractor has to present this Corrective Ac-tion Plan to the KO and COR. The Government may withhold payment for services until corrective action is taken.

The Government shall review the Contractor's Corrective Action Plan to determine acceptability.

Any CAR may become a part of the supporting documentation for contract payment deductions, fixed fee deduc-tions, award fee nonpayment, CPARS, or other actions deemed necessary by the KO.

CPARS: (Mandatory if contract is over $1 million) CPARS will be updated documenting contractor performance.

The Assessing Official (AO) will be the COR; and the Reviewing Official (RO) will be the COR Supervisor (COR- SUP). The contractor also has input prior to finalization. The CPARS Focal Point for this contract is TBD. The web-site for the CPARS is: http://www.cpars.gov

7. Mandatory or Regulatory Compliance Items:

The Contractor shall provide initial proof of compliance with regulatory and compliance items contained in the con-tract, e.g. subcontracting, intellectual property, maintaining operational security, training requirements, and standard reporting. The COR and the KO will evaluate the initial submission and will conduct ad hoc reviews of the Contrac-tor's compliance throughout the remainder of the period of performance. The Contractor will monitor compliance as required by Federal Acquisition Regulation 52.222-50, Combating Trafficking in Persons, and report any issues to the KO.

The Contractor shall be responsible for ensuring the safety of all company employees, other working personnel, and Government property. The Contractor is solely responsible for compliance with the Occupational Safety and Health Act (OSHA) (Public Law 91-596) and the resulting applicable standards, OSHA Standard 29 CFR 1910 (general), for the protection, safety, and health of their employees and any subcontractors assigned to the respective task orders under this Agreement. Without Government assistance, the Contractor shall make certain that all safety require-ments are met, safety equipment is provided, and safety procedures are documented per their quality management system.

The Contractor shall execute non-disclosure agreements when it works with sensitive or proprietary information provided from the Government and other contractors not having Government contracts, academic institutions, and other nonprofits. The Contractor agrees to enter into company-to-company agreements to (1) protect another com-pany’s information from unauthorized use or disclosure for as long as it is considered proprietary by the other com-pany, and (2) to refrain from using the information for any purpose other than that for which it was furnished. For information purposes, the Contractor shall furnish copies of these agreements to the COR. These agreements are not intended to protect information available to the Government or to the Contractor from other sources and are fur-nished voluntarily without restriction. This restriction shall be included in all subcontracts, teaming arrangements, and other agreements calling for performance of work. Any Contractor employee granted access to Classified infor-mation must have an SF-312 on file. The Government reserves the right to review the SF-312 at any time.

All Contractor personnel both on and off-site will be required to sign a non-disclosure agreement (NDA). The Con-tractor shall maintain the nondisclosure agreements and have these agreements available for Government review during the life of the contract. At the end of the contract, the Contractor shall turn over to the Government its per-sonnel non-disclosure agreements.

The Contractor shall handle all data received or generated under this Agreement as For Official Use Only (FOUO) material, in accordance with DoDI 5200.01 and NIST SP 800.171. The Contractor shall handle all classified infor-mation received or generated pursuant to the task order DD Form 254 and shall be in compliance with all applicable PWS references and other applicable Government policies and procedures cited herein and in the individual task order.

8. AQL:

Metrics related to compliance with the Performance standards outlined in section 5 shall be reported monthly. The Contractor shall develop, maintain, and report these metrics in order to construct assessments for quality of services delivered and contract performance in a monthly status report.

9. Evaluation Method:

The COR will conduct periodic inspections and random sampling of the deliverables and performance standards re-quired on the contract. Upon failure to meet the AQL for each performance standard or failure to provide the re-quired deliverables, the COR will notify the contractor and document the notice per the “Documenting Perfor-mance” requirements in Section 6 of this QASP.

10. Performance Rating:

In evaluating the quality of the Contractor’s performance, the following performance ratings may be used (same as CPAR ratings).

Performance Rating Criteria

Exceptional

Performance meets contractual requirements and exceeds many to the Govern-ment’s benefit. The contractual performance of the element or sub-element being assessed was accomplished with no problems and Contractor actions were highly effective.

Very Good Performance meets contractual requirements and exceeds some to the Govern-ment’s benefit. The contractual performance was accomplished with few minor problems for which corrective actions taken by the Contractor were effective.

Satisfactory Performance meets contractual requirements. The contractual performance con-tains some minor problems for which corrective actions taken by the Contractor were satisfactory.

Marginal

Performance does not meet some contractual requirements. The contractual perfor-mance reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor’s proposed actions appear marginally effective or were not fully implemented.

Unsatisfactory Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance contains serious problem(s) for which the Contractor’s corrective actions appear or were ineffective.

Appendix 1 - Deleted

Appendix 2– Performance Assessment Report

PERFORMANCE ASSESSMENT REPORT (PAR)

(If more space is needed, use reverse and identify by number)

1. CONTRACT/TASK ORDER

NUMBER

2. CONTRACTOR 3. TYPE OF SERVICES

4. QUALITY ASSURANCE PERSONNEL (COR) SIGNATURE AND DATE

5. COR PHONE 6. SUSPENSE DATE

I. PERFORMANCE

7. DEFICIENCY (CHECK ALL BOXES THAT APPLY)

NEW

REPEAT

NO DEFICIENCY NOTED

8. SERVICES SUMMARY or PWS PARAGRAPH ITEM

REVIEWED

9. BRIEF DESCRIPTION OF DEFICIENCY (IF DEFICIENCY

BOX WAS CHECKED)

10. DETAILED PERFORMANCE ASSESSMENT

II. CONTRACTOR VALIDATION

11. CONTRACTOR REPRESENTATIVE

CONCUR NON-CONCUR

12. CORRECTIVE ACTION ESTIMATED COMPLETION

DATE

13. CONTRACTOR REPRESENTATIVE CORRECTIVE ACTION AND PREVENTION OF RECURRENCE OR REASON

FOR NON-CONCURRENCE OF COR-CITED DEFICIENCY

III. ACTION CORRECTED

14. CONCUR NON-CONCUR COR SIGNATURE AND DATE

15. COR REMARKS (REQUIRED)

6. CONTRACTOR REPRESENTATIVE REMARKS

Appendix 3 – Customer Complaint Record

CUSTOMER COMPLAINT RECORD

DATE/TIME OF COMPLAINT

SOURCE OF COMPLAINT

ORGANIZATION

BUILDING NUM-

BER

INDIVIDUAL

PHONE NUMBER

NATURE OF COMPLAINT

CONTRACT REFERENCE

VALIDATION

DATE/TIME CONTRACTOR INFORMED OF COMPLAINT

ACTION TAKEN BY CONTRACTOR

RECEIVED/VALIDATED BY

Appendix 4 – Corrective Action Report

CORRECTIVE ACTION REPORT (CAR)

(If more space is needed, use reverse and identify by number)

1. CONTRACTOR

2. CONTRACT NUMBER

3. TYPE OF SERVICES

4. FUNCTIONAL AREA

5. SUSPENSE DATE

6. CONTROL NUMBER

7. DEFICIENCY MAJOR MINOR

FINDING:

FINDING IMPACT:

Please respond with a written Corrective Action Plan that details the corrective action of the cited deficiency, the cause of the defi-ciency, and actions taken to prevent recurrence by Suspense Date in Block 5. If date was not entered in Block 5, the Contractor is not required to provide a response.

8. QUALITY ASSURANCE PERSONNEL (COR)

TYPED NAME AND GRADE

SIGNATURE AND DATE

9. ISSUING AUTHORITY

TYPED NAME AND GRADE

SIGNATURE AND DATE

10. COR RESPONSE TO CONTRACTOR CORRECTIVE ACTION AND ACTION TAKEN TO PREVENT RECURRENCE

11. COR DETERMINATION

ACCEPTED REJECTED

12. CLOSE DATE