Attachment 4 - QASP Presentatin Layer (08.01.24).pdf

Solicitation: 1305M424R0047 Attachment 4 – QASP

NATIONAL OCEANIC AND ATMOSPHERIC ADMINISTRATION (NOAA)

QUALITY ASSURANCE SURVEILLANCE PLAN

For Office of Space Commerce (OSC).

Contract Number: TBD Contract Description: Traffic Coordination System for Space (TraCSS) Presentation Layer Contractor’s name: TBD (hereafter referred to as the contractor).

1. PURPOSE.

This Quality Assurance Surveillance Plan (QASP) provides a systematic method to evaluate performance for the stated contract. This QASP explains the following:

● Deliverable to be completed (what will be monitored?)

● Performance standard (what is expected?)

● Acceptable quality level (AQL)/compliance level (what constitutes acceptance?)

● Surveillance method/frequency (how will you perform your check: random sampling, customer complaint, etc.?)

This QASP does not detail how the contractor accomplishes the work. Rather, the QASP is created with the premise that the contractor is responsible for management and quality control actions to meet the terms of the contract. It is the Government’s responsibility to be objective, fair, and consistent in evaluating performance.

In addition, the QASP should recognize that unforeseen and uncontrollable situations may occur.

This QASP is a “living document” and the Government may review and revise it on a regular basis. However, the Government shall coordinate changes with the contractor. Updates shall ensure that the QASP remains a valid, useful, and enforceable document. Copies of the original QASP and revisions shall be provided to the contractor and Government officials implementing surveillance activities.

The following FAR clauses may apply depending on contract type:

52.212-4(a) Inspection/Acceptance.

2. GOVERNMENT ROLES AND RESPONSIBILITIES.

The following personnel shall oversee and coordinate surveillance activities.

a. Contracting Officer (CO) - The CO shall ensure performance of all necessary actions for effective contracting, ensure compliance with the contract terms, and shall safeguard the interests of the United States in the contractual relationship. The CO shall also assure that the contractor receives impartial, fair, and equitable treatment under this contract. The CO is ultimately responsible for the final determination of the adequacy of the contractor’s performance.

Assigned CO: Noelle Albert Organization or Agency: AGO/CSAD Telephone: 301-327-0912 Email: Noelle.Albert@noaa.gov

b. Contracting Officer’s Representative (COR) - The COR is responsible for technical administration of the contract and shall assure proper Government surveillance of the contractor’s performance. The COR shall keep a quality assurance file. At the conclusion of the contract or when requested by the CO, the COR shall provide this documentation to the CO. The COR is not empowered to make any contractual commitments or to authorize any contractual changes on the Government’s behalf. The contractor shall refer any changes they deem may affect contract price, terms, or conditions to the CO for action.

Assigned COR: TBD Telephone: TBD Email: TBD

3. CONTRACTOR REPRESENTATIVES:

The following employees of the contractor serve as the contractor’s representatives for this contract.

a. Program Manager - TBD Telephone: TBD Email: TBD

b. Task Manager – TBD Telephone: TBD Email: TBD

c. Other Contractor Personnel - <upon award, enter name or delete these lines if not applicable> Title: TBD Telephone: TBD Email: TBD

4. PERFORMANCE STANDARDS.

Performance standards define desired services. The Government performs surveillance to determine if the contractor exceeds, meets or does not meet these standards.

The Performance Requirements Summary, Appendix A in the Performance Work Statement includes performance standards. The Government shall use these standards to determine contractor performance and shall compare contractor performance to the Acceptable Quality Level (AQL).

Service Output Performance Standard Acceptable Quality Level

(AQL)

As delivered displays Displays must be reliable and available to support operations per the parameters specified for the operational system.

Availability with full operational capability better than 99.95% of the time (i.e. 4.38 hours of unscheduled downtime per year). Mean time to recovery from a failure in one hour with one standard deviation of 20 minutes.

Tested Code Code delivered under the order (including scripts and configuration files, where appropriate) must have substantial test code coverage and a clean code base. Version-controlled, shared repository of code

Minimum of 90% test coverage of all code comprising the product, which will remain in the government domain.

Properly Styled Code Appropriate format guide for language. 0 linting errors and 0 warnings Deployed Code must successfully build and deploy into staging environment Successful build with a single command

Documented All dependencies are listed and the licenses are documented. Major functionality in the software/source code or configuration file is documented. Individual methods are documented inline using comments that permit the use of documentation-generation tools. A system diagram (MBSE) is provided.

Combination of manual review and automated testing, if available

Security OWASP Application Security Verification Standard 4.0, Level 2

Code submitted must be free of medium- and high-level static and dynamic security vulnerabilities

User research Usability testing and other user research methods must be conducted at regular intervals throughout the development process (not just at the beginning or end)

Artifacts from usability testing and/or other research methods with end users are available at the end of every applicable sprint, in accordance with the vendor’s research plan

Website/Display user manual/documentation

Users must be provided sufficient information to navigate the website and the displays independently. Updates to the website and displays must be reflected in help and support documentation.

Easily available and thorough documentation that is sufficient for an untrained individual to navigate the website and access the displays.

5. INCENTIVES.

None

6. METHODS OF QUALITY ASSURANCE SURVEILLANCE.

Various methods exist to monitor performance. The COR shall use the surveillance methods listed below in the administration of this QASP.

https://www.owasp.org/images/d/d4/OWASP_Application_Security_Verification_Standard_4.0-en.pdf

Regardless of the surveillance method, the COR shall always contact the contractor's task manager or on-site representative when a defect is identified and inform the manager of the specifics of the problem. The COR, with assistance from the CO, if needed, shall be responsible for monitoring the contractor’s performance in meeting a specific performance standard/AQL.

a. DIRECT OBSERVATION. (Can be performed periodically or through 100% surveillance.)

b. MANAGEMENT INFORMATION SYSTEMS (MIS). (Evaluates outputs through the use of management information reports. Best used for general surveillance and may need to be supplemented by periodic inspections.)

c. PERIODIC INSPECTION. (Uses a comprehensive evaluation of selected outputs. Inspections may be scheduled [Daily, Weekly, Monthly, Quarterly, or annually] or unscheduled, as required.)

e. VALIDATED USER/CUSTOMER COMPLAINTS. (Relies on the user of the service to identify deficiencies.

Complaints are then investigated and validated. Highly applicable to services provided in quantity and where quality is highly subjective.)

f. 100% INSPECTION. (Evaluates all outputs. Most applicable to small quantity, but highly important services.

May be used where there are written deliverables and stringent requirements such as tasks required by law, safety, or security.)

g. PERIODIC SAMPLING. (Variation of random sampling. However, sample is only taken when a deficiency is suspected. Good follow-up to MIS analysis. Sample results are applicable only for the specific work inspected. Since sample is not entirely random, it cannot be applied to total activity performance.)

h. RANDOM SAMPLING. (Designed to evaluate the outputs of the award requirement by randomly selecting and inspecting a statistically significant sample. Highly recommended for large quantity repetitive activities with objective and measurable quality attributes.)

i. PROGRESS OR STATUS MEETINGS.

j. ANALYSIS OF CONTRACTOR’S PROGRESS REPORTS. (Evaluate cost, schedule, etc.)

7. QUALITY ASSURANCE SURVEILLANCE PLAN TABLE

Service Output Performance Standard

Acceptable Quality Level (AQL)

Method of Assessment

Incentive/ Disincentive

As delivered displays Displays must be reliable and available to support operations per the parameters specified for the operational system.

Availability with full operational capability better than 99.95% of the time (i.e. 4.38 hours of unscheduled downtime per year).

Mean time to recovery from a failure in one hour with one standard deviation of 20 minutes.

Measure of any down-time and subsequent time required for recovery.

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or delegate for review

Tested Code Code delivered under the order (including scripts and configuration files, where appropriate) must have substantial test code coverage and a clean code base. Version-controlled, shared repository of code comprising the product, which will remain in the government domain.

Minimum of 90% test coverage of all code

Combination of manual review by COR or delegate and automated static and dynamic testing. Test cases stress code beyond "happy path."

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or delegate for review

Properly Styled Code Appropriate format guide for language.

0 linting errors and 0 warnings

Combination of manual review by COR or delegate and automated testing

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or

Deployed Code must successfully build and deploy into staging environment

Successful build with a single command

Combination of manual review by COR or delegate and automated testing

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or delegate for review

Documented All dependencies are listed and the licenses are documented. Major functionality in the software/source code or configuration file is documented. Individual methods are documented inline using comments that permit the use of documentation-generation tools. A system diagram (MBSE) is provided.

Combination of manual review and automated testing, if available

Combination of manual review by COR or delegate and automated testing, if available

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or delegate for review

Security OWASP Application Security Verification Standard 4.0, Level 2

Code submitted must be free of medium- and high-level static and dynamic security vulnerabilities

Clean tests from static and dynamic testing, along with documentation explaining any false positives

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or delegate for review https://www.owasp.org/images/d/d4/OWASP_Application_Security_Verification_Standard_4.0-en.pdf

User research Usability testing and other user research methods must be conducted at regular intervals throughout the development process (not just at the beginning or end)

Artifacts from usability testing and/or other research methods with end users are available at the end of every applicable sprint, in accordance with the vendor’s research plan

Manual review by COR or delegate

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or delegate for review

Website/Display user manual/documentation

Users must be provided sufficient information to navigate the website and the displays independently.

Updates to the website and displays must be reflected in help and support documentation.

Easily available and thorough documentation that is sufficient for an untrained individual to navigate the website and access the displays.

Manual review by COR or delegate

Incentive = option period considered if funding available and the need exists.

Disincentive = Corrective action plan to remedy a deficiency to be submitted to the COR and/or

8. RATINGS.

Metrics and methods are designed to determine if performance exceeds, meets, or does not meet a given standard and acceptable quality level. A rating scale shall be used to determine a positive, neutral, or negative outcome. The following ratings shall be used:

EXCEPTIONAL: Performance significantly exceeds contract requirements to the Government’s benefit.

SATISFACTORY: Performance meets contractual requirements.

UNSATISFACTORY

: Performance does not meet contractual requirements.

9. DOCUMENTING PERFORMANCE.

a. ACCEPTABLE PERFORMANCE.

The Government shall document positive performance. A report template is attached. Any report may become a part of the supporting documentation for fixed fee payments, award fee payments, or other contractual actions.

b. UNACCEPTABLE PERFORMANCE.

When unacceptable performance occurs, the COR shall inform the contractor. This will normally be in writing unless circumstances necessitate verbal communication. In any case the COR shall document the discussion and place it in the COR file.

When the COR determines formal written communication is required, the COR shall prepare a Contract Discrepancy Report (CDR), and present it to the contractor's program manager or on-site representative. A CDR template is attached to this QASP.

The contractor shall acknowledge receipt of the CDR in writing. The CDR will specify if the contractor is required to prepare a corrective action plan to document how the contractor shall correct the unacceptable performance and avoid a recurrence. The CDR will also state how long after receipt the contractor has to present this corrective action plan to the COR. The Government shall review the contractor's corrective action plan to determine acceptability.

Any CDRs may become a part of the supporting documentation for contract payment deductions, fixed fee deductions, award fee nonpayment, or other contractual actions deemed necessary by the CO.

10. FREQUENCY OF MEASUREMENT.

a. Frequency of Measurement.

During contract/order performance, the COR or delegate shall take periodic measurements, weekly or monthly as specified in the AQL column of the Performance Standards Summary Matrix, and shall analyze whether the negotiated frequency of measurement is appropriate for the work being performed.

b. Frequency of Performance Assessment Meetings.

The COR shall meet with the contractor at a minimum once a year to assess performance and shall provide a written assessment.

Prepared by: TBD

Signature - Contracting Officer’s Representative Date

PERFORMANCE REPORT

1. CONTRACT NUMBER: TBD

2. Prepared by: TBD

3. Date and time of observation:

4. Observation:

<Examples of items to include in a report are:

- Method of surveillance.

- How frequently you conducted surveillance.

- Surveillance results.

- Number of observations.>

CONTRACT DISCREPANCY REPORT (CDR)

1. Contract Number: TBD

2. To: TBD

3. FROM: TBD

4. Date and time observed discrepancy:

5. Discrepancy or Problem:

<Describe in detail, including reference to contract requirement. Identify any attachments.>

6. Corrective Action Plan:

A written corrective action plan < is / is not > required.

< If a written corrective action plan is required include the following. > The written Corrective Action Plan will be provided to the undersigned not later than < # days after receipt of this

CDR. >

Received by:

Signature - Contractor Program Manager, Task Manager or Date on-site representative

NATIONAL OCEANIC AND ATMOSPHERIC ADMINISTRATION (NOAA)