GADS3_Attachment_J_9__QASP_Final.pdf

ATTACHMENT J-9 QASP

QUALITY ASSURANCE SURVEILLANCE PLAN

for the

GADS III (GADS3) Contract

TABLE OF CONTENTS

1.0 INTRODUCTION

1.1 PURPOSE

1.2 PERFORMANCE MANAGEMENT APPROACH

1.3 PERFORMANCE MANAGEMENT STRATEGY

2.0 ROLES AND RESPONSIBILITIES

3.0 IDENTIFICATION OF SERVICES TO BE PERFORMED

4.0 METHODOLOGIES TO MONITOR PERFORMANCE

5.0 QUALITY ASSURANCE REPORTING

6.0 ANALYSIS OF QUALITY ASSURANCE MONITORING

RESULTS

Table 1: REQUIRED PERFORMANCE METRICS (RPM) TABLE

ATTACHMENT I: QUALITY ASSURANCE MONITORING FORM

ATTACHMENT II: CUSTOMER COMPLAINT INVESTIGATION

ATTACHMENT III: CONTRACT DISCREPANCY REPORT (CDR)

QUALITY ASSURANCE SURVEILLANCE PLAN

1.0 INTRODUCTION

This Quality Assurance Surveillance Plan (QASP) is pursuant to the requirements listed in the Performance Work Statement (PWS) for GADS III. This QASP sets forth the procedures and guidelines the NIEHS will use in evaluating the technical performance of the Contractor.

1.1 PURPOSE

1.1.1. The purpose of the QASP is to describe the systematic methods used to measure GADS III performance and to identify the required reports and the resources to be employed. The QASP provides a means for evaluating whether the Contractor is meeting the performance standards identified in the PWS.

1.1.2 This QASP defines roles and responsibilities, identifies the performance objectives, defines the methodologies that will be used to monitor and evaluate the Contractor’s performance, describes quality assurance reporting, and describes the analysis of quality assurance monitoring results.

1.2 PERFORMANCE MANAGEMENT APPROACH

1.2.1 The PWS identifies required services and expected outcomes instead of how the Contractor should perform those services. Performance management consists of reviewing and analyzing information obtained through performance metrics, and making well-informed decisions based on the analysis of that performance data. The data generated in a performance management approach provides information that indicates whether or not expected outcomes for required services are being achieved adequately by the Contractor.

This QASP defines the performance management approach NIEHS intends to follow to monitor, manage, and take appropriate action on the Contractor’s performance against expected outcomes and performance objectives communicated in the PWS.

1.2.2 Performance management also represents a significant shift from the more traditional Quality Assurance (QA) concepts in several ways. Performance management focuses on assessing whether or not outcomes are being achieved and migrates away from scrutiny on compliance with the processes and practices used to achieve the outcome. The only exceptions to process reviews are those required by law (Federal, State, and local) and compelling business situations such as safety and health. A focus on outcome provides the Contractor with the flexibility to continuously improve and innovate over the course of the contract as long as the critical outcomes expected are being achieved at the desired levels of performance.

1.3 PERFORMANCE MANAGEMENT STRATEGY

1.3.1 The Contractor’s internal quality control system (Quality Control Plan, which is required with proposal submission) will set forth their staffing and procedures for self-inspecting the quality, timeliness, responsiveness, customer satisfaction, and other performance requirements in the PWS. The Contractor will utilize its internal quality control system to assess and report their performance to the designated Government representative.

1.3.2 This draft QASP provides standards and assessment metrics. After award, the draft QASP and the Contractor’s QCP will be reconciled through negotiation to result in a revised QASP that provides the Government the means to assess performance and allows the Contractor to monitor the quality of its performance. The Government can change QASP standards unilaterally at its discretion reflecting changes in priorities, Government requirements, and other considerations. PWS-level QASP performance standards may be reviewed and updated by mutual agreement. Negotiated updates to the QASP will be added to the contract in a bi-lateral contract modification. Reviewing trends and discussing corrective actions and performance successes regularly will allow the QASP to remain relevant and ensure the focus is always to measure and reward the performance that leads to service quality.

2.0 ROLES AND RESPONSIBILITIES

2.1. The Contracting Officer (CO) is responsible for monitoring contract compliance, contract administration and cost control; and resolving any differences between the observations documented by the GADS3 Contracting Officer’s Representative (COR) and the Contractor’s performance.

2.2 The CO will designate one full-time COR as the governing authority for performance management.

2.3 The COR is responsible for monitoring, assessing, and communicating the technical performance of the Contractor and providing technical assistance to the Contractor. The COR will have the responsibility for completing QA monitoring forms (refer to Attachments II and III) used to document inspection and evaluation of the Contractor’s work performance. Government surveillance may occur under the Inspection of Services clause for any service required under the contract.

2.4 The COR will designate Government Technical Monitors (GTMs) who are responsible for assisting the COR with technical monitoring of Contractor activities.

3.0 IDENTIFICATION OF SERVICES TO BE PERFORMED

The Contractor shall provide the NIEHS service and support in accordance with the PWS.

The performance standards are established in the PWS. The QASP Matrix draws directly and verbatim from the Performance Requirements Summary in PWS Section E. The acceptable level of performance is communicated in the acceptable quality level (AQL).

4.0 METHODOLOGIES TO MONITOR PERFORMANCE

4.1 Various methods exist to monitor performance. The COR shall use the surveillance methods listed below in the administration of this QASP.

Regardless of the surveillance method, the COR shall always contact the Contractor’s program manager or on-site representative when a defect is identified and inform the manager of the specifics of the problem.

a) Direct Observation – can be performed periodically or through 100% surveillance.

b) Management Information System – evaluates outputs through the use of management information reports.

c) Periodic Inspection – uses a comprehensive evaluation of selected outputs;

inspections may be scheduled (daily, weekly, monthly, quarterly, or annually) or unscheduled.

d) Validated User/Customer Complaints – relies on the user of the service to identify deficiencies; complaints are then investigated and validated.

e) 100% Inspection – evaluates all outputs, particularly where there are written deliverables and stringent requirements such as tasks required by law, safety, or security.

f) Periodic Sampling – sample is only taken when a deficiency is suspected.

g) Random Sampling – designed for randomly sampling and inspecting a statistically significant sample, particularly for large quantity repetitive activities with objectives and measurable quality attributes.

The Government may review Contractor data, whether in Contractor deliverables or via the Knowledge Management Portal, in emails, and in any other documentation or records, in order to conduct surveillance. The Government will use the GTM to conduct surveillance and to assist the COR in verifying measurement of Contractor performance against the AQLs.

4.2 The Contractor is expected to establish and maintain professional communication among its employees and customers. The primary objective of professional communication between employees and customers is customer satisfaction. Customer satisfaction is the most significant external indicator of the success and effectiveness of all services provided and can be measured through customer feedback and complaints. Performance management drives the Contractor to be customer-focused through initially addressing customer complaints and investigating the issues and/or problems.

The customer always has the option to communicate complaints to the COR as opposed to the Contractor. The COR will accept the customer complaints and will investigate using the Quality Assurance Monitoring Form – Customer Complaint Investigation identified in Attachment II.

4.3 The AQLs for Contractor performance, located in Table 1, Required Performance Metrics Table, are structured to allow the Contractor to manage how the work is performed while providing negative incentives for performance shortfalls. All AQLs are keyed to the relative importance of the task to the overall mission performance as well as other factors.

5.0 QUALITY ASSURANCE REPORTING

5.1 The performance management feedback loop begins with the communication of expected outcomes. Performance standards are expressed in the PWS and measured by the required performance metrics in Table 1.

5.2 The Government’s QA monitoring accomplished by the COR will be reported to the CO on a monthly basis using the monitoring forms in Attachments I and II. The forms, when completed, will document the COR’s evaluation of the Contractor’s performance under the contract to ensure that the PWS requirements are being met.

5.2.1 The COR will provide a copy of all completed QA monitoring forms to the Contractor and will retain a copy for the Government’s records.

6.0 ANALYSIS OF QUALITY ASSURANCE MONITORING

RESULTS

6.1 The Government will use the observation methods described herein to determine whether the AQLs have been met. When necessary based on evaluation results, the Government will apply the applicable negative incentives to adjust the Contractor’s monthly payment.

6.2 At the end of each month, the COR will prepare a written report for the CO. This monthly report shall state the overall results of the quality assurance monitoring of the Contractor’s performance. The written reports will consist of the completed Quality Assurance Monitoring Forms (Attachments I and II) and will become part of the QA documentation.

6.3 The CO may require the Contractor’s Program Manager, or a designated alternate, to meet with the CO and other Government personnel as deemed necessary to discuss performance evaluation results. This may take place during weekly status meetings. The COR will define a frequency of in-depth reviews with the Contractor, however, if the need arises, and the Contractor will meet with the CO as often as required or per the Contractor’s request. The agenda of the reviews may include discussion of:

• Progress against contract requirements

• Monthly performance metrics

• Issues and concerns of both parties

• Projected outlook for upcoming months

• COR recommendations based on Contractor information

• Issues arising from independent reviews and inspections

6.4 The COR will coordinate and communicate with the Contractor to resolve issues and concerns of unacceptable performance. This will normally be in writing unless circumstances necessitate verbal communication. In any case the COR shall document the discussion and place it in the COR file.

When the COR determines formal written communication is required, the COR shall prepare a Contract Discrepancy Report (CDR), and present it to the Contractor’s program manager or on-site representative. A CDR template is attached to this QASP.

The Contractor shall acknowledge receipt of the CDR in writing. The CDR will specify if the Contractor is required to prepare a corrective action plan to document how the Contractor shall correct the unacceptable performance and avoid a recurrence. The CDR will also state how long after receipt the Contractor has to present this corrective action plan to the COR. The Government shall review the Contractor’s corrective action plan to determine acceptability. Any CDRs may become a part of the supporting documentation for contract payment deductions or other actions deemed necessary by the CO.

Table 1: REQUIRED PERFORMANCE METRICS (RPM) TABLE

Performance Objectives Group 1: Lifecycle Maintenance Incident Response and Resolution Times

Required Service Performance Standard

Acceptable Quality Level

(AQL)

Monitoring Method

Incentives (Negative) (Impact on Contractor

Payments) Monthly Resolution of Severity 1 Incidents.

Two or more Severity 1 problems are resolved in over 4 hours in a given month.

No more than one (1) Severity 1 Incident is resolved in over 4 hours in a given month.

Direct Observation (100% Surveillance)

10% of Monthly Invoice

Monthly Resolution of Severity 2 Incidents.

Less than 95% of Severity 2 problems are resolved in 8 hours in a given month.

95% or greater of Severity 2 incidents are resolved within 8 hours in a given month.

3 Incidents.

Less than 95% of Severity 3 problems are resolved in 4 calendar days in a given month.

95% or greater of Severity 3 incidents are resolved within 4 business days in a given month.

5% of Monthly Invoice

4 Incidents.

Less than 95% of Severity 4 problems are resolved in 20 calendar days in a given month.

95% or greater of Severity 4 incidents are resolved within 20 business days in a given month.

Monthly Aggregate Pre- Resolution Response Times – Initial, Estimation, Subsequent

Less than 95% of Monthly Initial Response, Estimation Response, and Subsequent Response times are met in a given month.

95% or greater of Monthly Initial, Estimation and Subsequent Incident response times are within the parameters set forth in PWS Section C.7.1.1.8.4, Table 4.

Periodic Inspection (monthly)

Monthly Report Maximum Aging, Incident.

Any problem is older than 60 calendar days.

No incident is older than 60-calendar days, reference from date opened.

Performance Objectives Group 2: Lifecycle Maintenance Application Availability

Required Service Performance Standard

Acceptable Quality Level

Payments) Monthly Application Availability – Critical Applications

Availability falls below 99.8% as measured across the month during regular business hours.

Availability meets 99.8% or greater as measured across the month during regular business hours.

Monthly Application Availability – Important Applications

Availability falls below 99% as measured across the month during regular business hours.

Availability meets 99% or greater as measured across the month during regular business hours.

Monthly Application Availability – Supportive Applications

Availability falls below 98% as measured across the month during regular business hours.

Availability meets 98% or greater as measured across the month during regular business hours.

Performance Objectives Group 3: DME Project Performance Objectives

Required Service Performance Standard Acceptable Quality Level

Payments) Conventional, Directive Project Tracking:

Project or Task Order Milestone Completion Report

Delivery of project or milestone is more than 20% greater than planned (baseline) schedule.

Delivery of project or milestone is equal to or less than 120% of planned (baseline) schedule.

100% Inspection 10% of hours for the milestone multiplied by the average hourly billing rate for the

Project or Task Order Milestone, applied against the applicable monthly invoice.

Agile, Adaptive Project or Sprint or Iteration Tracking:

Sprint, Iteration, Release Summary Report

Total elapsed days from the start to the end of sprint are greater than planned.

Total elapsed days from the start to the end of Sprint or Iteration are equal to or less than planned.

100% Inspection 15% of amount for Sprint, or 15% of total planned hours (committed) for Sprint multiplied by the average hourly billing rate for the Sprint or Iteration, applied against the applicable monthly invoice.

Conventional, Directive; Agile, Adaptive Project, Milestone, Sprint, Iteration, Release required documentation.

Contractor provides any required Project, Sprint or Release related documentation on-time, per the Deliverables Schedule in Section F.

Contractor provides any required Project, Sprint or Release related documentation on-time, per the Deliverables Schedule in Section F.

100% Inspection 5% of the amount for Sprint, Milestone, Release or Project, applied against the applicable monthly invoice.

Performance Objectives Group 4: DME Software Quality Performance Objectives

Required Service Performance Standard Acceptable Quality Level

Payments) Success Rate at UAT:

New Functionality.

Less than 80% of user acceptance test cases in this month pass on the first iteration.

80% or greater of UAT Test Cases pass first iteration.

20% of amount for applicable Milestone

Project, Sprint, Iteration or Release, applied against the applicable monthly invoice.

Success Rate at UAT:

Existing Functionality.

Less than 100% of regression test suite fails.

100% Regression Test success rate.

20% of amount for applicable Milestone, Project, Sprint, Iteration or Release, applied against the applicable monthly invoice.

Post Production – Severity 1 Problem-Incident Report

There is more than a 15% increase in the number of Severity 1 problems for 30 calendar days after release to production.

Less than or equal to a 15% increase in the number of

Severity 1 problems for 30 calendar days after release to production.

15% of amount for

Project, Sprint, Iteration or Release, applied against the applicable monthly invoice.

Post Production – Severity 2 Problem-Incident Report

There is more than a 15% increase in the number of Severity 2 problems for 30 calendar days after release to production.

Less than or equal to a 15% increase in the number of

Severity 2 problems for 30 calendar days after release to production.

10% of amount for

Project, Sprint, Iteration or Release, applied against the applicable monthly invoice.

Post Production – Severity 3 Problem-Incident Report

There is more than a 15% increase in the number of Severity 3 problems for 30 calendar days after release to production.

Less than or equal to a 15% increase in the number of

Severity 3 problems for 30 calendar days after release to production.

10% of amount for

Project, Sprint, Iteration or Release, applied against the applicable monthly invoice.

ATTACHMENT I: QUALITY ASSURANCE MONITORING FORM

1. CONTRACT NUMBER:

2. Prepared by: (Name of COR)

3. Date and time of observation:

4. Service or standard:

5. Survey period (month/year):

6. Observation (to include method of surveillance, frequency of surveillance, surveillance results, and number of observations)

7. Service Provider’s Performance: _______ Meets Standard

_______ Does Not Meet Standard

ATTACHMENT II: CUSTOMER COMPLAINT INVESTIGATION

2. Prepared by: (Name of COR)

3. Date and time of observation:

4. Service or standard:

5. Source of Complaint:

a. Organization:

b. Individual

6. Nature of Complaint

7. Date/Time Contractor Informed of Complaint:

ATTACHMENT III: CONTRACT DISCREPANCY REPORT

(CDR)

2. TO: (Contractor Program Manager of on-site representative):

3. FROM: (Name of COR)

4. Date and time observed discrepancy:

5. DISCREPANCY OR PROBLEM:

6. Corrective Action Plan:

a. A written correction plan is/is not required. __________________

b. If required, the written Corrective Action Plan will be provided to the undersigned no later than: (# days after receipt of this CDR).

Prepared by: (COR’s name)

COR’s Signature ___________________________ Date _____________

_________________________________________ Date _____________ Contractor Program Manager or on-site representative Signature