QASP.docx

U. S. Department of Commerce

National Technical Information Service

QUALITY ASSURANCE

SURVEILLANCE PLAN

Contract No. SB1342-XXXX

Contractor:

TABLE OF CONTENTS

Introduction Purpose This QASP describes the procedures that the Contracting Officer’s Representatives (COR’s) for Contract No. SB1342-XXXX, (hereafter referred to as “the contract”) will use to monitor Contractor performance. It includes, as Appendix A, the deliverables from the PWS included in the contract, and it is important to note NTIS’s primary concern is with the services provided by the Contractor and not with the procedures used to produce them.

Therefore, the QASP focuses on examining the services provided by the Contractor and not the processes used to produce them. It is intended that the QASP be a tool to guide the COR’s in assessing Contractor performance. In some cases, specific metrics are used to measure Contractor performance, in other cases subjective judgment and evaluation by NTIS personnel will be the determining criteria. This plan describes the methodology utilized to make both quantitative and qualitative evaluation of Contractor performance under the contract.

QASP Relation to Contract Within reason, NTIS will retain the right to change the surveillance methods and QASP procedures, or to increase or decrease the degree of surveillance efforts at any time necessary to assure contract compliance.

Revisions to the QASP The QASP is a tool for use in Government administration of the contract. Changes must be made bilaterally between the Government and the Contractor. As the performance period progresses, the levels of surveillance may be altered for service areas in cases where performance is either consistently excellent or consistently unsatisfactory. If observations reveal consistently good performance, the amount of surveillance may be reduced. If observations reveal consistent deficiencies, increased surveillance may be implemented.

Performance Description

The COR’s will monitor the Contractor’s performance through various surveillance methods described in Section 4: Performing Quality Assurance. The COR’s will gather and evaluate performance data to assess Contractor performance against contract requirements.

Performance Standards and Acceptable Quality Levels (AQLs) For selected activities in the PWS, performance standards and AQLs are listed in the Performance Requirements Summary (PRS) in the contract. A performance standard is the expected level of Contractor performance. An AQL defines the level of performance that is satisfactory. Depending on the service evaluated and the evaluation method selected, performance standards and AQLs may be stated as a number of occurrences or as a percentage. Performance standards and AQLs for random sampling and 100 percent inspection are generally stated as percentages. For periodic inspections, performance standards may be stated as either percentages or as absolute numbers.

The contract requires the Contractor to perform all work as specified. Any inaccuracies or omissions in services are referred to as “defects” on the part of the Contractor. The Contractor shall be held responsible for all identified defects. The AQLs take into account that in some instances an allowable level of deficiencies (deviations) is possible while overall performance continues to meet NTIS’s desired level of service.

Allowable Deviation The AQLs define the level or number of performance deficiencies the Contractor is permitted to reach under this contract. AQLs take into account the difference between an occasional defect and a gross number of defects. AQLs can be expressed as a percentage of or as an absolute number (e.g., three per month).

Substantially Complete In some cases, service outputs are evaluated using subjective values (e.g., excellent, satisfactory, unsatisfactory). The criteria for acceptable performance and for defects must be defined for these service outputs. The concept of “substantially complete” should be the basis for inspections based on subjective scales.

Work is considered “substantially complete” where there has been no significant departure from the terms of the contract and no omission of essential work. In addition, the Contractor has performed the work required to the best of its ability and the only variance consists of minor omissions or deficiencies.

Non-Performance Non-performance occurs when the Contractor’s performance does not meet the AQL for a given requirement. Requirements may contain multiple performance elements, and therefore, deficiencies may occur in one or more aspects of performance (e.g., timeliness, accuracy, completeness, etc.) or subject areas of effort.

When surveillance indicates that the Contractor’s service output is not in compliance with the contract requirements, the COR must determine whether the contract or the Government caused the deficiency. If the cause of the defect rests with the Government, corrective action must be taken through Government channels. If the cause of the defect is due to action or inaction by the Contractor, the Contractor is responsible for correction of the problem.

Documentation Thorough documentation of unperformed or poorly performed work is essential for tracking Contractor performance throughout the period of performance. The COR’s, as inspectors, will document deficient work by compiling facts describing the inspection methods and results. A sample reporting form is provided in Appendix B: Contract Discrepancy Report. The COR’s will develop documentation to substantiate nonconformance with the contract. The documentation, together with any recommendations, is forwarded to the Contractor and decides whether to elevate the problem to the Contracting Officer (CO) for corrective action.

Remedial Actions The Federal Acquisition Regulation (FAR) allows for penalties in the event that the Contractor fails to perform the required services. Penalties are defined as those actions taken under the direction of the CO against the Contractor within the general provisions of the contract for nonconformance to the PWS.

The CO will determine the penalty for nonconformance based upon the guidelines in the PRS in the contract.

Roles and Responsibilities

The purpose of QA is to ensure that the customers are satisfied with the services received from the Contractor and to ensure that the Contractor is meeting its obligation to NTIS. The roles and responsibilities of the stakeholders involved in QA are described below.

Contractor Responsibility The Contractor is responsible for delivering services in accordance with the contract. The Contractor is responsible for producing, maintaining, and providing for audit, quality control records and reports and all records associated with the investigation and resolution of customer complaints. The Contractor appoints the Program Manager as the single quality control point-of-contact to act as a central recipient of communication from the Government.

Government Responsibility This section of the QASP briefly defines the duties and responsibilities of key Government personnel involved in contract administration and quality assurance. The key personnel who will be responsible for QA are the CO and the COR(s).

Contracting Officer The CO has the authority to administer the contract. The CO may delegate many of the day-to-day contract administration duties to the COR(s). However, certain contractual actions such as negotiation and issuance of contract modifications, resolution of Contractor claims and disputes, issuance of cure notices (notification that unless unacceptable performance is corrected, the Government may terminate the contract for default, IAW FAR 49.607), issuance of show-cause letters (following a cure notice, requesting facts bearing on the case), termination of the contract, and contract close-out functions are retained by the CO. Administrative actions such as invoice approval and issuance of Contract Discrepancy Reports may be, and normally are, delegated by the CO to the COR(s). The Contractor shall direct all communication regarding questions or issues related to QA and inspection to the CO or the COR(s). The CO is the sole Government representative that can negotiate modifications to the terms of the contract or this QASP.

COR(s) The COR(s), as federal employees within the NTIS Office of the Chief Information Officer, are designated by name and/or position to act as a liaison between the Government and the Contractor on all issues pertinent to the daily operation of the contract. The COR(s) represent the CO in the Contracting Officer’s Representative (COR) functions and therefore are the Contractor’s initial point-of-contact with the Government. If there are problems with Contractor performance, the COR(s) will inform the Contractor of the problems and recommend to the CO that adverse contractual actions are appropriate (e.g., cure notice) if the Contractor fails to correct the problem. Also, the COR(s) must refer differences of contract interpretation to the CO.

Office of the Director Project Leads Office of the Director Project Leads are the Government staff responsible for the on-going projects within Office of the Director. The information gained from the Customer Feedback Program may be used in conjunction with other methods of observation to rate the performance of the Contractor.

Performing Quality Assurance Surveillance Methods The surveillance methods used in the QA process are the Government’s tools to monitor the Contractor’s services. The best means of determining whether the Contractor has met all contract requirements is to inspect the Contractor’s service products and analyze the results.

Further, documented inspection results are an effective tool in contract administration. Inspections either confirm the Contractor’s successful achievement of all performance requirements or highlight areas where defects exist and improvements are necessary.

The surveillance methods described below include: 100 percent inspection, periodic inspection, random sampling, and customer feedback. The number of inspections conducted may be reduced in those instances where the Contractor has established a good performance record. In cases of poor performance, NTIS may increase the level of surveillance and focus on known problem areas. In either case, the reasons for the change in surveillance will be documented.

100 Percent Inspection The 100 percent inspection method requires complete inspection of a contract requirement and will be used for requirements that are especially critical or where there is some reason for suspecting that the performance standard or AQL is not being met (and therefore, should be more closely monitored). Evaluation schedules for 100 percent inspections will be prepared each month.

Application

The COR will review 100% of the scores from the qualified applicants. Monitoring of this task(s) will be done after every review is held. Performance standard will be 100%; a mean score for each qualified applicant is supplied, and the scores given in rank order. For all applicants where scores are not supplied, an explanation or code shall be given (incomplete application, withdrew application, etc).

Likewise, the COR will review 100% of the process of coordinating the information for acceptances/declines of fellowships. The performance standard is 100%, all data needed (documentation of every candidate, when offer went out, when they replied, and/or follow-up, when they accepted or declined, when they contacted NTIS, etc.) must be provided by the Contractor.

Evaluation Procedures

The COR’s will use inspection evaluation sheets and follow a detailed inspection schedule. Observed defects for services monitored by periodic inspection will be totaled for each batch of scores received. For each service, the total number of defects will be compared to the performance standard and AQL.

Periodic Inspection Periodic inspection provides a systematic way of looking at service outputs and forming conclusions about the Contractor's level of performance in accordance with a planned schedule of surveillance. Evaluation by periodic inspection is designed to inspect some part but not all of the products and services being monitored.

Application

The COR will monitor collection of application forms, conducting review of applications and supporting documents from applications on a quarterly basis via review of the monthly reports. The Contractor shall submit each monthly report in accordance with the statement of work 100% of the time.

The COR will monitor disbursement of stipends and relocations allowances and evaluation of Associates’ productivity on a monthly basis.

Evaluation Procedures

To ensure valid results, the COR’s will use periodic inspection evaluation sheets and follow a detailed inspection schedule. Observed defects for services monitored by periodic inspection will be totaled at the end of each month. For each service, the total number of defects will be compared to the performance standard and AQL.

Random Sampling Random sampling evaluation is a quality assurance method designed to evaluate some, but not all, of a specific contract requirement. This method, based on statistical principles, estimates the Contractor’s overall level of performance for a given contract requirement based on a representative sample drawn from a population. Random sampling is most often used when the number of occurrences of a service is very high.

Application

The random sampling procedures are based on those set by the American National Standards Institute (ANSI). The random sampling procedures consider the AQL (maximum allowable deviation from the performance standard), the level (intensity) of the evaluation effort, and the population size. There are two ways of applying random sampling for QA surveillance. The first is used only for performance evaluation and allows deductions to be taken only for observed defects; the second is random sampling for performance evaluation and deduction projection (also called extrapolated deductions), which allows deductions against the whole population based on the inspection of the sample. To obtain valid results, random sampling procedures must be followed precisely.

Performance Standards and AQLs

Performance standards and AQLs may be specified as percentages or absolute numbers.

Evaluation Procedures

Random Sampling is based solely on a statistical analysis whereby a conclusion is drawn about a population based on a randomly selected sample of that population. For the conclusion to be valid, the sample selected must be representative of the population. A truly representative sample can be achieved by ensuring that the sample is selected randomly and the size of the sample is sufficient. A conclusion about Contractor performance can then be made based on the representative sample drawn.

Office of the Director Project Lead Feedback

Validated Office of the Director Project Lead feedback is a quality assurance method based on project lead and Contractor interaction. Project leads continually receive the outputs of Contractor performance and are in a position to evaluate the Contractor on a recurring basis. Because project leads have a clear stake in the quality of Contractor services, they are valuable resource for the QAEs.

Application

Project leads are made aware of contract requirements and monitor the services provided by the Contractor, both positive and negative. Where there is a case of poor performance or non-performance, project leads notify the COR’s. The COR’s then investigate the report and, if found to be valid, document their findings. The number of complaints and resulting inspections depend upon project lead awareness and response. If the complaint is valid and caused by poor performance or non-performance by the Contractor, the Contractor must take appropriate corrective action. A valid complaint is one in which the COR confirms that poor performance or non-performance violates contract requirements.

Project Lead Feedback Process

Upon contract award, the COR’s should send letters to the project lead(s). These letters will inform them of the need for their active participation in the overall Quality Assurance Program. The COR’s will also provide a Customer Feedback Record (sample at Appendix C) for the project lead to use to either document performance problems or identify when superior services are received.

The COR’s will validate the Customer Feedback Records submitted. It is primarily the responsibility of the Contractor to investigate each complaint to determine the problem. While COR’s can also investigate project lead complaints, the responsibility for initial review shall remain with the Contractor. At the Government’s discretion, the COR will investigate problems from project leads and complaints involving major problems with services being provided.

The Contractor shall take action when a Customer Feedback Record is received. If a valid complaint exists, the Contractor shall re-perform the product or service. Corrective actions shall be implemented to prevent the recurrence of similar problems in the future or detect and fix such problems before a service is delivered to a project lead. If the project lead complaint is found to be invalid, the COR’s shall educate the project lead regarding contract requirements as they pertain to the project lead’s expectations.

Evaluation Procedures

The Contractor shall report validated complaints each month, so the COR’s may review the valid complaints and formulate action items if necessary. Trend analysis may be used to test for variations in the number of complaints received each month and identify changes in Contractor performance.

Analysis and Results When the inspections and customer feedback record validations have been completed, the COR performs an analysis of the Contractor’s performance. The purpose of the analysis is to ensure NTIS is receiving high-quality services from the Contractor. COR’s review the results, rate Contractor compliance with the performance standards and AQLs, and characterize the Contractor’s overall performance. Analysis of all types of contract monitoring result in one of the following outcomes: outstanding performance, very good performance, satisfactory performance, or unsatisfactory performance.

Outstanding Performance Outstanding performance is the result of the Contractor substantially exceeding the performance standards with significant achievements and no significant deficiencies. NTIS may reduce its level of surveillance when the COR’s determine that the Contractor provides sustained performance that significantly exceeds the requirements with no significant deficiencies.

Very Good Performance When the Contractor’s performance is very good, performance exceeds acceptable quality levels and achievement(s) exist with no significant deficiencies. Strengths in performance are substantially greater than minor performance weaknesses.

Satisfactory Performance When the Contractor’s performance is good, performance meets acceptable quality levels and deficiencies are correctable without adverse impact to mission accomplishment. Strengths and weaknesses in performance are on balance where any deficiencies are identified and corrected immediately by the Contractor.

Unsatisfactory Performance When the performance for any service does not meet the AQL, the Contractor’s performance is unsatisfactory, and is, therefore, unacceptable. The following responses are available to the COR’s regarding that task/subtask:

· The CO and/or COR meet with the Contractor to discuss discrepancies, trends, and intended corrective measures;

· The level of surveillance is increased until the Contractor demonstrates acceptable performance over a period of time;

· The COR issues a Contract Discrepancy Report for each service that does not meet its AQL;

· Should deficiencies be significant and affect multiple requirements, CO action such as a ‘Cure’ notice may be appropriate.

Appendices

2. TO: (Contractor)

3. FROM: (Name of COR)

DATES

PREPARED

4. DISCREPENCY OR PROBLEM (Describe in Detail. Include PWS references. Attach Continuation Sheet if Necessary.)

5. SIGNATURE OF COR

6. TO: (Name of COR)

7. FROM: (Service Provider)

8. CONTRACTOR RESPONSE AS TO CAUSE, EFFECT, CORRECTIVE ACTION AND ACTIONS TO PREVENT RECURRENCE. (Attach Continuation Sheet if necessary. Cite applicable Contractor QC program procedures or new QC procedures.)

9. SIGNATURE OF CONTRACTOR REPRESENTIVE

10. DATE

11. GOVERNMENT EVALUATION (Acceptance, partial acceptance, or rejection. Attach Coordination Sheet if necessary.)

12. GOVERNMENT ACTIONS (Cure notice, show cause, other.)

CLOSE OUT

QAE

COR

CUSTOMER FEEDBACK RECORD

DATE AND TIME OF COMPLAINT

SOURCE OF COMPLAINT

ORGANIZATION

INDIVIDUAL

NATURE OF COMPLAINT

PWS REFERENCE

VALIDATION

DATE AND TIME SERVICE PROVIDER INFORMED OF COMPLAINT

NAME OF SP REPRESENTATIVE INFORMED OF COMPLAINT

ACTION TAKEN BY CONTRACTOR (Responsible officer)

RECEIVED AND VALIDATED BY

Determination: Complaint Valid Complaint Invalid

Sampling Guide/Inspection Checklist

SERVICE FUNCTION: ________________________________________________________

PWS SECTION: ________________________________________________________

NOTE: E = Excellent Performance S = Satisfactory Performance U = Unsatisfactory Performance N/A = Not Applicable

Performance: Excellent (E), Satisfactory (S), Unsatisfactory (U), Not Applicable (N/A)

Overall Rating Of Inspection (E, S, U, or N/A)

COR Comments:

Contractor Signature: Date: _________________

COR Signature: Date:

Page | 11