AFPSL_Recompete_QASP.pdf

Contract Number:

QUALITY ASSURANCE SURVEILLANCE PLAN

(QASP)

for

AIR FORCE PRIMARY STANDARDS LABORATORY (AFPSL)

and

CALIBRATION TECHNICAL ORDER SERVICES

21 August 2019

Approved:

Procuring Contracting Officer (PCO) Date

DRAFT

PART I - MANAGEMENT

1. Goal

2. Objectives

3. Roles and Responsibilities of Multi-Functional Team Members

4. Roles and Responsibilities of other Stakeholders

5. Assessment of Contractor’s Performance

6. Assessment of Overall Acquisition throughout the Contract Term

PART II - ASSESSMENT

1. Purpose

2. Roles and Key Personnel

3. Surveillance Responsibilities

4. Required Records, Reports, and Forms

5. Functional Area Evaluator Schedule

6. Actual Surveillance

7. Random Sample Surveillance

8. Other Surveillance Methods

9. Notifying the Contractor

10. Revising the QASP

11. Abbreviated Surveillance Schedule

12. Performance Objectives

APPENDIX A

A. RANDOM SAMPLE GUIDES

Preface

Guide #1 Accuracy of Calibrations Guide #2 Accuracy of TO Development Guide #3 Facility Control Guide #4 Maintain AFPSL Standards Guide #5 Service Customers within Allotted Turn-Around-Times (AFPSL) Guide #5a Service Customers within Allotted Turn-Around-Times (TOs) Guide #6 Maintain LABMET Guide #7 Maintain Publication, TO, and CPIN Libraries (TOs) Guide #7a Maintain Publication, TO, and CPIN Libraries (AFPSL) Guide #8 Receipt Processing Guide #9 Store and Locate Equipment Guide #10 Packaging (AFPSL) Guide #10a Packaging (TOs) Guide #11 Inventory Accuracy (GFE and ADPE) Guide #12 Funds and Man-Hour Expenditure Report (CDRL A010) Guide #13 TO Distribution TAT (TOs) Guide #14 Reports required in Technical Exhibit 4 Guide #15 Maintain JCALS Guide #16 Maintain AFPSL Measurement Area Capabilities

SAMPLE GUIDE ATTACHMENTS

Atch 1 AFPSL Audit Checklist Atch 2 TO Development Accuracy Checklist (Used for New & Rewrites) Atch 3 TO Changes/Revisions Checklist Atch 4 TO Library Checklist, Pub., TO CPIN, Comm Data with TO Library Worksheet Atch 5 Quality Assurance Evaluator Schedule (Example only - COR format acceptable.)

Atch 6 Inventory Sample Size, Table I Atch 7 Developing a Random List of Equip. Inventory, Using Random No. Generator Atch 8 Inventory Accuracy & Property Control Checklist Atch 9 Contract Discrepancy Report (CDR), (Letter -- Example only.)

Atch 10 TO Distribution Worksheet Atch 11 JCALS entry Checklist Atch 12 AFMETCAL COR MACD Audit Checklist

SECTION I

MANAGEMENT

1. GOAL

1.1 The overall program goal is to sustain high performance for cost, schedule and quality for

AFPSL operations, Calibration Technical Orders, services, and related support.

2. OBJECTIVES

2.1 The objectives in having this service provided include such requirements as:

2.1.1 Providing primary level metrology and calibration of Air Force PMEL Base reference standards.

2.1.2 Performing specialized testing in support of the Air Force Program Office Research and

Development Laboratories, item managers, single managers and Air Force Metrology and Calibration (AFMETCAL) Program.

2.1.3 Providing calibration technical assistance to lower echelon laboratories.

2.1.4 Performing laboratory scheduling and inventory actions.

2.1.5 Complying with AFMETCAL quality and facility requirements, adhering to TO 00-20-14, as well as higher level quality management system requirements of ISO 17025.

2.1.6 Developing, revising, and distributing calibration Technical Orders (TOs) in support of the entire AFMETCAL Program.

2.1.7 Operating and maintaining a Technical Order Library.

3. ROLES AND RESPONSIBILITIES OF MULTI-FUNCTIONAL TEAM MEMBERS

3.1 Contracting Officer (CO): The CO for this contract will be located on-site and assigned from AFMETCAL. The CO will be assisted by a contract administrator from AFMETCAL. The CO is authorized to contractually obligate the Government in the administration of this contract. The CO ensures the contract is administered IAW the Federal Acquisition Regulation (FAR) and other applicable directives. The CO’s duties and responsibilities are reflected in AFI 63-124, and this document.

3.2 Functional Director (FD): The FD for this contract will be located on-site and assigned from AFMETCAL. The duties and responsibilities of the FD are reflected in AFI 63-124, and this document.

3.3 Contract Officer Representative (COR): The COR is required to evaluate the contractor's performance and document non-compliance. The CORs will perform the surveillance activities set forth in this QASP. The responsibilities of the COR are reflected in AFI 63-124, and this document.

CORs will make their names, organizational address, and phone numbers available to all customers supported by the AFPSL. To avoid confusion within the program office, all unresolved questions, interpretations and disputes shall be addressed to the CO who will provide resolution.

4. ROLES AND RESPONSIBILITIES OF OTHER STAKEHOLDERS

4.1 DCMA: Roles and responsibilities are defined by a documented Memorandum of Agreement and Delegation of Duties. The purpose of this document is to identify AFMETCAL retention and delegation of contract administrative functions. DCMA-Newark, is delegated some limited surveillance duties for portions of FAR Part 42 and other delegations as specified in the Memorandum.

4.2 Award Fee Review Board (AFRB): Roles and responsibilities are defined in a documented Award Fee and Incentive Option Plan. The AFRB consists of members specified in the Plan. These typically consist of a chairperson, an executive secretary/recorder, and advisor members. The AFRB members will evaluate the contractor’s performance by reviewing evaluation reports, considering all information obtained from pertinent sources, and arriving at a fee recommendation. The AFRB obtains contractor data and briefings as required. The AFRB recommends changes to the Award Fee and Incentive Option Plan.

4.3 Contractor: The contactor roles and responsibilities are documented in authoritative contractual documents. Weekly Technical Interchange Meetings and Monthly Metrics facilitate effective communication to enable high levels of performance for cost, schedule, and quality for AFPSL operations and Calibration Procedure Development.

5. ASSESSMENT OF CONTRACTOR’S PERFORMANCE

5.1 The approach, methods and tools the multi-functional team uses to assess the contractor’s performance include surveillance IAW the Quality Assurance Surveillance Plan (QASP) as outlined in Section II and reporting of Monthly Metrics.

6. ASSESSMENT OF OVERALL ACQUISITION THROUGHOUT THE CONTRACT

TERM

6.1 The approach, methods, and tools the multi-functional team uses to assess the acquisition throughout the Contract Term include Weekly Technical Interchange Meetings with the Contractor;

Contractor initiated Program Management Reviews and Semi-Annual Award Fee Review by an AFRB. These are accomplished in addition to surveillance IAW a QASP, as outlined in Section II, and reporting of Monthly Metrics.

SECTION II

ASSESSMENT

1. PURPOSE.

1.1 This Quality Assurance Surveillance Plan (QASP) has been developed to implement AFI 63- 124, Performance Based Service Contracts. It is designed to provide the COR an effective and systematic surveillance method for each listed service on the Service Delivery Summary (SDS) in the AFPSL contract. Any observations made by other than COR personnel are only informational and shall not result in any direct action against the contractor.

1.2 Where appropriate, methods for administering and evaluating services included in the SDS are to be developed by the COR with concurrence of the Contracting Officer (CO).

1.3 The QASP provides a systematic method to evaluate the services the contractor is required to furnish and not the details of how the contractor accomplishes the work. The plan uses a combination of the surveillance methods described in AFI 63-124, which adequately assures the government of the contractor’s performance.

1.4 This QASP is based on the premise that the contractor, and not the government, is responsible for management and quality control actions to meet the terms of the contract. The Indifference Quality Levels (IQL) recognizes that the contractor is not a perfect manager and that unforeseen and uncontrollable problems do occur. Good management and use of an adequate quality control plan will allow the contractor to operate within the acceptable performance limits. CORs are to be objective, fair, and consistent in evaluating contractor performance against the SDS Standards and Performance Work Statement (PWS).

2. ROLES AND KEY PERSONNEL.

2.1 Contract Officer Representative (COR): The COR is required to evaluate the contractor's performance and document non-compliance. The CORs will perform the surveillance activities set forth in this QASP. The responsibilities of the COR are reflected in AFI 63-124 and this document.

CORs will make their names, organizational address, and phone numbers available to all customers supported by the AFPSL. To avoid confusion within the program office, all unresolved questions, interpretations and disputes shall be addressed to the CO who will provide resolution.

3. SURVEILLANCE RESPONSIBILITIES.

3.1 Surveillance shall be performed IAW the SDS, PWS, and Contractors entire Technical Proposal requirements.

3.1.1 The method of surveillance for TO production will be 100% inspection for TO Development New/Rewrite. Surveillance for Revision/Change TOs will be random sampling based upon a tiered scale, 100%, 50%, and 25% (minimum sample of 25) of TOs completed.

Revision/Change surveillance will be 100% inspection until performance has been acceptable for 6 consecutive months then the sampling rate will change to 50%. After continued acceptable performance for 6 consecutive months at 50% sampling, the sampling rate will change to 25% (minimum sample of 25) and will stay at that rate as long as the contractor continues to meet the

SDS requirement. The sampling rate will revert back to 50% or 100% inspection when SDS requirement is no longer met. The level of upward change in sampling will be decided by the COR Chief based upon the magnitude of deficient performance. The results of surveillance will be documented on the TO Cover Sheet, when approved, or on AFTO Form 158, Technical Order Review Comment Sheet when there are deficiencies.

3.1.2 The method of surveillance for the TO Library will be random sampling. Results of surveillance will be documented using the TO Library Checklist as a guide. Results may be documented electronically.

3.1.3 The method of surveillance for Calibration Services (Laboratories) will consist of sampling record-keeping IAW the MACD, and random sampling based on up to 3% of completed items as selected by LABMET. Results of surveillance will be documented on the AFPSL Audit Checklist.

For increasing the laboratory random sampling of completed items above 3% see paragraph 7.4 of this QASP.

3.1.4 The method of surveillance for JCALS entries will be random sampling of TOs approved for print as selected from the AFMETCAL TO Workload database. Results of surveillance will be documented using the JCALS entry Checklist as a guide. Results may be documented electronically.

3.1.5 The method of surveillance for TO Distribution will be random sampling selected from the TO Distribution Report, CDRL A032. Results of surveillance will be documented using the TO Distribution Worksheet as a guide. Results may be documented electronically.

3.2 Guidelines for surveillance documentation are as follows:

3.2.1 (Labs only) The COR shall complete the checklist, which will identify the particular area(s) inspected and what was observed during the inspection.

3.2.2 (Labs only) Entries of “UNACCEPTABLE” (non-compliance with SDS, PWS, and Contractor’s entire Technical Proposal requirements) shall be fully documented and initialed by a contractor representative. An info copy of non-compliance which exceeds the SDS requirements should be provided to the CO and FD within three (3) working days of discovery. The COR shall advise the contractor to immediately contact the CO and FD for resolution should the contractor disagree with the deficiencies noted during surveillance.

3.2.3 (Labs & TOs) A Contract Discrepancy Report (CDR) will be issued the first time the contractor violates contract performance requirements paragraphs for the month or award fee period. The CO will be provided the original CDR for processing within three (3) working days.

Continued unacceptable performance will be addressed in end of month reports. TOs: a summary report will be maintained in the COR file, which will include SDS requirements that were not met, a CDR (if contractor violates SDS requirements), a list of all rejected TOs and their 158s, a monthly metrics report with the contractors signature. (This report contains metrics data to consolidate reporting AFMETCAL requirements. Metrics data is not used for award fee purposes, only those associated with SDS requirements. Labs: after the first CDR is provided to CO, audit reports of failures within the same SDS element will be provided as they occur. No monthly summary report will be provided for the Labs. Completed CDRs shall be provided to the FD, and

CO.

4. REQUIRED RECORDS, REPORTS, AND FORMS.

4.1 COR FILES: Maintain and control records IAW AFI 37-138. It is necessary for the CORs to organize their files so they can efficiently locate essential information. Thorough and complete documentation must be on file and made available to the CO and their designated representatives upon request.

4.2 SUSPENSE FILES: A suspense system shall be established to ensure Contract Discrepancy Reports are submitted through the FD to CO.

4.3 FORMS: Quality Assurance Evaluator Schedule; Contract Discrepancy Report; AFTO Form 158, Technical Order Review Comment Sheet; and Surveillance Activity Checklist.

NOTE

When used, these documents or a suitable substitute may be retained in electronic format. Surveillance Activity Checklists provided in the QASP may be supplemented or revised, where necessary, and consistent with the Performance Work Statement (PWS) and Service Delivery Schedule (SDS) or IAW CO direction.

5. QUALITY ASSURANCE PERSONNEL SCHEDULE.

5.1 The COR will develop, where applicable, a surveillance schedule based on SDS requirements.

5.2 Calibration Services uses LABMET to select the items for random sampling using a formula based on a percentage of completed items. For the requirement of maintaining the AFPSL Measurement Area Capabilities and Lab SDS element 7.01-7.03 the sampling will be based on either random or selected sampling selected by the COR on a monthly basis.

5.3 Scheduled inspections should not interfere with random sampling.

5.4 Management Information Systems (MIS) are used in the surveillance of SDS requirements. The LABMET and Metrology Network (METNET) are considered MIS.

6. ACTUAL SURVEILLANCE.

6.1 Actual surveillance for Calibration Services should be comparable to the contractor’s monthly production except for Calibration Accuracy Major Defects which is comparable to contractor’s six

(6) months (Award Fee period) production. TO Production should be comparable to the contractor’s monthly production. The TO Library surveillance is based on the total inventory. Selections for the TO Distribution surveillance will be based on CDRL A032, TO Distribution Report.

6.2 CORs will perform surveillance IAW this plan and the SDS. The CORs may be required to perform less than or more than the required numbers of inspections. Defectives recorded by the COR during any surveillance may be used to determine the level of contractor performance. If inspections are made outside the randomly selected sample, the findings shall be noted in the Award Fee evaluations, where appropriate.

7. RANDOM SAMPLING SURVEILLANCE.

7.1 Random selection refers to the required service. All characteristics or requirements of the SDS will be applied to each required service (or item). When random sampling is the method of surveillance, record the results of the inspection on the Appropriate Checklist(s) provided in Appendix A attachments. These completed documents become the official Air Force record of the contractor’s performance and shall be retained in COR files.

7.2 When a surveillance observation identifies a defect, the COR shall record it on the checklist.

The specific reason the sample is considered defective must be recorded on the Appropriate Checklist.

7.3 The contractor is required to correct, IAW SDS/PWS, all defects detected during surveillance by the COR. Any defects corrected by the contractor shall still count as a defect.

7.4 During the month, the COR may receive customer complaints about the quality of the services provided by the contractor, may observe unacceptable performance outside of scheduled surveillance or observe unacceptable performance during scheduled surveillance. This data may be used by the COR to identify or support negative trends observed during normal surveillance. These complaints and observations will be noted, investigated, and filed. The CO should be notified of these complaints and observations. They may use this information to adjust the sampling rate, (for the sampling rate of lab items, the sampling rate may be increased up to 15% of monthly production.

This increased sampling rate may be maintained until the CO or FD is assured the quality of end items have improved). Only defects discovered during sample observations will be counted when random sampling is the method of surveillance. Only one surveillance method may be used during an inspection period.

8. OTHER SURVEILLANCE METHODS.

8.1 Services surveyed by methods other than random sampling shall have results of the surveillance documented on the appropriate Surveillance Activity Checklist or customer complaint form. The CO shall be notified of all such noted deficiencies.

9. NOTIFYING THE CONTRACTOR.

9.1 The COR must contact the contractor representative and inform him/her of the unacceptable observations made during scheduled surveillance.

9.2 The contractor representative should be notified of any unacceptable observations made outside of the scheduled surveillance. This can be done by phone notification, e-mail, or face-to-face.

Email or letter is preferred for documentation purposes.

9.3 If the number of defectives recorded on the Surveillance Activity Checklist exceed the monthly performance requirement, the COR will determine the possible cause of this unacceptable performance. If any government action or lack of action caused unacceptable performance, the defects may not be counted. Any determination as to the reason for the defects will not be shared with the contractor.

10. REVISIONS TO QASP.

10.1 Revisions to this surveillance plan are the responsibility of WNMLO. All such revisions, however, shall be coordinated with the FD and CO. For additional guidance, refer to AFI 63-124.

11. PERFORMANCE OBJECTIVES

11.1 Performance objectives are as shown in SDS Technical Exhibit 1.

AFPSL QUALITY ASSURANCE SURVEILLANCE PLAN

SAMPLE GUIDES

CHECKLISTS & WORKSHEETS

12 April 2005

TABLE OF CONTENTS

Preface....................................................................................................................................A-4

Accuracy of Calibrations (Sampling Guide #1) .....................................................................A-6

Accuracy of TO Development (Sampling Guide #2) ............................................................A-8

Facility Control (Sampling Guide #3) ...................................................................................A-9

Maintain AFPSL Standards (Sampling Guide #4).................................................................A-11

Service Customers Within Allotted Turn-Around-Time (AFPSL) (Sampling Guide #5) .....A-12

Service Customers Within Allotted Turn-Around-Time (TOs) (Sampling Guide #5a) ........A-13

Maintain LABMET/PAMS II (Sampling Guide #6) .............................................................A-14

Maintain Publication, TO, and CPIN Libraries (TOs) (Sampling Guide #7) ........................A-15

Maintain Publication, TO, and CPIN Libraries (AFPSL) (Sampling Guide #7a) .................A-17

Receipt Processing (Sample Guide #8)..................................................................................A-18

Store and Locate Equipment (Sampling Guide #9) ...............................................................A-19

Packaging (AFPSL) (Sample Guide #10) ..............................................................................A-20

Packaging (TOs) (Sample Guide #10a) .................................................................................A-21

Inventory Accuracy (Government Property) (Sample Guide #11) ........................................A-23

Funds and Man-Hour Expenditure Report (CDRL A010) (Sample Guide #12) ...................A-24

TO Distributions (Sample Guide #13) ...................................................................................A-25

Reports Required in Technical Exhibit 4 (Sample Guide #14) .............................................A-26

Maintain JCALS (Sample Guide #15) ...................................................................................A-27

Maintain AFPSL Measurement Area Capabilities (Sample Guide #16) ...............................A-28

ATTACHMENTS

AFPSL Audit Checklist .........................................................................................................Atch 1

TO Development Accuracy Checklist (Use for new and rewrites) .......................................Atch 2

TO Changes/Revisions Checklist ..........................................................................................Atch 3

TO Library Checklist, Publications, TO CPIN, Commercial Data with TO

Library Worksheet .................................................................................................................Atch 4

Quality Assurance Evaluator Schedule, (Example only - COR format acceptable)..............Atch 5

Inventory Sample Size, Table 1: ............................................................................................Atch 6

Developing a Random List of Equipment Inventory using EXCEL Random

Number Generator .................................................................................................................Atch 7

Inventory Accuracy and Property Control Checklist .............................................................Atch 8

Contract Discrepancy Report, (Letter - Example only.) ........................................................Atch 9

TO Distribution Worksheet Atch 10

JCALS entry Checklist Atch 11

PREFACE

1. Forms used in auditing the contractor’s AFPSL performance are:

a. AFMETCAL AUDIT Checklist - Used for recording the results of audits of the Metrology Laboratories and support functions (attachment 1). The checklist is used to audit the following Performance Requirements:

1) Accuracy of Calibrations (SDS 1.01 & 1.03)

2) Facility Control (SDS 2.00)

3) Maintain AFPSL Standards (SDS 3.00)

4) Maintain Automated Management System (LABMET/PAMS) (SDS 5.01)

5) Maintain Publication, TO, CPIN Libraries (SDS 6.00)

6) Receipt Processing (SDS 7.01)

7) Store and Locate Equipment (SDS 7.02)

8) Packaging (SDS 7.03)

b. The checklist contains:

1) The contract number,

2) A short description of the requirement being surveyed

3) The Service Delivery Summary (SDS) paragraph numbers (The SDS includes the contract paragraph numbers referencing the requirements)

4) The method of surveillance, and the observations

5) The date, time and measurement area of surveillance

6) The results of the surveillance

7) The signature and title of the individual doing the surveillance

8) Description of unacceptable performance or anomalies

9) Initial/signature of contractor’s Quality Manager or representative, and date

2. The following Checklists and Worksheets are used for auditing TO production. See the individual attachments for details.

a. TO Development Accuracy Checklist (Use for new and rewrites)

b. TO Changes/Revisions Checklist

c. TO Library Checklist, Publications, TO, CPIN, Commercial Data with TO Library Worksheet

d. TO Distribution Worksheet

e. JCALS index Checklist

3. Where the CO requires a notification of scheduled evaluations, a Quality Assurance Evaluator Schedule will be completed and used to notify CO of scheduled audits. Form includes the contract number, week, month/year and days scheduled for audit(s). The form will show the items to be audited.

4. Contract Discrepancy Report (CDR). Report is completed by the COR the first time each month that a SDS requirement is not met by the contractor. An example Contract Discrepancy Letter is provided to define format and content. COR fills in detail including but not limited to discrepancy or problem, described in detail including PWS references. A continuation sheet may be used.

Following FD coordination, forward to CO within 5 days. The CDR will be returned to the CO with the contractor’s response and signature. The COR will provide the government evaluation and recommended government actions. The COR will retain one copy of the CDR and related documentation and responses for the COR files.

5. Property Control and Inventory Accuracy Checklist. Checklist is used for auditing equipment inventories and property control. Checklist includes the Contract Number, brief description, sampling method, date, time completed, Location and audit results. Discrepancies are identified by PIN, S/N Noun Room #, Contractor #, Problem, and POC. COR performing the audit signs each report and presents it to contractor for initials. DRAFT

SAMPLING GUIDE #1

Accuracy of Calibrations (SDS 1.01 and 1.03)

1. Method of Surveillance: Random sampling.

2. Lot Size: Monthly production.

3. Sample Size: Typically 1% of monthly production up to 3% of monthly production.

4. Performance Requirement: Performance SDS 1.01 is acceptable if there is only one major defect noted during an award fee period and two or less minor defects noted during monthly production.

For RADIAC calibration, performance is acceptable if SDS 1.03 requirements are met.

Performance SDS 1.01 is unacceptable if there are two or more major defects noted during an award fee period or three or more minor defects noted during monthly production. For RADIAC calibration, performance is unacceptable if SDS 1.03 requirements are not met

Major and minor defects are defined as:

a. Major defects:

1) The item is unsafe or hazardous to use, or

2) The item does not meet calibration accuracy for all functions certified.

3) As defined in TO 00-20-14 and PWS 2.2.19

b. Minor defects:

Any defect that is not major.

5. Sampling Procedure:

a. Random sampling is computer generated, selecting samples from LABMET. The computer selects from items that have been completed in the month being sampled, excluding commercial workloads.

b. After the item has been selected, determination is made by the flip of a coin whether the audit will be a full audit or a paperwork audit.

6. Inspection Procedure:

a. Notify the contract manager or designated representative, which items are to be checked.

b. Observe the calibration of the item by the designated contractor employee.

c. Observe that proper technical data and software are used.

d. Verify the parameters of the items against the appropriate technical data.

e. Observe that the item is checked for safety compliance.

f. Verify that PAM ID#, Nomenclature, P/N, Calibration Interval, Serial Number, have been correctly entered and AFTO 108/99 and Calibration Certificate properly completed.

g. Document the specific reasons for any unacceptable performance in

REMARKS/COMMENTS of the AFMETCAL AUDIT Checklist.

h. On the AUDIT Checklist, record the Audit Results, PASS or FAIL. If the audit has failed, indicate if the failure(s) are MAJOR or MINOR.

i. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

Request contractor representative initials/signature on the audit sheet.

j. Ensure the contractor immediately corrects, if possible, all defects detected during surveillance. Any defects corrected by the contractor shall still count as defects.

k. The first time that the Performance Requirements of the SDS listed in section 6, paragraph

(h) in this QASP are exceeded, complete a CDR and attach copy(s) of the AUDIT checklist and the deficiencies and forward to CO within 5 days. Retain a COR file copy. If the SDS requirements are exceeded again during the month, another CDR is not required.

m. The six-month period is defined as the applicable Award Fee Period.

SAMPLING GUIDE #2

Accuracy of TO Development

1. Method of Surveillance: 100% inspection TO Developments (New) and (Rewrites), Random for Revision/Changes after six months acceptable performance at 100% inspection.

2. Lot Size: Monthly TO production.

3. Sample Size: Monthly TO production

4. Performance Requirement: All New, Revisions/Changes, and Rewrites are accurate.

a. A major defect is one that does not achieve the required calibration parameters of the item to be calibrated.

b. A minor defect is one that does not affect the required calibration parameters.

c. TO is acceptable if there are no major defects and two or less minor defects noted.

d. TO is not acceptable (Rejected) if there are one or more major defects and/or three or more minor defects noted.

e. Performance is acceptable if there is 1 or less than 10% Rejected New and Rewrites and less than 2% Rejected Revision/Changes noted in monthly production. Refer to SDS 1.02.

f. Performance is not acceptable if there are more than 1 or 10% Rejected New and Rewrite and/or more than 2% Rejected Revision/Changes noted in monthly production. Refer to SDS 1.02.

a. All TOs developed New/Rewrites will be reviewed by CORs. All Revisions/Changes will be reviewed by CORs at a rate based upon a tiered scale, 100%, 50%, and 25% (minimum sample of 25) of TOs completed, for Revision/Change. (Revision/Change surveillance will be 100% inspection until performance has been acceptable for 6 consecutive months then the sampling rate will change to 50%. After continued acceptable performance for 6 consecutive months at 50% sampling, the sampling rate will change to 25% (minimum sample of 25) and will stay at that rate as long as the contractor continues to meet the SDS requirement. The sampling rate will revert back to 50% or 100% inspection when SDS requirement is no longer met. The level of upward change in sampling will be decided by the COR Chief based upon the magnitude of deficient performance.

b. Random sampling is computer generated, selecting samples from TO COR Database. The computer selects from Revision/Changes that have been completed in the month being sampled.

a. The appropriate checklist will be used to for the type of TO being reviewed. Deficiencies are noted along with the type of defect (major or minor).

b. A monthly report is prepared indicating all observed deficiencies and retained in the COR file.

c. All deficiencies are reviewed with the contractor and the contractor’s QA Office or representative is asked to acknowledge receipt of the results.

d. If the Performance Requirements for the month are exceeded, complete a CDR and attach copy(s) of the audit checklist with the deficiencies and forward to CO within 5 working days.

Retain a COR file copy.

SAMPLING GUIDE #3

Facility Control

2. Lot Size: All 57 rooms of the Air Force Primary Standards Laboratories (AFPSL), as listed in the

PWS.

3. Sample Size: Facilities used in calibrating up to three percent of AFPSL monthly production.

4. Performance Requirement: Environmental systems will be in tolerance 90% of the time. All out of tolerance conditions will be annotated and corrective action documented.

a. Performance is acceptable if environment is in control 90% of the time and all out of tolerance conditions annotated and corrective action taken and documented.

b. Performance is unacceptable if environment is out of control more than 10% of the time or where no corrective action has been taken or not documented.

5. Sampling Procedure: Sampling is in consonance with AFPSL production. Random sampling is computer generated, selecting samples from LABMET. The computer selects from items that have completed certification in the month being sampled, excluding commercial workloads. The facility requirements of the laboratory associated with the item being audited are checked at the same time.

a. Notify the Contract Manager or designated representative which laboratory is to be checked.

b. Observe the temperature, humidity, atmospheric pressure, and cleanliness as required by the Laboratory at the time of the audit.

c. The environmental requirements are listed in the PWS.

d. Verify that temperature and humidity recording devices are currently calibrated and all out of tolerance conditions have been annotated during the month.

e. Record in REMARKS/COMMENTS any specific reasons for unacceptable performance during the month.

f. Record the results PASS or FAIL in combination with the rest of the audit, assuring that any discrepancies are recorded as directed in 6.d. above. A major failure could be included if it affected the accuracy of calibration (defect would then be charged to SDS paragraph 1.01 or 1.03, do not count the defect in both places.)

g. Notify the contractor’s Quality Manager or representative of all discrepancies noted. Request Contractor’s representative initials.

h. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

i. The first time in the month that the Performance Requirements are exceeded, complete a CDR and attach copy(s) of the audit checklist with the deficiencies and forward to CO within five (5) days. Retain a file copy. If the SDS requirements are exceeded again during the month, another CDR is not required.

SAMPLING GUIDE #4

Maintain AFPSL Standards

2. Lot Size: The lot size is all standards used by the AFPSL, recorded in the LABMET inventory.

3. Sample Size: Standards used with up to three percent of the monthly production.

4. Performance Requirement: Ensure all AFPSL Standards are maintained and calibrated by

AFMETCAL approved sources. No defects are allowed.

a. Performance is acceptable if one or less defects are noted.

b. Performance is unacceptable if two or more defects are noted.

computer generated, selecting samples from LABMET. The computer selects from items that have completed certification in the month being sampled, excluding commercial workloads. The standards used with the production item being audited are checked at the same time.

a. Notify the contract manager or designated representative which laboratory is to be checked.

b. Determine and verify the required calibration interval and the responsible calibration center.

c. Verify the parameters of the items against the appropriate technical data.

d. Document the specific reasons for unacceptable performance on the AFPSL Audit Checklist.

e. Record the results PASS or FAIL in combination with the rest of the audit, assuring that any discrepancies are recorded as directed in 6.d. above. A major failure could be included if it affected the accuracy of calibration (defect would then be charged to SDS paragraph 1.01 or 1.03, do not count the defect in both places.)

f. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

g. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

h. The first time in the month that the Performance Requirements are exceeded, complete a

CDR and attach copy(s) of the audit checklist with the deficiencies and forward to CO within five (5) days.

Retain a COR file copy. If the SDS requirements are exceeded again during the month, another CDR is not required.

SAMPLING GUIDE #5

Service Customers Within Allotted Turn-Around-Time

(Laboratories)

1. Method of Surveillance: One hundred percent inspection.

3. Sample Size: Monthly production.

4. Performance Requirement: Ensure work flow times do not exceed those standards defined in the

PWS.

a. Performance is acceptable if 1% or fewer times that work flow times exceed the standards of the PWS, per month.

b. Performance is unacceptable if there are more than 1% times that work flow times exceed the standards of the PWS, per month.

a. AFPSL - Data is extracted from LABMET, at the end of each month, for all units shipped in the month being audited. Units include Calibrate and Return, Transfer Standards, AFPSL standards, Exchange Standards, new AF equipment acquisitions (acceptance testing) and FMS equipment. Commercially repaired and/or calibrated equipment (N64) are not included.

a. Determination is made by PAMS ID#, P/N, nomenclature, and TAT of units exceeding the acceptable TAT given in the contract.

b. Time is deleted from the TAT calculations for units that are deferred, changed as the result of an AFMETCAL Project Plan, or as specified in a response to a Request for Disposition Instructions (RDI). TAT restarts for items requiring contract or AFPSL repair, part way through the calibration process, when they return from repair.

c. TAT calculations will consider emergency and mission essential requirements.

d. Data is summarized monthly. If the contract requirements are exceeded, a CDR will be prepared and forwarded to CO within five (5) days.

SAMPLING GUIDE #5a Service Customers Within Allotted Turn-Around-Time

(TOs)

3. Sample Size: Monthly production.

4. Performance Requirement: Ensure work flow times do not exceed those standards defined in the

PWS.

a. Performance is acceptable if there are three or fewer times that work flow times exceed the standards of the PWS, per month.

b. Performance is unacceptable if there are more than three times that work flow times exceed the standards of the PWS, per month.

Audit all TOs produced during the month.

a. Monitor the workflow of all TOs produced by contractor for the month.

b. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

c. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

d. Data is summarized monthly. If the contract requirements are exceeded, a CDR will be prepared and forwarded to CO within five working days.

SAMPLING GUIDE #6

Maintain LABMET

2. Lot Size: Data entered monthly

3. Sample Size: Data entered associated with up to three percent of monthly production.

4. Performance Requirement: Ensure all LABMET entries are accurate for all TMDE supported, and all listings are produced and distributed IAW the PWS

a. Performance is acceptable if deficiencies are three percent or less of monthly production.

b. Performance is unacceptable if there are more than three percent or more deficiencies noted in monthly production.

have completed certification in the month being sampled, excluding commercial workloads.

Data entries associated with the calibration of each sample item are observed during the audit of the item.

a. Notify the contract manager or designated representative which items are to be checked.

b. Observe if data being entered is complete and accurate. Types of entries are associated with the unit being calibrated, including scheduling, shipping and technical data.

c. One or more errors on a single line count as one error.

d. Record the results PASS or FAIL in combination with the rest of the audit, assuring that any discrepancies are recorded as directed in 6.c. above. A major failure could be included if it affected the accuracy of calibration (defect would then be charged to SDS paragraph 1.01 or 1.03, do not count the defect in both places.).

e. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

f. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

g. The first time in the month that the Performance Requirements are exceeded, complete a five (5) days.

SAMPLING GUIDE #7

Maintain Publication, TO and CPIN Libraries

1. Method of Surveillance: Random sampling of documents, on a scheduled, quarterly basis.

2. Lot Size: Total number of publications and software maintained, as of the month being evaluated (Master Inventory List)

3. Sample Size: 25 documents Select 28

4. Performance Requirement: Must be maintained IAW TO 00-5-1, AFI 33-360 and AFI 37-160.

a. Performance is acceptable if there are three or less nonconformities per audit.

b. Performance is unacceptable if there are four or more nonconformities per audit.

a. Sequentially number all documents to be maintained beginning with number 101. An automated list may be used if available (contractor Library Inventory List).

b. Using the EXCEL Random Number Generator, selecting pages and rows, select 28 samples and mark accordingly on the numbered sheet and transfer them from the random sample list to the Random Sample Tally Checklist. This figure includes additional samples to be used when regular samples cannot be accomplished. Document the random sample techniques used on the Random Sample Worksheet and retain with the inspection results for the month.

a. After randomly selecting samples, use the TO Library Checklist, attachment 4, as a guide for auditing the TO Library.

b. Ensure each sample document is:

1) Current, as identified in the JCALS ETIMS TO index.

2) Correctly enter into contractor’s database (if the sample is a TO).

3) Complete and serviceable. If not complete or serviceable, a replacement should be on order.

4) Check CPIN shipping time line (Review contractor’s CPIN Library Distribution Report)

5) Verify that CPINs in the Library match the latest approved versions and dates.

c. Document the specific reasons for unacceptable performances on the back of the tally checklist. One or more errors on a single item will count as one error.

d. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

e. Multi-defects on same selection (data) counts as one.

g. If the Performance Requirements are exceeded, complete a CDR and attach copy(s) of the audit checklist with the deficiencies and forward to CO within 5 working days. Retain a COR file copy. Only one (1) CDR is required for each month when SDS requirements are exceeded.

SAMPLING GUIDE #7a Maintain Publication, TO and CPIN Libraries

(AFPSL)

1. Method of Surveillance: Random sampling

2. Lot Size: All documents used in monthly AFPSL production.

3. Sample Size: All documents used by all AFPSL production used during randomly selected production audits.

4. Performance Requirement: Must be maintained IAW TO 00-5-1, AFI 33-360 and AFI 37-160.

a. Performance is acceptable if there are three or less nonconformities per month.

b. Performance is unacceptable if there are four or more nonconformities per month.

have completed certification in the month being sampled, excluding commercial workloads.

Documents and software used are observed during audit.

b. Record any specific reasons for unacceptable performance during the audit in the

REMARKS/COMMENTS section of the Audit Checklist.

c. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

d. Ensure the contractor immediately corrects, if possible, all defects detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

e. The first time in the month that the Performance Requirements are exceeded, complete a

CDR and attach copy(s) of the audit checklist with the deficiencies and forward to CO within five (5) working days. Retain a COR file copy. If the SDS requirements are exceeded again during the month, another CDR is not required.

SAMPLING GUIDE #8

Receipt Processing

1. Method of Surveillance: Random or selected sampling; on a monthly basis.

2. Lot Size: Monthly receipts.

3. Sample Size: Receipts of monthly production selected for audit.

4. Performance Requirement: 100% of all assets are entered into LABMET records within same day of receipt.

a. Performance is acceptable if there are eight (8) or less defects per month.

b. Performance is not acceptable if there are more than eight (8) defects per month.

5. Sampling Procedure: Select the sample being audited. Receipt information is observed from the LABMET data and receipt data.

b. Record the Job Control Number and whether or not the data was correctly entered within the same day of receipt.

c. Record any specific reasons for unacceptable performance during the audit.

d. An error involving any of the entries for one item counts as one failure.

e. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

f. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

g. The first time in the month that the Performance Requirements are exceeded, complete a five (5) working days.

SAMPLING GUIDE #9

Store and Locate Equipment

2. Lot Size: Monthly receipts

3. Sample Size: Items of monthly production selected for audit.

4. Performance Requirement: One hundred percent of all assets are assigned warehouse locations, and stored within three workdays after receipt (Unless placed directly into work).

a. Performance is acceptable if there are two or less defects observed, during the month.

b. Performance is not acceptable if there are three or more defects observed, during the month.

5. Sampling Procedure: Select the sample being audited. Storage information is collected for the production item being audited.

b. Record if the equipment was stored IAW the contract within three workdays of receipt in assigned location (as required).

c. Record any specific reasons for unacceptable performance during the audit.

d. Equipment not stored IAW the contract, within three days of receipt (as required), counts as one defect.

e. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

f. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

g. The first time in the month that the Performance Requirements are exceeded, complete a CDR and attach copy(s) of the audit checklist with the deficiencies and forward to CO within five (5) working days.

Retain a COR file copy. If the SDS requirements are exceeded again during the month, another CDR is not required.

SAMPLING GUIDE #10

Packaging (AFPSL)

2. Lot Size: Total shipped per month.

3. Sample Size: Items of monthly production selected for audit.

4. Performance Requirement: All items shipped shall be packaged IAW Transportation Packaging Order (TPO) or Special Packaging Instructions (SPI) (AFMJMAN 24-204, AFI 24-202, CFR Title

49. and MIL-STD 2073-1).

a. Performance is acceptable if there is 1% of Lot Size or fewer deficiencies

b. Performance is not acceptable if there are more than 1% defects of Lot Size per month.

5. Sampling Procedure: Select the sample being audited. Observe the shipping process of the production units selected for audit. Packaging of equipment is sampled by observing the processing of the production sample.

b. Ensure the contractor’s scheduler has properly filed out the shipping label/document with correct shipping address, nomenclature, part number, PAMS identification number, and serial number. Verify correct priority was assigned and any special handling requirements are identified. Track the items selected through the packaging process. Ensure the packers are certified if handling RADIAC type items. Ensure that the packers are following established packaging requirements.

c. Record any specific reasons for unacceptable performance during the audit.

d. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

e. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

f. The first time in the month that the Performance Requirements are exceeded, complete a CDR and attach copy(s) of the audit checklist with the deficiencies and forward to CO within five (5) working days. Retain a COR file copy. If the SDS requirements are exceeded again

SAMPLING GUIDE #l0a Packaging (TOs)

1. Method of Surveillance: Random sampling, monthly basis.

2. Lot Size: Total number of TOs shipped per month

3. Sample Size: Total Number of TOs shipped per month

4. Performance Requirement:

a. Performance is acceptable if there is no more than two (2) defects per month.

b. Performance is not acceptable if there are three (3) or more defects per month.

a. Customer complaint notices will be used to determine if correct TOs and number were shipped.

b. Using the Random Number Generator computer program generate numbers to be selected for audit. Use the TO Distribution Report CDRL A032 for identification of TOs. May be the same sample as used in Sample Guide 13

a. Monitor TO delivery complaint notices and/or TO shipment survey to check to see if customer received proper TOs.

b. Compare the TO Distribution Report samples to JCALS ID listing to assure all TODOs and quantities were identified.

c. Document the specific reasons for unacceptable performance on the back of the TO Distribution worksheet.

d. Notify the contractor’s Quality Manager or representative of all discrepancies noted.

e. Ensure the contractor immediately corrects, if possible, all defectives detected during surveillance. Any defectives corrected by the contractor shall still count as defectives.

f. The first time in the month that the Performance Requirements are exceeded for the month, complete a CDR and forward to CO. within five (5) days. Retain a COR file copy. If SDS requirements are exceeded again during the month, another CDR is not required.

SAMPLING GUIDE #11

Inventory (Government Property) Control

1. Method of Surveillance: Random sampling, on a scheduled quarterly basis after six…