FA4890-13-R-0117_Draft_QASP_5_Dec_13.pdf

Uncontrolled when printed or removed from the AMIC QA Community of Practice (QA SharePoint).

Please refer to the QA SharePoint to validate you are using the latest version of this document.

HQ ACC ACQUISITION

MANAGEMENT AND

INTEGRATION CENTER

25 June 2013

Forward Operating Location

Base Operating Support

FOL-BOS (F44890-xx-C-xxxx)

Quality Assurance Surveillance

Plan

COMPLIANCE WITH THIS PLAN IS MANDATORY

NOTICE: This Publication is available in digital format via the ACC AMIC QA SharePoint.

//signed/wjm/5 Sep 13// //signed/xxx/xx xxx xx//

OPR/Prepared by: Mr. William McKee (QAM) Reviewed by: John Metzger

ACC AMIC/DRQM ACC AMIC/PMSF (PM)

//signed, xxx/xx xxx xx//

Received by: Ms. Christin Lockhart

ACC AMIC/PKB (CO)

Revision: 0 Pages: 48

Distribution: F

Table of Contents

1.0 AUTHORITY

1.1 SQA SharePoint

1.2 Purpose

1.3 Application

1.4 Summary of Revisions

1.5 Related Documents

2.0 ROLES

3.0 RESOURCE MANAGEMENT

3.1 AMIC Guidance & Policy

3.2 Duty Hours

3.3 Manning

3.4 Training

3.5 Automated Data Processing Equipment (ADPE)

3.6 Applications

3.6.1 Government SharePoint (QA/PM)

3.6.1.1 QA SharePoint

3.6.1.2 FOL SharePoint

3.6.3 Contractor’s SharePoint

3.6.4 Contractor Applications

3.8 Records Management

3.9 Vehicles

4.0 AUDIT PROGRAM EXECUTION PROCESS

4.1 Introduction

4.2 General Policy

4.2.1 Periodic Meetings

4.2.2 COR Limitations

4.2.3 COR Audits

4.2.4 Audit Key Points

4.3 Audit Performance

4.3.1. Audit Preparation

4.3.1.1 Initial Preparation and General Instructions

4.3.1.2 Preparing for the Audit

4.3.1.3 Points to Remember

4.3.2 Performance Phase

4.3.3 Audit Process/Concepts

4.3.4 Analyze Data (Before, During and After an Audit)

4.3.5 Data Patterns & Trends

4.3.6 Assess Performance

4.3.7 Nonconformances

4.4 Audit Results Analysis

4.5 Audit Reporting

4.5.1 Nonconformances (Findings)

4.6 Control of Nonconformance

4.6.1 Major Nonconformance

4.6.1.2 CAR Routing

4.6.1.3 Track Corrective Action Requests

4.6.1.4 Accepting Contractor’s CAPA/Closing the CAR

4.6.2 Minor Nonconformance(s)

4.6.2.1 Open Minor Nonconformance(s)

4.6.2.2 Minor Nonconformance Suspenses

4.6.2.3 Minor Nonconformance Corrective/Preventive Actions

4.6.3 Re-validate & Re-verify Audit Actions

4.6.4 Communicate Closure Actions & File Audit Report

4.7 Records Maintenance

5.0 INTERNAL PRODUCTS

5.1 Internal Product Compliance

6.0 AWARD FEE/INCENTIVE FEE

6.1 Purpose

7.0 Risk

7.1 Risk Training

7.2 Planning Surveillance

7.3 Risk Based Surveillance

8.0 QA DELIVERABLES

ATTACHMENT 1 Service Summary

SS Item No. 1 SS Item No. 2

SS Item No. 3 SS Item No. 4 SS Item No. 5

SS Item No. 6 SS Item No. 7

SS Item No. 8 ATTACHMENT 2 Acronyms, Definitions, and Websites

Section A - Acronyms

Section B - Definitions Section C - Websites

ATTACHMENT 3, CDRL MATRIX (CDRLs are located in Appendix B of the Contract)

1.0 AUTHORITY

The Federal Acquisition Regulation (FAR) provides the Government the authority to audit the

Contractor. Clause 52-246-X, where X is dependent on what the Contractor is providing (supplies, services, R&D, etc and whether it is firm fixed price or cost reimbursable) provides the exact verbiage. This link will take you to the Hill AFB FAR website; on the left side of the page, select

52246 and you will be provided with detailed information covering your application(s). Remember, your contract may use multiple clauses; typically you can look in Section E (Inspection and

Acceptance) of your contract to locate the applicable clauses.

For the FOL program, the following clauses apply (From section E of the contract):

52.246-3 Inspection Of Supplies Cost-Reimbursement MAY 2001

52.246-4 Inspection Of Services--Fixed Price AUG 1996

52.246-5 Inspection Of Services Cost-Reimbursement APR 1984

52.246-11 Higher-Level Contract Quality FEB 1999

52.246-15 Certificate of Conformance APR 1984

52.246-16 Responsibility For Supplies APR 1984

1.1 SQA SharePoint

This Quality Assurance Surveillance Plan (QASP) has been developed to implement AFI 63-138, AFI 63-501 Air Force Acquisition Quality Program and Mandatory Procedure MP5301.602-2(d).

The QASP is applicable to personnel performing contract surveillance audits IAW FAR Part 46, AFI 63-501, Mandatory Procedure MP5301.602-2(d), and organizational policy. It is designed to provide the Contractor Officer’s Representative (COR) a systematic surveillance method for each service identified in the contract, for the Air Combat Command (ACC) Operation, Maintenance, and Support of the Forward Operating Location Base Operating Support (FOL BOS) contract.

Methods for administering and evaluating other provisions of the contract are to be developed by the Flight Chief Quality Assurance (FCQA), and/or Quality Assurance Manager (QAM), the applicable Functional Area Manager (FAM) or Subject Matter Expert (SME), and approved by the

Contracting Officer.

1.2 Purpose

This document defines the surveillance policy, procedures, and associated methods used by the

Quality Assurance Division for planning, preparing, performing, analyzing, and reporting

Contractor performance.

1.3 Application

To all parties involved with assessing Contractor performance of the FOL BOS contract. This document is for the use of the FCQA/CORs/auditors and contracting officer. The Contractor may receive a courtesy QA SharePoint in order to further enhance communications; however, this is at the discretion of the QAM.

1.4 Summary of Revisions

This document has been completely revised and is considered a new document.

1.5 Related Documents

http://farsite.hill.af.mil/vffara.htm

AFI 63-138

AFI 63-501

Contract FA4890-xx-C-00xx (FOL BOS)

ISO 9000:2005

ISO 9001:2008

ISO 9004:2009

Mandatory Procedure MP5301.602-2(d)

Memo dated 29 Mar 2010, Subject: DoD Standard for Certification of Contracting Officer’s

Representatives (COR) for Service Acquisitions

Quality Management System Manual and supporting documents

2.0 ROLES

Acquisition Management and Integration Center (AMIC)

11817 Canon Blvd, Suite 306

Newport News, VA 23606-4516

Contracting Administrator (CA): The individual within the contracting office who performs the day-to-day administration of the contract. The contract administrator may also be the contracting officer if that person has a warrant.

Contracting Officer (CO): (Ms. Liz Walker, DSN 575-3913, Commercial 757-225-3913)

The Government agent authorized to award or administers contracts. Appoints CORs in writing to surveil the Contractor. The contracting officer is the ONLY person authorized to contractually obligate the Government.

Functional Area Manager (FAM): This person is typically in charge of a functional area

(Logistics, CE, etc.). Occasionally referred to as Chief of Logistics/Chief of Civil Engineering.

Program Manager (PM): (Mr. John Metzger, DSN 574-9496, Commercial 757-574-9496) The person in charge of managing the program.

Quality Assurance Division Chief: (Mr. Scott Shelton, DSN 574-9468, Commercial 757-574-

9468) In charge of Quality Assurance Division.

Contracting Officer’s Representative (COR): A functionally qualified person who performs quality assurance functions for a contracted service. Personnel must complete QAPC led training, CO led training, and DAU courses prior to consideration being given for certification.

Quality Assurance Manager (QAM): (Mr. Bill McKee, DSN 574-9461, Commercial 757-574-

9461) Serves as a supervisory quality assurance specialist for contracted aircraft maintenance operations. This position plans, organizes, supervises Contracting Officer’s Representatives, and manages the activities and contract services of the MAJCOM.

Alternate COR: A Functional Area Evaluator (FAE) who is fully trained and qualified in a related functional specialty (e.g., CE, Logistics, System Communications, etc.). Provides technical support/expertise to program CORs through participation in formal CPETs, review

Contract Data Requirement Lists (CDRLs) and review/coordination of COR issued CARs.

Personnel must complete QAPC led training, CO led training, and DAU courses prior to consideration being given for certification.

Subject Matter Expert (SME): The person who is qualified in a related specialty and provides technical support/expertise to a program, but does require QAPC led training, CO led training, and DAU course completion and is not appointed by contracting officer to perform COR duties.

Contractor: Insert Address

A detailed list of Contractor personnel is located at (username/password required):

Insert contractor’s website address

Customers: USSOUTHCOM, 12AF, ACC, 429th Expeditionary Operation Squadron, 612th

Theater Operations Group, 612th Support Squadron.

3.0 RESOURCE MANAGEMENT

Manage resources IAW AF Directives and the Quality Management System Manual, Section 6, Resource Management.

3.1 AMIC Guidance & Policy

All guidance & policy is located on the QA SharePoint. This is a mandatory review folder for all

FCQA’s, supervisors, and is highly recommended for all CORs. This folder contains all AMIC and DRQ policy letters, EPR, decoration, and awards guidance, and a host of additional information that will make your job easier. You should review this folder frequently.

3.2 Duty Hours

All audits will be performed during the contractor's hours of operation. Hours of operation are established by the Commander; however, the contractor has some 24/7 operations and requires contractor staffing for aircraft that arrive/depart outside normal duty hours to include weekends and holidays. Surveillance will be on a random basis of all shifts, weekends, and holidays that the contractor works.

3.3 Manning

The FOL COR force will consist of a FCQA and three CORs at Curacao (OL-C). The FCQA shall have overall responsibility of all CORs. ACC AMIC/DRQM OL-C personnel will be administratively managed through ACC AMIC IAW Memorandum of Understanding between the 612 th

Theater Operations Group (TOG) and ACC AMIC. This document is located in the

FOL-BOS SharePoint (Registration required). Personnel assignment is controlled through AFPC and the QAM as to what AFSC and skill level are required in order to successfully audit the FOL

BOS contract and ensure compliance with the contract. Current information is listed in the table below:

PAS

Code

Position Number PEC AFSC AFSC TITLE

Rank Required

LEICFK29 000247081C 12411A 3E571 Engineering Craftsman MSgt

LEICFK29 004199971C 12411A 2D172 Cyber transport MSgt

LEICFK29 004433861C 12411A 2A672 Aerospace Ground Equipment Craftsman TSgt

LEICFK29 004355071C 12411A 3E771 Fire Protection Craftsman TSgt https://www.my.af.mil/gcss-af/USAF/ep/contentView.do?contentType=EDITORIAL&contentId=c6925EC19CAC20FB5E044080020E329A9&channelPageId=s6925EC1339B20FB5E044080020E329A9&programId=t6925EC2F7AA60FB5E044080020E329A9 https://afkm.wpafb.af.mil/DocView.asp?DocID=7800977

3.4 Training

CORs are required to complete an on-line DAU Courses CLC222 and CLM003 prior to arrival.

Additional training (QAPC led training and CO led training) will be provided by the Flight

Chief/QAM/CO. CORs are also trained in areas of responsibility and duties outside their normal

AFSC to cover areas of the contract where we don’t have career field coverage. Air Force / ACC directives and technical orders and the contractor’s publications will be used to complete OJT in areas where the individual COR lacks extensive knowledge. Training will be documented on an

AF Form 797, and maintained in individual AF Form 623s (or using the Training Business Area

(TBA)). The training plan is located on the QA SharePoint in Quality Management/Training Plan.

This training plan and associate documents are very detailed and should guide you through the entire training process.

3.5 Automated Data Processing Equipment (ADPE)

The FOL BOS Contractor IAW the contract will control all computers and computer-supported equipment. Should the FCQA determine additional ADPE is required to support our mission, this will be coordinated through Bill McKee at DSN 574-9461.

3.6 Applications

3.6.1 Government SharePoint (QA/PM)

There are many SharePoint’s available; the ones we use most frequently are the QA SharePoint and FOL-BOS SharePoint.

3.6.1.1 QA SharePoint

The use of our QA SharePoint (and the program specific SharePoint’s) alleviates the limitations of a share drive. Regardless of your base and location, you can access the QA SharePoint. The QA

SharePoint also allows us to transfer large files between users alleviating e-mail limitations. To gain access to the QA SharePoint click here. Please contact william.mckee@langley.af.mil for permissions. Request access and include the program you will audit. This will send an e-mail to us and you will be given the appropriate access. The best way to understand what is in the QA

SharePoint is to set aside some time and navigate through the various folders and links. There are two key folders that you should be very familiar with; “00. Quality Management System” and

“Programs”. The Quality Management System folder contains all of our mandatory procedures and manuals, templates, forms, etc. The Programs folder will lead you to your specific program, and within that folder, you will see seven mandatory folders. These have been standardized across all programs and are self-explanatory.

3.6.1.2 FOL SharePoint

To gain access to the FOL SharePoint click here and you will be taken to the FOL SharePoint.

Please contact accamic.folbos@langley.af.mil for permissions. This is where the latest QA

SharePoint of the contract is maintained (see left side, Documents/PWS-Current).

3.6.3 Contractor’s SharePoint

mailto:William.mckee@langley.af.mil mailto:william.mckee@langley.af.mil mailto:accamic.folbos@langley.af.mil

3.6.4 Contractor Applications

3.8 Records Management

Each OL must maintain a filing system in accordance with AFRIMS our records management procedure; please reference this document. Additionally, there is an AFRIMS QA SharePoint that will provide you a wealth of information on the subject.

3.9 Vehicles

ACC AMIC/COR OL-C personnel will be supplied vehicles IAW the Memorandum of

Understanding (MOU) between the 612 th

TOG and ACC AMIC. The current MOU mandates two vehicles at Curacao.

4.0 AUDIT PROGRAM EXECUTION PROCESS

4.1 Introduction

The role of the COR is to ensure the contractor is meeting contractual requirements. The COR contribution is comprised of professional, non-adversarial relationships, which includes positive and open communications with our customers and the contractor. The foundation of this relationship is built upon objective, fair, and consistent COR evaluations of contractor performance against contract requirements, the ability to discuss inspection results, trends, and items of mutual interest with the contract. The COR uses the methods in this QASP, contract, and ISO 9001 to achieve this goal. The COR function is responsible for a wide range of audits requirements that effectively measure and evaluate the contractor’s performance. Additionally, this QASP is based on the premise that the Contractor, not the Government, is responsible for management and Quality Control (QC) / Quality Assurance (QA) actions to meet the terms of the contract.

4.2 General Policy

4.2.1 Periodic Meetings

CORs will meet periodically to exchange information and discuss items of importance. Topics of discussion will include, but are not limited to, information from staff meetings, safety, noted weak/problem areas, audit problems, and any reported customer complaints. Additionally, FCQAs, should discuss issues with other FCQAs and their QAM.

4.2.2 COR Limitations

CORs are the "eyes and ears" of the PM and CO relative to actual contract performance.

Remember: CORs WILL NOT DIRECT WORK, DIRECT RE-ACCOMPLISHMENT OF

WORK, CHANGE THE CONTRACT, OR FORMALLY INTERPRET THE CONTRACT. The

CO, in coordination with functional area/technical experts, resolves these types of issues.

4.2.3 COR Audits

It is essential that CORs accomplish sufficient in-depth audits in all areas to measure the quality of contractor performance and ensure the contractor is meeting contract requirements. This is where the Risk Management Plan, your previous audit results to include trends, recent contract https://www.my.af.mil/afrims/afrims/afrims/rims.cfm https://afkm.wpafb.af.mil/DocView.asp?DocID=7333632 https://afkm.wpafb.af.mil/DocView.asp?DocID=7333632 https://www.my.af.mil/afknprod/community/views/home.aspx?Filter=OO-SC-EI-M4 changes, Contractor manpower turnover, etc. comes into play. This is the information (along with discussions with the CO, PM, QAM, and Functionals) that should be used to determine what should be audited and to what depth. Is there really a need to audit Technical Orders (TOs) every single time if you have not found any problems in the last two years? Maybe a sample of

TOs semi-annually would be ample and allow you to focus your audits on critical elements.

Proper planning will ensure you audit the right elements! Remember that as part of our audit preparation we will review the Contractor’s documentation for trends and ensure the Contractor is providing proper corrective and preventive actions. Additionally, compliance with contractor-developed plans and regulations will be part of the surveillance as they are deliverables via a

CDRL or as part of their Quality Management System (QMS).

4.2.4 Audit Key Points

COR audits will be based on contract requirements. Within the contract, the Performance Work

Statement (PWS) will dictate types of audit actions needed and what processes are to be surveyed. Within the PWS an audit method or process is driven by words that stipulate what is expected of the Contractor and drives each inspection.

Points to remember:

The Service Summary (SS) are critical auditing points, they should be audited in accordance with Attachment 1 unless prior coordination with the QAM has taken place

The contract is primarily performance-based NOT TO/AFI driven. Unless a TO, AFI, Regulation, Publication is mandatory (A complete list of Mandatory/Advisory publications can be found in Appendix C of the PWS), the Contractor may develop their own procedures to complete the requirement (which are typically auditable) and can be found XXX o When auditing against a mandatory AFI, TO, Regulation, etc. often times each of these publications reference additional publications and then those publications reference additional publications and so on. Unless those additional publications are mandatory, please contact the QAM prior to auditing against those additional publications.

4.3 Audit Performance

4.3.1. Audit Preparation

There are two parts to this step, which overlap. If you have just arrived on station, you need to take the time to become familiar with the entire contract. This first part is called the “Initial

Preparation and General Instructions” (4.3.1.1) which certainly applies for new CORs, but is also very important to review these prep/instructions frequently as part of your audit preparation and to ensure you stay familiar with the contract. The second part is called “Preparing for the Audit”

(4.3.1.2) which you should typically refer to before performing your audits each month. Finally, “Points to Remember” (4.3.1.3) applies to all our audits and should be reviewed frequently.

4.3.1.1 Initial Preparation and General Instructions

Read the entire contract (available on the FOL SharePoint); not just the PWS

Read this QASP!

Contractor’s data management system/website – take the time familiarize yourself

As a minimum look at the last six months of CDRLs, CO and PMO letters (from this point forward, keep up with these)

Read the contractor’s QMS along with the six mandatory procedures plus contractor’s applicable checklists, work instructions, site operating procedures, etc. that applies to your auditable areas o All the above documents are auditable – many are CDRLs, deliverables, or as a minimum the contractor must adhere to because they are required to be ISO compliant

Read the risk management plan (Government and Contractor)

Perform a thorough site walk around

Don’t assume anything; it is easy to think that it has always been that way so it must be right o If it doesn’t seem right ask your FCQA, QAM, etc

Don’t be afraid to ask questions

4.3.1.2 Preparing for the Audit

Review previous months’ (as a minimum) audit (MARs)

Discuss overall Contractor’s performance with CORs

Review CO and PMO Ltrs

Review what CDRLs are due during the audit period o Refer to Attachment 3 for due dates o Remember that we review all CDRLs for timeliness and accuracy and coordinate with the owning functional as well

Any contractor personnel changes (hiring, firing, resignations, duty changes)?

o Stay in tune with personnel changes and adjust your audit accordingly

Coordinate with the Commander (typically FCQA does this), Contracting, Program Manager, Quality Assurance Manager, FCQA, and Functionals to determine if they have any issues/concerns – add to your audit

Talk with the permanent party and TDY personnel (customers) to get concerns

Look at upcoming PMI’s, work orders in progress (status) o Add to audit as required

Review recent changes to the contract o Be sure to incorporate these changes in your audits o Adjust frequency/depth of audit

Review QA identified trends

SII for this period?

Review 332’s and WOMS (3215s) o Be sure to put these on your audit plan to validate status/conformity to requirements o What shipments are due in – how are they handled; audit their process

Risk Plan

Inputs provided by functionals, PM, CO

Mission changes – increase/decrease in sorties/personnel o Has the contractor’s workload increased/decreased (especially increased with no manpower increases)

Use random sampling (see attachment 1 for table) and document your sample in the audit report. Just “grabbing” a few items is not objective!

MOST OVERLOOKED!!! - Review contractor’s internal audits to review contractor’s findings and status of corrective/preventive actions – follow-up/validate.

o These should always be part of your audit (check latest status just prior to audit) o Remember; if the Contractor as already documented a problem(s), we do not need to create (typically) a Government 1 st notice, 2 nd notice, or CAR. We should document it in our audit report and of course track the finding to closure. If the contactor is not taking appropriate action we can certainly document that as a nonconformance;

typically documented in the QMS arena

Do as much as possible from your desktop prior to the audit (Contractors data management system/website)

Coordinate with Contractor if you want to watch a PMI being performed, audit being performed by the Contractor, etc.

4.3.1.3 Points to Remember

As part of all your audits, plan on working an early morning, evening, and night shift along with audits on the weekends. Check flight schedules and plan these audits accordingly

Document, document, document; if you don’t document it in your audit, it didn’t happen;

keep good notes!

Remember, your audit plan can be adjusted according to the mission

Remain flexible

You can look at the same area more than once in a month if there are concerns

If you are performing a random sample and see potential problems outside the sampling area;

finish your random sample and then focus on the potential problem (Unless it is a critical/imminent safety, environmental or health issue)

Occasionally perform audits with the Contractor’s QC

Always review what the Contractor has audited o We don’t typically double document – if the Contractor has already documented it, we validate they have preformed sufficient corrective/preventive actions.

Audit preparation is the KEY to a successful audit – be prepared

Keep an open mind during the audit process – no preconceived notions

During the audit listen, listen, listen, don’t judge

When you write your audit report, your words should paint a picture and be OBJECTIVE in your writing

Contract requirements encompass the entire contract, not just the PWS

Contract requirements include most of the contactor’s plans and procedures and are typically auditable

It is critical that you coordinate with your Quality Assurance Manager, functionals, Program

Manager, Contracting Officer prior to performing your audit. They may have concerns/areas of interest they would like us to add to our audit. Examples include; LG may want us to validate turn in items, CE may want us to validate the status of a 332 project, and SC may want us to validate the status of a 3215 project. A good working relationship is a must.

As part of audit preparation CORs must review:

o Contract o Previous audits o Contracting Officer and Program Management Office (contractor) letters o Contractor’s documents (QMS, CDRLs, Operating Manuals, Risk Management Plan, Contractor QA reports/trends) o Applicable publications/regulations o Contractor’s QC audits

Do as much as you can from your desktop prior to starting the actual audit. Most contractors now have websites that contain a wealth of information. Typically, a COR can review all CO

& PMO letters, work orders, purchase orders, PMI schedules, etc. You will be much better prepared and can focus your audit accordingly.

CDRLs are always a concern as to who should audit them. Before the contract started (and refreshers as needed), the CORs, PM, and functionals determine who is responsible for each

CDRL. This information can be found in Attachment 3 of this Quality Assurance

Surveillance Plan. As CORs we look at all CDRLs; however, we must coordinate with the appropriate functional to ensure the data accurate/complete.

If you are performing a multiple-day audit, document as you go; waiting until you have completed the audit to start your documentation typically results in missed information and rushed/poor quality products.

If questions arise contact the FCQA, QAM, or fellow CORs.

The audit plan can be adjusted according to the mission.

The same area may be audited more than once in a month if there are concerns.

Occasionally plan to perform audits with the Contractor’s QA.

Coordinate with the Contractor if evaluating a PMI/audit/etc. being performed by the

Contractor.

4.3.2 Performance Phase

This is where the COR/auditor interfaces with the contractor’s employees. Communication, (both verbal and nonverbal), appearance, and a mature disposition are key elements and shall be of the highest quality. CORs/auditors shall focus all efforts to ensure a professional job. Ethics, COR/auditor “Do’s and Don’ts” learned in QAPC led training COR training provide the boundaries or rules of engagement. Strict adherence to CO guidelines and common sense ensure success in this phase. Performance phase starts with an effective (brief, but comprehensive) opening meeting describing what is to be accomplished, where and by whom. Normally, you will be briefing the functional area chief (i.e. AGE Supervisor). Remember, during the audit process…trust, but verify!

4.3.3 Audit Process/Concepts

During your audit, keep the following process/concepts in mind:

Treat the Contractor the way you would want to be treated. Remember those inspectors that have showed up in your office, asked questions, immediately started writing notes, and then left without telling you anything – that is not the way to do business!

Put the interviewee at ease, be non-threatening, and be very sensitive to language barriers

(you may need to follow-up with an American supervisor)

Fully explain what you are doing and why – starting off by asking questions seldom puts the interviewee at ease, typically they will take a defensive role

Ask the employee to explain their job, you can learn a lot o Listen, listen, listen

Don’t get tunnel vision; we are often too checklist driven!

o Tunnel vision example; you are validating that a PMI has been performed so you look in the contractor data management system and perform a visual check – all is good o The proper way example; you are validating that a PMI has been performed so you look in the data management system, ensure it was completed and properly documented, and perform a visual check; the PMI required the use of a oscilloscope, you validate the oscilloscope was on site, in calibration, and in working condition, ensured any calibration limitations did not affect the PMI; you validate the technician that performed the PMI was on-duty that day; you review the technician’s training record to ensure he is qualified and all ancillary training is up to date; parts were used, you validate that parts were used, removed from supply correctly and reordered as required.

If you are watching an employee perform a preventive maintenance action, repair action, etc.

give the Contractor “space” and ensure you do not have five auditors standing over the employee observing; this would make anyone nervous!

Once you have completed the required action(s) with the employee provide a mini-out-brief.

If the employee is meeting contract requirements, let him/her know. If you found problems or potential problems that require research, be up front with the employee. If you need to perform research, once you have completed it, follow-up with that employee and provide results

Don’t jump to conclusions; gather the data and then analyze the facts – be objective!

Be flexible; mission priorities change frequently

4.3.4 Analyze Data (Before, During and After an Audit)

CORs shall gather data as objective evidence and analyze it to determine compliance to contract requirements, quality objectives and/or standards. The FCQA and/or the person performing the audit shall perform analysis of collected performance data during the audit. Typically, FOL

CORs perform individual audits; however, office meetings should take place during the month to exchange performance data, concerns, questions, etc. Data shall be examined, sorted, and verified before being used as objective evidence in an audit report. If data appears to be ambiguous or contradictory then the FCQA/COR shall take necessary actions such as collecting more data to confirm or deny a position. Data shall be sorted under observations/concerns and findings (nonconformances). Each finding shall be a clear and concise statement of a problem and will typically result in a 1 st or 2 nd notice (remember a 2 nd notice follows a 1 st notice, you do not “jump” straight to a 2 nd notice) or CAR. Each finding does not typically require an individual notice or CAR; they can be grouped by contract or functional area. For example, if you found two minor logistics problems they can be grouped to support a single finding. All findings shall be supported by linkage to a contract requirement! It would be nice to be able to tell the Contractor just to fix the problem/nonconformance rather than go through the process to document it and track it to closure. The problem is how we can determine Contractor performance. Additionally, it is often the “small problems” that are key/leading indicators of much larger/mission impact problems that are about to happen. With only a one year tour, if we don’t document it, it is very difficult to stay on top of negative trends.

4.3.5 Data Patterns & Trends

After the data sorting and analysis the COR/FCQA shall use the results to determine what the data supports and what supports it. They will need to look at the results in a systemic view in relation to the purpose of the audit and its importance to the mission. A thorough analysis supports the auditor’s capability to distinguish between isolated and systemic problems. Risk shall be a prime element in determining the seriousness of an isolated or systemic problem.

Functional area managers must be fully engaged in this requirement. They own the contract requirement and are positioned to make the necessary decision concerning performance risk.

Don’t hesitate to call the functional! Additionally, this is where those office meetings along with good note taking can identify a system trend. A good example is documentation errors; we don’t create a nonconformance for every documentation error; however, if we see a trend developing we will create a first notice likely against the Contractor’s QMS. In your discussions with the

CORs you realize everyone has been noticing documentation errors in all functional areas.

Individually, they don’t mean much, but now you are seeing a trend across the contract; likely a first notice would need to be issued.

4.3.6 Assess Performance

Assess performance as a whole instead of individual findings and observations. Seldom do we pay for nor can we afford perfection so we should expect minor problems. From an auditing standpoint we expect the problems be isolated and not reoccurring (we expect corrective/preventive action). If we are finding a multitude of problems in the Logistics arena, for example, that may indicate a systemic problem. How does that affect the overall performance of the contract; what is the risk? Maybe these individual failures when looked at as a whole have caused a moderate or high risk? If this is the case a CAR may be warranted.

Remain objective, compare performance to contract requirements, look at only the objective evidence (not “hear say” unless you can validate it), and make solid decisions based on these facts.

4.3.7 Nonconformances

Audit results that are below standards shall be identified as nonconformances. Findings that generate a nonconformance shall be further classified using the following definitions/guidance:

Major Nonconformance. A nonconformance that adversely impacts (or has the potential to) mission, safety of personnel and/or equipment, environment, performance (quality), schedule

(delivery), and/or cost. This type of nonconformance has a risk assessment rating of moderate or high. It is communicated to the Contractor using a single type of notification:

• Corrective Action Request (CAR)

Minor Nonconformance. A nonconformance that does not necessarily adversely impact mission, safety of personnel and/or equipment, environment, performance (quality), schedule (delivery), and/or cost. It is communicated to the Contractor using two types of notification:

• First Notice

• Second Notice (Issued when the same nonconformance is identified again or the Contractor fails to correct a first notice in a timely manner)

4.4 Audit Results Analysis

The FCQA/COR shall assess each contract activity based on the process in Figure 1 below.

4.5 Audit Reporting

Audit results must be formally reported. The FCQA/COR prepares the report and validates that the results conform to the audit plan and expectations. A formal written report is prepared using the audit report template. The FCQA has ultimate responsibility for the report; however, other members of the audit team should review report contents, as appropriate, to verify its accuracy and objectivity. A team review of the audit report prior to providing to the Contractor is the preferred method of review.

4.5.1 Nonconformances (Findings)

When a nonconformance is identified that has the potential to be a CAR, it is critical the COR gathers/evaluates all data/evidence. Ensure the data the COR has gathered:

Is the most current and fact based

Is a contract requirement

Is validated through mandatory AFIs, TOs, etc. (physically validate to ensure these publications have not changed recently)

Is not negated by a CO letter

NO

YES

Conform Non-Conform

MEET/EXCEED

MINOR

MAJOR

Review Data

Is It a Requirement?

Observation/Concern?

Is It a Conformance or

Nonconformance?

Does it Meet or Exceed?

Annotate in TMS

MAR, ES,

WAWF, COS

Minor / Major

1st or *2nd Notice?

Issue CAR – Must reach Moderate/High Risk

Contractor Places in Their QMS

Follow-up Corrective Action Plan

FCQA/COR

Monitor Close Out

FCQA/COR

Monitor Close Out

Figure 1. Analysis Results of Audit Diagram

* To create a 2nd Notice, there must already be a first notice where the Contractor failed to take action

Document as required in the

Additionally, CORs should review the Government’s and the Contractor’s nonconformances and trend analysis to determine and history with this nonconformance. Had the Contractor already identified this problem but instituted ineffective corrective and preventive action? Once all this information has been validated go to the QA SharePoint and bring up the latest CAR form.

Using section 10 (Risk Rating) of the CAR form determine the level of risk this brings to the

Government. If it is a moderate or high risk then a CAR is likely substantiated; however, if the risk is low then a first notice is warranted. This process will hold true in most cases; however, there can be circumstances that require exceptions to the norm. In these cases coordination with your FAMs, QAM, PM, and CO is a must!

Once the research establishes a CAR is warranted, complete the CAR form (QMSF 8.3-1).

Instructions are contained within the form; do not hesitate to ask for assistance from the QAM.

At this point the CAR will be ready for routing/coordination (See paragraph 4.6.1.2).

Remember, when you outbrief the contractor it is okay to notify them that a potential CAR exists, but only the contracting officer can issue the CAR.

4.6 Control of Nonconformance

When applicable, the COR shall prepare a CAR. The CAR will be controlled by using the (2-digit) Year/ (2-digit) Month / (3-digit) sequential number / (3-digit) Location / (1-digit) Identifier of “C” to identify it’s a CAR. (e.g. 1305001CUR-C).

First and second notices can be directly issued by the FCQA and CORs. A CAR will be formally issued by the CO to the Contractor. Again, we can notify the Contractor that a potential

CAR is pending; however, it is not a CAR until the CO issues it as such.

4.6.1 Major Nonconformance

4.6.1.2 CAR Routing

If a CAR is initiated, the FCQA, QAM, FAM (as applicable), and PM must review/validate and it shall be routed to the CO NLT 10 calendar days from initiation per the Quality Manual. The

CO will evaluate the CAR and, if appropriate, sign and send to the contractor and suspense the contractor for corrective/preventive actions. Typically, the FCQA will recommend a suspense and provide a recommended CO letter. Upon receipt of the contractor’s response, the CO/PM will forward the CAR to the QAM/FCQA. We as QA are responsible to control the CAR and ensure it is issued and reviewed in a timely manner. The FCQA, COR, and QAM will ensure this takes place. Due to the geographical separation of Curacao, the QAM will take the lead on all CARs to ensure its timeliness.

4.6.1.3 Track Corrective Action Requests

All CARs shall be tracked by the FCQA/QAM until actions taken have satisfied the responsible functional area. This will also provide an audit trail, facilitate pro-active audit program execution and maintain audit program integrity and credibility. The Quality Assurance Division expects the FCQA to track the CAR through each step of the process and alleviate “bottlenecks”.

If it is important enough to be a CAR, then is should be timely to include the evaluation of the contractor’s response.

https://www.my.af.mil/afknprod/DocView.asp?DocID=8479703

4.6.1.4 Accepting Contractor’s CAPA/Closing the CAR

Section 11 of the CAR form identifies all the inputs (minimum) required by the contractor. The

FCQA/COR is responsible for reviewing the Contractor’s action plan and ensuring the FAM(s), PM, and QAM have coordinated on the action plan as previously discussed. Once the review is completed, there are several options; the action plan can be rejected, partially accepted, accepted but remain open until all actions are completed, or accepted and closed.

4.6.1.4.1 Rejecting the Contractor’s response

Should the Contractor’s action plan in block 11 of the CAR form fail to provide sufficient detail/CAPA to resolve the nonconformance, the entire response can be rejected. In block 12 of the CAR the FCQA/COR should fully evaluate the response and explain why the action plan is being rejected. The Government will not provide answers/recommended solutions; however, it is very important to provide sufficient detail so the Contractor understands where the action plan was rejected and what is expected. Once this is complete, it should be routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a

CO ltr with a recommended suspense will be forwarded to the CO (typically through the PM), for review and forwarding to the Contractor. When the revised action plan is received, the process starts over.

4.6.1.4.2 Partial Acceptance

This is very similar to paragraph 4.6.1.4.1, except the Contractor’s action plan (block 11 of the

CAR) appears to provide effective resolution with exceptions. As an example the contractor’s plan provides sufficient depth with the exception of “d. Action(s) taken to prevent recurrence to include the follow-up plan (how and when)”. In block 12 of the CAR provide sufficient detail so the Contractor understands why “d” was rejected and what is expected. Once this is complete, it should be routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a CO ltr with a recommended suspense will be forwarded to the CO (typically through the PM), for review and forwarding to the Contractor. When the revised action plan is received, the process starts over.

4.6.1.4.3 Accept/CAR Remains Open

When the Contractor provides an effective action plan (block 11 of the CAR); however, the implementation will take several weeks or months the Government can accept the Contractor’s action plan and the CAR will remain open until the action plan is fully implemented and evaluated by the Government. Whether to leave the CAR open or close it and follow-up on future audits is dependent on the maturity of the Contactor’s QMS, history with the Contractor and the depth of the CAPA and associated risk level. Should the FCQA/COR determine the

CAR will remain open, it will be routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a CO ltr will be forwarded to the CO (typically through the PM), for review and forwarding to the Contractor.

The CO ltr will indicate how long the CAR will remain open for. The FCQA/COR will continue to monitor progress and at interval indicated in the CO ltr, the action plan will be fully evaluated.

If the action plan is still not fully implement, this will be annotated in block 12 of the CAR and then be routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a CO ltr will be forwarded to the CO (typically through the

PM), for review and forwarding to the Contractor. The CO ltr will indicate how long the CAR will remain open for. The FCQA/COR will continue to monitor progress and at interval indicated in the CO ltr, the action plan will be fully evaluated. Once the Government is satisfied the nonconformance has been corrected and the CAPA is effective the CAR will be closed.

Annotate block 12 of the CAR and route for closure along with the draft CO closure ltr. Blocks

13-17 of the CAR must be initialed and dated as part of the closure process – OFTEN

OVERLOOKED!

4.6.1.4.4 Accept and Close the CAR

When the Contractor provides an effective action plan the CAR can be closed. As discussed in the previous paragraph the CAR can remain open to completely evaluate the “fix” or close and perform follow-ups as part of follow-on audits. Again it is fully dependent on the maturity of the

Contractor’s QMS, history with the Contractor and the depth of the CAPA and associated risk level. Assuming maturity and history is positive is there any value added to keeping a CAR open for a year because parts are on order and they have a long lead time? As a FCQA it is important to look at the entire picture and make the call – with FAM(s), QAM, and PM coordination. Once the Government is satisfied the nonconformance has been corrected and the CAPA is effective the CAR will be closed. Annotate block 12 of the CAR and route for closure along with the draft

CO closure ltr. Blocks 13-17 of the CAR must be initialed and dated as part of the closure process – OFTEN OVERLOOKED!

4.6.2 Minor Nonconformance(s)

4.6.2.1 Open Minor Nonconformance(s)

As previously discussed, minor nonconformances will be issued as either first or second notices.

Remember, you must issue a first notice before you issue a second notice. Formal notification will be provided to the contractor during the outbrief.

4.6.2.2 Minor Nonconformance Suspenses

First and second notices will not be suspended by CORs. If the Contractor has failed to respond or initiate corrective/preventive action within a reasonable time, the nonconformance may be elevated to the next level (2 nd notice or CAR); however, CORs should receive concurrence from

FCQA prior to elevating. We expect the contractor to incorporate these findings as part of their

Quality Management System (QMS), “rack and stack” them with their internal findings, and work them accordingly. As part of our future audits, we evaluate the progress the Contractor has made in closing the minor nonconformances. When should we consider a second notice due to lack of progress? This is subjective. Keep the following in mind when evaluating the progress:

It is a minor nonconformance

Where did it rack and stack in the Contractor’s system

Has the contractor made progress (is the notice issued moving up to the top of the list for corrective/preventive action)

Are there any mitigating circumstances (Especially Government caused)

How long ago was the first notice issued

If you audit multiple locations performing the same mission (or different shops (i.e. AGE and

VM)); how long ago was the first notice issues, was there sufficient time for cross tell to be effective?

4.6.2.3 Minor Nonconformance Corrective/Preventive Actions

The Contractor is not required to formally report their corrective/preventive actions for minor nonconformances, but may do so if desired (enter the reply in TMS). A review of their QMS will show their process for closing Government issued 1 st and 2 nd notices. Remember they are contractually (and ISO) required to follow their QMS procedures; this is auditable.

Corrective/preventive actions initiated and/or completed by the Contractor will be validated by

CORs on the next or subsequent site visit.

4.6.3 Re-validate & Re-verify Audit Actions

The FCQA shall ensure open items that are recommended for closure are again validated and verified to ensure they comply with contract requirements; trust but verify.

4.6.4 Communicate Closure Actions & File Audit Report

Completed reports, including supplemental information and documents, shall be retained until contract closeout under direction of the contracting officer. It is important to define what records to keep and what to discard for liability purposes. In addition to liability protection, retaining audit records (e.g. audit working papers, reports, and other documents) serves a variety of purposes such as:

Providing proof of compliance with audit standards

Facilitating future audit planning

Facilitating review of audit work

CARs are filed on our QA SharePoint which is discussed later in this plan.

4.7 Records Maintenance

Records will be maintained IAW with the Records Management Procedure.

5.0 INTERNAL PRODUCTS

The QA division is responsible for multiple products related to reporting compliance to contract requirements. These products and their associated distribution and quality characteristics are located in Table 7.2.1 of the Quality Assurance Division’s Quality Manual.

5.1 Internal Product Compliance

Many of the products we produce are templated and are available on our QA SharePoint. These templates and forms shall be used and not altered. It is important to ensure your products meet template requirements. Often times we use last month’s product and cut and paste. Over time fields are accidently removed or added resulting in template non-compliance. Our division is registered to ISO 9001:2008; altering a template without approval will result in our division receiving a nonconformance!

https://afkm.wpafb.af.mil/DocView.asp?DocID=7800977

6.0 AWARD FEE/INCENTIVE FEE

6.1 Purpose

7.0 Risk

7.1 Risk Training

It is recommended that each QAM/FCQA/COR receive risk management training prior to performing risk analysis in order that each individual performing risk analysis is knowledgeable of the risk assessment process. Training may be requested by contacting the ACC AMIC Risk

Manager, ACC AMIC/DRQP, (757) 764-9480 or DSN 574-9480. Risk Management Training

Power Point can be accessed at the QA SharePoint.

7.2 Planning Surveillance

When planning government surveillance, the QAM/FCQA/COR shall base intensity, frequency, and surveillance observations on risk analysis results, contract requirements, and other directed

(mandatory) surveillance. Where AFIs, regulations, law, and/or mandatory surveillance is directed, they shall take precedence, however; every effort shall be made to increase or decrease government surveillance based on risk analysis/assessment. Adjustments to routine and mandatory surveillance shall be documented and approved by the applicable responsible authority.

7.3 Risk Based Surveillance

When determining government surveillance based on risk management, it is recommended that the following process be accomplished:

1. Read/Review the contract.

2. Identify & document mandatory surveillance (i.e. service summary items, etc.).

3. Review Program Risk Management Plan (RMP).

4. Identify High, Moderate and Low risks that are documented in the RMP.

5. Determine whether or not the documented RMP risks should be incorporated into the mandatory/planned surveillance.

6. Identify and incorporate RMP risks that are rated High and Moderate into planned surveillance as deemed appropriate. Low risks should also be included; however, the emphasis should be on High and Moderate risks with minimal surveillance for Low risks.

7. Identify and document any other surveillance deemed appropriate (i.e. ISO 9001 processes, etc.)

8. Collect Data relating to processes to be surveyed. Use government, customer, and

Contractor data when possible.

9. Determine how far back data will be collected (i.e. beginning of contract, past year, 6 months, etc.).

10. Perform risk analysis on those observations to be performed during surveillance.

a. Determine how data will be evaluated to determine Likely Hood/Probability of risk and Consequence/Impact of the risk. Using the DoD Risk