WRM_QASP.docx

HQ ACC ACQUISITION

War Reserve Materiel (WRM) and Program and Exercise & Contingency Logistics Support Quality Assurance Surveillance Plan (QASP)

NOTICE: This Publication is available in digital format via the AMIC QA SharePoint.

//signed/srrb/3 Jun 15// Received by: Mrs. Sheila R. Reshard-Bryant

HQ ACC AMIC/PKBA (PCO)

Revision: 0 Pages: 49 Distribution: F Solicitation Number: FA4890-15-R-0004

Table of Contents

1.0 INTRODUCTION

This Quality Assurance Surveillance Plan (QASP) has been developed to implement AFI 63-501 Air Force Acquisition Quality Program and AFI 63-138, Chapter 6, Service Acquisition Management & Oversight. The QASP is applicable to personnel performing contract surveillance audits IAW FAR Part 46, AFI 63-501, AFI 63-138, Chapter 6 and organizational policy. It is designed to provide the Contracting Officer Representative (COR) a systematic surveillance method for each service identified in the contract, for the Air Combat Command (ACC) War Reserve Materiel (WRM) and Program and Exercise & Contingency Logistics Support contract. Methods for administering and evaluating other provisions of the contract are to be developed by the Flight Chief Quality Assurance (FCQA), and/or Quality Assurance Manager (QAM), the applicable Functional Area Manager (FAM) or Subject Matter Expert (SME), and approved by the Contracting Officer.

1.1 AUTHORITY The Federal Acquisition Regulation (FAR) provides the Government the authority to audit the Contractor. Clause 52-246-X, where X is dependent on what the Contractor is providing (supplies, services, R&D, etc and whether it is firm fixed price or cost reimbursable) provides the exact verbiage. Select the FAR website; on the left side of the page click 52 and you will be provided with detailed information covering your application(s). Remember, your contract may use multiple clauses; typically you can look in Section E (Inspection and Acceptance) of your contract to locate the applicable clauses.

For the WRM program, the following clauses apply (From section E of the contract):

1.2 Purpose

This document defines the surveillance policy, procedures, and associated methods used by the Quality Assurance Division for planning, preparing, performing, analyzing, and reporting Contractor performance.

1.3 Application

To all parties involved with assessing Contractor performance of the WRM contract. This document is for the use of the FCQA/CORs/auditors and contracting officer. The Contractor may receive a courtesy copy in order to further enhance communications; however, this is at the discretion of the QAM.

1.4 Summary of Revisions

This document has been completely revised and is considered a new document.

1.5 Related Documents

AFI 63-138, Chapter 6

AFI 63-501

WRM Contract in its entirety

ISO 9000:2005

ISO 9001:2008

ISO 9004:2009

Quality Management System Manual and supporting documents

2.0 ROLES

Acquisition Management and Integration Center (AMIC) 11817 Canon Blvd, Suite 214 Newport News, VA 23606-4516

Contracting Manager (CM): (TBD) The individual within the contracting office who performs the day-to-day administration of the contract.

Procuring Contracting Officer (PCO): (Mrs. Sheila Reshard-Bryant, DSN 575-3369, Commercial 757-225-3369) The Government agent authorized to award or administer contracts. Appoints the CORs in writing to surveil the Contractor. The contracting officer is the ONLY person authorized to contractually obligate the Government.

Functional Area Manager (FAM): This person is typically in charge of a functional area (Logistics, CE, etc.). Occasionally referred to as Division Chief, Logistics/Division Chief, Civil Engineering.

Functional Director (FD): (Mr. Edward Lewis, DSN 575-6291, Commercial 757-225-6291) The person responsible for a functional area. Reviews contractor performance documentation, prepared by Program Manager, on a regular basis to ensure performance is compatible with contract and mission objectives while supporting a higher level of contractor performance..

Program Manager (PM): (Mr. Steven Peters, DSN 574-9456, Commercial 757-764-9456) The person in charge of managing a contract with groups of related projects in a coordinated manner to obtain benefits and control NOT available from managing them individually. Coordinates and prioritizes resources such as SharePoint, time and cost, to bring about successful completion of the contract. Manages the overall risk and past performance program.

Quality Assurance Division Chief: (Mr. Scott Shelton, DSN 574-9468, Commercial 757-764-9468) In charge of Quality Assurance Division. Selects primary and alternate CORs. Ensures personnel are properly trained and technically qualified to perform COR duties. Ensures personnel serving in the position of Functional Area Management perform as an alternate COR when necessary.

Contracting Officer Representative (COR): A qualified Functional Area Evaluator (FAE) who performs quality assurance functions for a contracted service. Personnel must complete QAPC and PCO Led training prior to consideration being given for certification. Occasionally referred to as an auditor or Contacting Officer Representative (COR).

Quality Assurance Manager (QAM): (Mr. Stephen Decker, DSN 575-6786, Commercial 757-225-6786) Serves as a supervisory quality assurance specialist for contracted radar operations and maintenance. This position plans, organizes, supervises the FCQA, Contracting Officer Representatives, and manages the activities and contract services of the MAJCOM.

Alternate COR: A Functional Area Evaluator (FAE) who is fully trained and qualified in a related functional specialty (e.g., CE, Logistics, System Communications, etc.). Provides technical support/expertise to program CORs through participation in formal CPETs, review Contract Data Requirement Lists (CDRLs) and review/coordination of COR issued Corrective Action Request CARs.

Subject Matter Expert (SME): The person who is qualified in a related specialty and provides technical support/expertise to a program, but does not have COR training and is not appointed by contracting officer to perform COR duties.

Customers: United States Air Forces Central (USAFCENT), the designated air component of US Central Command (USCENTCOM), and all deployed forces in the USAFCENT Area of Responsibility (AOR).

3.0 RESOURCE MANAGEMENT

Manage resources IAW AF Directives and the Quality Manual, Section 6, Resource Management.

3.1 AMIC Guidance & Policy

All guidance & policy is located on the DRQ SharePoint. This is a mandatory review folder for all FCQA’s, and supervisors who work for ACC AMIC and is recommended for AFCENT personnel. This folder contains all AMIC and DRQ policy letters, EPR, decoration, and awards guidance, and a host of additional information that will make your job easier.

3.2 Manning

WRM Quality Assurance will consist of a Flight Chief and CORs. The FCQA shall have overall responsibility of all CORs. HQ ACC AMIC is responsible for training and providing guidance and direction to CORs. Personnel assignment is controlled through AFPC and AFCENT as to what AFSC and skill level are required in order to successfully audit the WRM contract and ensure compliance. Current information is listed in the table below:

MSgt

3.3 Training

The training plan is located on the SharePoint. This training plan and associate documents are very detailed and should guide you through the entire training process. Training will be documented on an AF Form 797, and maintained in individual AF Form 623s (or using the Training Business Area (TBA)). CORs are also cross-utilized/trained in areas of responsibility and duties outside their normal AFSC. Air Force/ACC directives and technical orders will be used to complete OJT in areas where the individual COR lacks extensive knowledge. All CORs must complete DAU Courses CLC 222 - Contracting Officer Representative (COR) Training, CLM 003 - Overview of Acquisition Ethics, CLC 206 - CORs in a Contingency Environment and QAPC/PCO Led Training prior to consideration being given for certification.

Follow these instructions to register for DAU Courses:

Click here DAU to get to the DAU website and follow these instructions:

1. Once at the DAU site click on Course Login

2. Click on Login using my CAC (right side)

3. Click on Continuous Learning Modules

4. Click the Black Arrow - Continuous Learning Modules for a drop down list

5. Click Air Force, Military, and Civilian

6. You should now be here: ACQ Now for DAU Training (sometimes going directly to this site does not work)

7. Click in the middle if this is your first time to the site to set up your profile (follow instructions), otherwise, Click on Apply for Training (upper left)

8. Step 1, Click Continuous Learning Modules

9. Step 2, Click the Black Arrow for the drop drown list

10. Select one of the courses mentioned above.

CLC222 is extremely lengthy and is a bit boring, but it will give you the necessary contract language and overview.

CLM003 is relatively quick and easy - basic Ethics information related to contracting

CLC206 is also fairly quick and covers some specialized requirements while in a contingency environment (only required prior to deploying – good idea to complete anyway).

After you have completed the training, send your completion certificates individually in .pdf format to the QAM for uploading into CMS.

3.4 Applications

3.4.1 SharePoint

There are many SharePoint’s available; the ones we use most frequently is the DRQ SharePoint and the WRM SharePoint. You will see them listed on the left hand side of our QA SharePoint. SharePoint is a program developed by Microsoft and used widely throughout industry and the Government. There are two “SharePoint’s” you need to be aware of; one the Contractor uses and one the Government uses. The Contractor’s SharePoint is used to store all contractually required documents as well as a host of additional information. Some may even use a separate location to maintain all their PMIs, work orders, QC findings, purchases etc. You should be cognizant of the different applications of websites and realize people frequently confuse them. The use of our QA SharePoint (and the program specific SharePoints) alleviates the limitations of a share drive. The SharePoint also allows us to transfer large files between users alleviating email limitations.

3.4.2 Defense Travel System (DTS)

The DTS website is the platform we use to create all TDY orders and vouchers. If you are new to DTS or need a refresher, a good place to start is where the link above takes you. You will see a “Learn More” and “Training” button right below the “Login to DTS” button. These links, especially the Training link will provide you a wealth of how to information. It is important to take the time to review all applicable training; it will alleviate a lot of headaches down the road. Additionally, the QA SharePoint contains a DTS folder which provides commonly required/requested information (how to cancel a trip, adjust an authorization. etc.). Each base should have DTS business rules such as the Langley/Eustis Joint Base DTS Business Rules posted on the DRQ SharePoint. It is important you review the business rules applicable to your base. You are responsible for knowing your base’s business rules. Common problems the QAMs see when approving authorizations and vouchers:

· Authorizations – Didn’t provide:

· PERSTEMPO code for military (typically “D”),

· Full justification for pre-audit flags

· Cost comparison worksheet when driving or purchasing own tickets for leave purposes

· Other: Alternate means, such as SVTC or other web based communications are not sufficient to accomplish travel objective.

· Variations Authorized: Dates, times, and modes of travel may change without notice

· Vouchers – Didn’t provide:

· All substantiating documentation (should be one continuous file in the correct landscape/portrait format, all receipts $75 and above, all hotel receipts, all airline receipts (typically from you travel agent), all rental car receipts, write on your substantiating documentation to clarify!)

· Credit card international exchange costs

· Travel agent fee

· Hotel sales tax as an additional expense/not included in lodging cost – except overseas where it is all inclusive

· Non-availability statement where applicable

· Note 1: Although not mandated, providing billed and unbilled transactions from the bank is a plus and assists approving officials with validating vouchers.

· Note 2: All WRM CORs are required to fill out the worksheet below when filing your travel voucher. Make sure you upload this into DTS with your receipts.

Total Cost

$1175.00

4.0 AUDIT PROGRAM EXECUTION PROCESS

4.1 Introduction

The goal of all CORs is to ensure surveillance and measurement of Contractor services is performed in a manner that ensures maximum contribution by the COR to successful contract performance and mission accomplishment. The COR contribution is comprised of professional, non-adversarial relationships, which include positive and open communication with our customers and the Contractor. The foundation of this relationship is built upon objective, fair, and consistent COR evaluations of Contractor performance against contract requirements, the ability to discuss inspection results, trends, and items of mutual interest with the contract. The COR uses the methods in this QASP, contract, ISO 9001, and AFI 63-138, Chapter 6 to achieve this goal. The COR function is responsible for a wide range of audit requirements that effectively measure and evaluate the Contractor’s performance. Additionally, this QASP is based on the premise that the Contractor, not the Government, is responsible for management and Quality Control (QC) and Quality Assurance (QA) actions to meet the terms of the contract. Basic points to remember when preparing for your audit:

· Audits are performed using process-based auditing techniques. Most people are familiar with using a checklist to perform some sort of inspection/audit. In WRM we use process-based auditing to ensure we capture the most significant problems/highest risk to the program. We still use a checklist, but that is only in the event a nonconformance has to be written and for verification of references applicable to the contract. We have many books in our library that will help you better understand process-based auditing.

· It is critical that you coordinate with your Quality Assurance Manager, functionals, Program Manager, and Contracting Officer prior to performing your audit. They may have concerns/areas of interest they would like us to add to our audit. Examples include; LG may want us to validate plant clearance cases (disposal of GFP), CE may want us to validate the status of a 332 project, and SC may want us to validate the status of a 3215 project. A good working relationship is a must.

· Do as much as you can from your desktop prior to starting the actual audit. Most Contractors now have websites that contain a wealth of information. Typically you can review all PCO & PMO letters, work orders, purchase orders, PMI schedules, etc. You will be much better prepared and can focus your audit accordingly.

· CDRLs are always a concern as to who should audit them. Before the contract started, the CORs, PM, and Functionals determined who is responsible for each CDRL. This information can be found in Attachment 3 of this Quality Assurance Surveillance Plan. As CORs we look at all CDRLs because it is a Service Summary item; however, we must coordinate with the appropriate functional to ensure the data is accurate/complete to their satisfaction.

· If you are performing a multiple-day audit, document as you go; waiting until you have completed the audit to start your documentation typically results in missed information and rushed/poor quality products. A good rule of thumb, “carry a notebook with you and document as you see”.

4.2 General Policy

4.2.1 Periodic Meetings

The CORs and QAM will discuss and exchange information periodically items of importance. Topics of discussion will include, but are not limited to, information from audits, safety issues, nonconformances, noted weak/problem areas, and any reported customer complaints.

4.2.1.1 Post Audit Discussions

The first duty day, if schedules permit, following an audit, the CORs, FCQA, and QAM will discuss audit results.

4.2.2 COR Limitations

CORs are the "eyes and ears" of the QAM, PM, and PCO relative to actual contract performance. Remember: CORs WILL NOT DIRECT WORK, DIRECT RE-ACCOMPLISHMENT OF WORK, CHANGE THE CONTRACT, OR FORMALLY INTERPRET THE CONTRACT. The PCO, in coordination with CORs, functional area/technical experts, resolves these types of issues.

4.2.3 COR Audits

It is essential that CORs accomplish sufficient in-depth audits in all areas to measure the quality of Contractor performance and ensure the Contractor is meeting contract requirements. This is where the Risk Management Plan, your previous audit results to include trends, recent contract changes, Contractor manpower turnover, etc. come into play. This information (along with discussions with the PCO, PM, QAM, and Functionals) should be used to determine what will be audited and to what depth. Is there really a need to audit Technical Orders (TOs) every single time if you have not found any problems in the last two years? Maybe a sample of TOs semi-annually would be ample and allow you to focus your audits on critical elements. Proper planning will ensure you audit the right elements! Remember that as part of our audit preparation we will review the Contractor’s documentation for trends and ensure the Contractor is providing proper corrective and preventive actions. Additionally, compliance with Contractor developed plans and regulations will be part of the surveillance, as they are deliverables via CDRLs or part of their Quality Management System (QMS).

4.2.3.1 Types of Audits

CORs will be performing three types of audits. Initial Contract Performance Review, Desktop audits, and On-site audits.

4.2.3.1.1 Initial Contract Performance Review

The initial evaluation of contractor performance is a joint determination by a multi-functional team that may include PCO, PM, QAM, and CORs to determine if the contractor has successfully started performance and completed transition, is fully operational, and is within the estimated cost, schedule, and performance parameters of the contract. See para 4.3.2.1 below for more information.

4.2.3.1.2 Desktop Audits

When performing desktop audits, CORs rely on information and data filed on the contractor’s Data Management or other system and validation of CDRLs and other reports, procedures, and plans. Some of the systems the contractor is responsible for, but not limited to: Combat Ammunition System (CAS), Tactical Munitions Reporting System (TMRS), Integrated Maintenance Data System (IMDS), Automated Identification Technologies (AIT) systems, Standard Base Supply System (SBSS), etc.

4.2.3.1.3 On-site Audits

CORs will travel to numerous locations to ensure the contractor is performing IAW contract requirements at these locations.

4.2.4 Audit Key Points

COR audits will be based on contract requirements. Within the contract, the Performance Work Statement (PWS) will dictate types of audit actions needed and what processes to be surveyed. Within the PWS, an audit method or process is driven by words that stipulate what is expected of the Contractor and drives each inspection. A master checklist is built, updated, and managed by the FCQA and is based according to each functional area within the PWS.

Points to remember:

· The Service Summary (SS) are critical auditing points and should be validated every audit and the results documented on the audit report

· Refer to Attachment 1 for instructions on how to audit each SS

· Contract requirements encompass the entire contract, not just the PWS

· Contract requirements include the contactor’s plans and procedures and are auditable

· The contract is typically performance-based NOT T.O./AFI driven. Unless a T.O., AFI, Regulation, Publication is mandatory, the Contractor may develop their own procedures to complete the requirement (again, auditable) and can be found in contractor’s SharePoint and Data Management System (DMS)

· Audit preparation is the KEY to a successful audit – be prepared

· Keep an open mind during the audit process – no preconceived notions

· During the audit listen, listen, listen, don’t judge

· If you don’t document it, it didn’t happen; your words should paint a picture and be OBJECTIVE

· Stick to facts and keep emotion out of it

4.3 Audit Performance

4.3.1. Audit Preparation and Planning

There are two parts to this step which overlap. If you have just arrived on station you need to take the time to become familiar with the entire contract. This first part is called the “Initial Preparation and General Instructions” (4.3.1.1) which certainly applies for new CORs, but is also very important to review these prep/instructions frequently as part of your audit preparation to ensure you stay familiar with the contract. The second part is called “Preparing for the Audit” (4.3.1.2) which you should refer to before performing your audits.

4.3.1.1 Initial Preparation and General Instructions

· Read the entire contract; not just the PWS

· Contractor’s SharePoint – take the time to go through each folder to familiarize yourself

· As a minimum look at the last six months of CDRLs, PCO and PMO letters (from this point forward, keep up with these)

· Contractor’s DMS – Take the time to become familiar with the system

· Read the Contractor’s QMS along with the six mandatory procedures plus Contractor’s applicable checklists, work instructions, site operating procedures, etc. that apply to auditable areas

· All the above documents are auditable – many are CDRLs, deliverables, or required by ISO, as the Contractor is required to be compliant

· Read the risk management plan

· Perform a thorough site walk around

· Don’t assume anything; it is easy to think it has always been that way so it must be right

· If it doesn’t seem right ask your FCQA, QAM, PM, Functional, etc.

· As part of all your audits, plan on working an early morning, evening, and night shift along with audits on the weekends. Check flight schedules and plan these audits accordingly

· Document, document, document; if you don’t document it in your audit, it didn’t happen; and ensure you keep good notes!

· Fully explain the “how you validated” and how the Contractor met requirements – not just “no problems noted”.

· Remember, your audit plan can be adjusted according to the mission

· You can look at the same area more than once in a given audit if there are concerns

· If you are performing a random sample and see potential problems outside the sampling area; finish your random sample and then focus on the potential problem (Unless it is a critical/imminent safety, environmental or health issue, maintenance practice, or risk)

· Occasionally perform audits with the Contractor’s QC

· Always review what the Contractor has audited. We don’t typically double document – if the Contractor has already self-identified a problem, validate they performed sufficient corrective/preventive actions.

· Don’t be afraid to ask questions

4.3.1.2 Preparing for the Audit

· Review previous audits, especially the Desktop Audit. This will determine what needs to be audited on site and will be the basis for creating your audit plan.

· Discuss overall Contractor performance with CORs

· Review PCO and PMO Ltrs

· Review what CDRLs are due during the audit period

· Refer to Attachment 3 for due dates

· Remember that we review all CDRLs for timeliness and accuracy and coordinate with the owning functional as well

· Any Contractor personnel changes (hiring, firing, resignations, duty changes)?

· Stay in tune with personnel changes and adjust your audit accordingly

· Coordinate with the PCO, PM, QAM, FCQA, and Functionals to determine if they have any issues/concerns – add to your audit

· In the contractor’s DMS look at upcoming PMI’s, work orders in progress (status)

· Add to audit as required

· Review recent changes to the contract

· Be sure to incorporate these changes in your audits

· Adjust frequency/depth of audit

· Review QA identified trends and historical data

· SII for this period?

· Review 332’s, 3215s, Disposal Cases

· Be sure to put these on your audit plan to validate status/conformity to requirements

· What shipments are due in – how are they handled; audit their process

· Risk Plan/any risks identified or have been identified through trend analysis.

· Mission changes – increase/decrease in sorties/personnel

· Has the Contractor’s workload increased/decreased (especially increased with no manpower increases)

· MOST OVERLOOKED!!! - Review Contractor’s internal audits to review Contractor’s findings and status of corrective/preventive actions – follow-up/validate.

· These should always be part of your audit (check latest status just prior to audit)

· Remember; if the Contractor has already documented a problem(s), we do not need to create (typically) a Government 1st notice, 2nd notice, or CAR. We should document it in our audit report and of course track the finding to closure. If the Contactor is not taking appropriate action we can certainly document that as a nonconformance

· Do as much as possible from your desktop prior to the audit

· Validate PMIs are being performed by the Contractor correctly and IAW OEM, procedures, and work cards

· Remain flexible

· Prepare your audit plan – refer to the audit plan template QMST 7.1-1.

4.3.2 Performance Phase

This is where the COR/auditor interfaces with the Contractor’s employees. Communication, (both verbal and nonverbal), appearance, and a mature disposition are key elements and shall be of the highest quality. CORs/auditors shall focus all efforts to ensure a professional job. Ethics, COR/auditor “Do’s and Don’ts” learned in QAPC/PCO Led COR Training provide the boundaries or rules of engagement. Strict adherence to PCO guidelines and common sense ensure success in this phase. Performance phase starts with an effective (brief, but comprehensive) opening meeting describing what is to be accomplished, where and by whom. CORs shall perform an opening meeting with the site manager, QC Specialist, or their designated representative(s). Remember, during the audit process…trust, but verify!

4.3.2.1 Initial Contract Performance Review (ICPR)

IAW AFI 63-138, Chapter 6, Para 6.4, the initial evaluation of contractor performance is a joint determination by the MFT that the contractor has successfully started performance, completed transition, is fully operational, and is within the estimated cost, schedule, and performance parameters of the contract. The ICPR shall take place within 30 days after the contractor assumes full performance responsibility (for your purposes – after contract start). For all contracts, which are >$100M, the ICPR report shall include an assessment of schedule, management, technical, and cost performance. Negative variations in cost, schedule, staffing, and performance shall be reported with an assessment of the root causes and corrective action plan. What does this mean to you? Your responsibility is to ensure that whatever the contractor put in their respective proposal that we the Government included as part of the contract (most O&M contracts include the contractors entire proposal as an Appendix to the PWS) is reviewed. Has the contractor started performance? (There are usually deliverables due to the Government within the first 30-days, e.g., CDRL) How is the contractor meeting their Transition Plan? Are they on track with schedule and milestones? Where do they stand in regards to their QMS? Have all personnel been hired (if not, are operations impeded?) These questions must be asked and reported. Use the reports that have already been accomplished as examples, located on the AFCENT ACO & COR SharePoint, embedded in the Monthly Activity Reports (MAR). Report contract performance assessment values in the following method:

· Green--No issues.

· Yellow--Issue(s) but contractor has an adequate mitigation or corrective action plan in place.

· Red--Issue(s) with inadequate or no contractor mitigation or corrective action plan. Any “red” assessment shall include the government proposed actions for the failing contractor.

Additionally, any significant modifications to the contract made since contract award shall be included in the initial performance report as special interest items.

4.3.3 Audit Process/Concepts

During your audit keep the following process/concepts in mind:

· Treat the Contractor the way you would want to be treated. Remember those inspectors that have showed up in your office, asked questions, immediately started writing notes, and then left without telling you anything – that is not the way to do business!

· Put the interviewee at ease, be non-threatening, and be very sensitive to language barriers

· Fully explain what you are doing and why – starting off by asking questions seldom puts the interviewee at ease, typically they will take a defensive role

· Ask the employee to explain their job, you can learn a lot

· Listen, listen, listen

· Don’t get tunnel vision

· Tunnel vision example; you are validating that a PMI has been performed so you look in the contractor’s DMS and perform a visual check – all is good

· The proper way; you are validating a PMI has been performed so you look in the contractor’s DMS, ensure it was completed and properly documented, and perform a visual check; the PMI requires the use of an oscilloscope, you validate the oscilloscope was on site, within calibration dates, and in working condition, ensured any calibration limitations did not affect the PMI; you validate the technician that performed the PMI was on-duty that day; you review the technician’s training record to ensure he is qualified, training is up to date; parts were used, you validate parts used were removed from supply correctly and reordered as required. (This is process-based auditing). A checklist question is single-pointed and does not allow for all aspects to be validated.

· If you are watching an employee perform a preventive maintenance action, repair action, etc. give the Contractor “space” and ensure you do not have five auditors standing over the employee observing; this would make anyone nervous!

· Once you have completed the required action(s) with the employee, provide a mini-out-brief. If the employee is meeting contract requirements, let him/her know. If you found problems or potential problems that require research, be up front with the employee. Once your research is completed, follow-up with the employee and provide results

· Don’t jump to conclusions; gather the data and analyze the facts – be objective!

· Be flexible; mission priorities change frequently

4.3.4 Analyze Data

CORs/auditors shall gather data as objective evidence and analyze it to determine compliance to contract requirements, quality objectives, and/or standards. The FCQA and/or the person performing the audit shall perform analysis of collected performance data during the audit. For audit teams (audits lasting more than a day) this analysis shall take place during daily meetings as a minimum. Typically WRM CORs perform individual audits; however, office meetings should take place during the month to exchange performance data, concerns, questions, etc. Data shall be examined, sorted, and verified before being used as objective evidence in an audit report. If data appears to be ambiguous or contradictory then the FCQA/COR shall take necessary actions such as collecting more data to confirm or deny a position. Data shall be sorted under observations/concerns and findings (nonconformances). Each finding shall be a clear and concise statement of a problem and will typically result in a 1st or 2nd notice (remember a 2nd notice follows a 1st notice, you do not “jump” straight to a 2nd notice) (Discussed in 4.3.6). Each finding does not typically require an individual notice or CAR; they can be grouped by contract or functional area. For example, if you find two minor logistics problems, they can be grouped to support a single finding. All findings shall be supported by a contract requirement! Additionally, it is often the “small problems” that are key/leading indicators of much larger/mission impact problems that are about to happen. It’s critical to identify problems, so trend analysis can be used later to determine the most critical areas requiring additional emphasis.

4.3.5 Data Patterns & Trends

After the data sorting and analysis, the FCQA shall use the results to determine what the data supports and what supports it. They will need to look at the results in a systemic view in relation to the purpose of the audit and its importance to the mission. A thorough analysis supports the auditor’s capability to distinguish between isolated and systemic problems. To assist in this requirement, the FCQA/COR can use Pareto charts to visualize data patterns and trends. Risk shall be a prime element in determining the seriousness of an isolated or systemic problem and the likelihood of recurrence. Additionally, this is where office meetings along with good note taking can identify a system trend. A good example is documentation errors; we don’t create a nonconformance for every documentation error; however, if we see a trend developing we will create a first notice likely against the Contractor’s QMS. In your discussions with the CORs you realize everyone has been noticing documentation errors in all functional areas. Individually, they don’t mean much, but now you are seeing a trend across the contract; likely a first notice would need to be issued – see next paragraph.

4.3.6 Assess Performance

Assess performance as a whole instead of individual findings and observations. Seldom do we pay for, nor can we afford perfection, so we should expect minor problems. From an auditing standpoint, we expect the problems to be isolated and not reoccurring (we expect corrective/preventive action). If we are finding a multitude of problems in the Logistics arena for example, it may indicate a systemic problem. How does it affect the overall performance of the contract? What is the risk? Maybe these individual failures when looked at as a whole have caused a moderate or high risk. If this is the case, a CAR may be warranted. Remain objective, compare performance to contract requirements, look at only the objective evidence (not “hear say” unless you can corroborate it), and make solid decisions based on these facts.

4.3.7 Nonconformance

Audit results that are below standards shall be identified as nonconformances. Findings that generate a nonconformance shall be further classified using the following definitions/guidance:

Major Nonconformance: A nonconformance that adversely impacts (or has the potential to) mission, safety of personnel and/or equipment, environment, performance (quality), schedule (delivery), and/or cost. This type of nonconformance has a risk assessment rating of moderate or high. Before deciding if it is a major, run the Risk Matrix embedded in the CAR Form. Make sure macros are enabled. It is communicated to the Contractor using a single type of notification:

· Corrective Action Request (CAR), QMSF 8.3-1 – available on our DRQ SharePoint.

Minor Nonconformance: A nonconformance that does not necessarily adversely impact mission, safety of personnel and/or equipment, environment, performance (quality), schedule (delivery), and/or cost. It is communicated to the Contractor using two types of notification:

· First Notice

· Second Notice (Issued when the same nonconformance is identified again or the Contractor fails to correct a first notice in a timely manner)

· Both notices are used on the same form: QMSF 8.3.1-1 – available on our DRQ SharePoint.

When a nonconformance is identified that has the potential to be a CAR, it is critical the COR gathers/evaluates all data/evidence. Ensure the data the COR has gathered:

· Is the most current and fact based

· Is a contract requirement

· Is validated through mandatory AFIs, TOs, Contractor OIs, etc. (physically validate to ensure these publications have not changed recently)

· Is not negated by a PCO letter Additionally, CORs should review the Government’s and the Contractor’s nonconformances and trend analysis to determine the history with this nonconformance. Had the Contractor already identified this problem, but instituted ineffective corrective and preventive action? Once all this information has been validated go to the DRQ SharePoint and bring up the latest CAR form as mentioned above. Using section 10 (Risk Rating) of the CAR form determine the level of risk this brings to the Government. If it is a moderate or high risk then a CAR is likely substantiated; however, if the risk is low then a first notice is warranted. This process will hold true in most cases; however, there can be circumstances that require exceptions to the norm. In these cases, coordination with your FAMs, QAM, PM, and PCO should occur. Once the research determines a CAR is warranted, complete the CAR form (QMSF 8.3-1). Instructions are contained within the form; do not hesitate to ask for assistance from the QAM. At this point, the CAR will be ready for routing/coordination (See paragraph 4.4.1). Remember, when you out brief the contractor it is okay to notify them that a potential CAR exists, but only the contracting officer can issue the CAR.

4.4 Corrective Action Request (CAR) – Major Nonconformance

When applicable, the COR shall prepare a CAR. The CAR will be controlled by using the (2-digit) Year/ (2-digit) Month / (3-digit) sequential number / (3-digit) Location / (1-digit) Identifier of “C” to identify it’s a CAR. (e.g.,1409001KUW-C). The same applies as above for 1st and 2nd notices. In addition to the CAR, prepare a draft PCO Letter. Ask your PCO for their version of a PCO letter. First and second notices can be directly issued by the FCQA and CORs. A CAR will be formally issued by the PCO to the Contractor. We can notify the Contractor that a potential CAR is pending; however, it is not a CAR until the PCO issues it as such.

4.4.1 CAR Routing

If a CAR is initiated, the FCQA, QAM, FAM (as applicable), and PM must review/validate and it shall be routed to the PCO NLT 10 calendar days from initiation per the Quality Manual. However, if for some reason you do not receive a reply from the FAM in a timely manner (roughly 2-days); submit it to next in the chain. The PCO will evaluate the CAR and will send it to the Contractor for corrective/preventive actions. The FCQA will recommend the suspense date on the draft PCO Letter. If deduction from payment is the appropriate action (as determined in coordination with PCO, PM, and applicable FAM), the PM/PCO will compute the payment deduction based on the performance period in which the performance occurred. Upon receipt of the Contractor’s response, the contracting officer, in consultation with the PM and FCQA, and if applicable, the FAM must evaluate the Contractor’s response and determine if the CAR should be closed, monitored for a period of time, or returned for additional information.

We as QA are responsible to control the CAR and ensure it is issued and reviewed in a timely manner. Ultimately, this is the FCQAs responsibility.

4.4.2 Track Corrective Action Requests

All CARs shall be tracked by the FCQA/QAM until actions have been taken. This will also provide an audit trail, facilitate pro-active audit program execution, and maintain audit program integrity and credibility. The Quality Assurance Division expects the FCQA to track the CAR through each step of the process and alleviate “bottlenecks”. If it is important enough to be a CAR, then is should be timely to include the evaluation of the Contractor’s response.

4.4.3 Accepting Contractor’s CAPA/Closing the CAR

Section 11 of the CAR form identifies all the inputs (minimum) required by the contractor. The FCQA/COR is responsible for reviewing the Contractor’s action plan and ensuring the FAM(s), PM, and QAM have coordinated on the action plan as previously discussed. It is important to note the contractor must provide the root cause analysis, how correction has been or will be performed, and corrective/preventive actions. Once the review is completed, there are several options; the action plan can be rejected, partially accepted, accepted but remain open until all actions are completed, or accepted and closed.

4.4.4 Rejecting the Contractor’s response

Should the Contractor’s action plan in block 11 of the CAR form fail to provide sufficient detail/CAPA to resolve the nonconformance, the entire response can be rejected. In block 12 of the CAR the FCQA/COR should fully evaluate the response and explain why the action plan is being rejected. The Government will not provide answers/recommended solutions; however, it is very important to provide sufficient detail so the Contractor understands where the action plan was rejected and what is expected. Once this is complete, it should be routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a PCO Ltr with a recommended suspense will be forwarded to the PCO (typically through the PM), for review and forwarding to the Contractor. When the revised action plan is received, the process starts over.

4.4.5 Partial Acceptance

This is very similar to paragraph 4.4.1.4, except the Contractor’s action plan (block 11 of the CAR) appears to provide effective resolution with exceptions. As an example the contracts plan provides sufficient depth with the exception of “d”. Action(s) taken to prevent recurrence to include the follow-up plan (how and when)”. In block 12 of the CAR provide sufficient detail so the Contractor understands why “d” was rejected and what is expected. Once this is complete, it should be routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a PCO Ltr with a recommended suspense will be forwarded to the PCO (typically through the PM), for review and forwarding to the Contractor. When the revised action plan is received, the process starts over.

4.4.6 Accept/CAR Remains Open

When the Contractor provides an effective action plan (block 11 of the CAR); however, the implementation may take several weeks or months the Government can accept the Contractor’s action plan and the CAR will remain open until the action plan is fully implemented and evaluated by the Government. Whether to leave the CAR open or closed with follow-up on future audits, is dependent on the maturity of the Contactor’s QMS, history with the Contractor, and the depth of the CAPA and associated risk level. Should the FCQA/COR determine the CAR will remain open, it will be routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a PCO Ltr will be forwarded to the PCO (typically through the PM), for review and forwarding to the Contractor. The FCQA/COR will continue to monitor progress and at an interval indicated in the PCO Ltr (the action plan will be fully evaluated). If the action plan is not fully implemented, it will be annotated in block 12 of the CAR and routed to the FAM(s), PM, and QAM for their review/input/concurrence. Once this process is complete, the CAR and a PCO Ltr will be forwarded to the PCO (typically through the PM), for review and forwarding to the Contractor. The PCO Ltr will indicate how long the CAR will remain open. The FCQA/COR will continue to monitor progress and at an interval indicated in the PCO Ltr. The action plan will be fully evaluated. Once the Government is satisfied the nonconformance has been corrected and the CAPA is effective, the CAR will be closed. Annotate block 12 of the CAR and route for closure along with the draft PCO closure Ltr. Blocks 13-17 of the CAR must be initialed and dated as part of the closure process – OFTEN OVERLOOKED!

4.4.7 Accept and Close the CAR

When the Contractor provides an effective action plan the CAR can be closed. As discussed in the previous paragraph, the CAR can remain open to completely evaluate the “fix” or close and perform follow-ups as part of follow-on audits. Again it is fully dependent on the maturity of the Contractor’s QMS, history with the Contractor, and the depth of the CAPA and associated risk level. Assuming maturity and history is positive, is there any value added to keeping a CAR open for a year because parts are on order and they have a long lead time? As a FCQA, it is important to look at the entire picture and make the call – with FAM(s), QAM, and PM coordination. Once the Government is satisfied the nonconformance has been corrected and the CAPA is effective, the CAR will be closed. Annotate block 12 of the CAR and route for closure along with the draft PCO closure Ltr. Blocks 13-17 of the CAR must be initialed and dated as part of the closure process – OFTEN OVERLOOKED!

4.5 1st and 2nd Notices - Minor Nonconformance(s)

4.5.1 Open Minor Nonconformance(s)

As previously discussed, minor nonconformance’s will be issued as either first or second notices. Remember, you must issue a first notice before you issue a second notice. Formal notification will be provided to the Contractor during the out brief.

4.5.2 Minor Nonconformance Suspense’s

First and second notices will not have a suspense date. If the Contractor has failed to respond or initiate corrective/preventive action within a reasonable time (usually within 60-90 days, unless there are extenuating circumstances), the nonconformance may be elevated to the next level (2nd notice or CAR); however, CORs should receive concurrence from FCQA prior to elevating. We expect the Contractor to incorporate these notices as part of the Quality Management System (QMS), “rack and stack” (based on risk) them with their internal findings, and work them accordingly. As part of our future audits, we evaluate the progress the Contractor has made in closing the minor nonconformances. When should we consider a second notice due to lack of progress? This is subjective unless specifically stated. Keep the following in mind when evaluating the progress:

· It is a minor nonconformance

· Where did it rack and stack based on risk in the Contractor’s system

· Has the Contractor made progress (is the notice issued moving up to the top of the list for corrective/preventive action)

· Are there any mitigating circumstances (Especially Government caused)

· How long ago was the first…