Atch_5_QASP_AFEC_II_PMEL.pdf

Air Force Enterprise Contracted (AFEC) Precision Measurement Equipment Laboratory (PMEL) Quality Assurance Surveillance Plan (QASP)

AFEC II PMEL QASP 1 August 2017

BY ORDER OF THE HQ ACC ACQUISITION

DIRECTOR MANAGEMENT AND

INTEGRATION CENTER (AMIC)

Date: 01 August 2017

AFEC PMEL

Contract FA4890-17-R-0011 Quality Assurance Surveillance Plan

COMPLIANCE WITH THIS DOCUMENT IS MANDATORY

NOTICE: This Document is available in digital format through HQ ACC AMIC/DRQP

OPR: HQ ACC AMIC/DRQP Reviewed by: HQ ACC AMIC/PMSP Sean P. Jenigen, NH-03, DAF David Hyman IV, NH-03, DAF Quality Assurance Manager Program Manager

Accepted by: HQ ACC AMIC/PKC Phillip J. Nease, NH-03, DAF Contracting Officer

HYMAN.DAVID

.IV.1049430970

Digitally signed by

HYMAN.DAVID.IV.1049430970

DN: c=US, o=U.S. Government, ou=DoD, ou=PKI, ou=USAF, cn=HYMAN.DAVID.IV.1049430970 Date: 2017.08.10 14:30:59 -04'00'

JENIGEN.SEAN.

PATRICK.10786

58203

Digitally signed by

JENIGEN.SEAN.PATRICK.1078658203

DN: c=US, o=U.S. Government, ou=DoD, ou=PKI, ou=USAF, cn=JENIGEN.SEAN.PATRICK.107865820 Date: 2017.08.08 13:12:18 -04'00'

NEASE.PHILLI

P.J.1082985596

Digitally signed by

NEASE.PHILLIP.J.1082985596

Date: 2017.08.15 12:53:04 -04'00' i AFEC II PMEL QASP 1 August 2017

TABLE OF CONTENTS

SECTIONS PAGE

1. SCOPE

1.0. INTRODUCTION

1.1. PURPOSE

1.2. AUTHORITY

1.3. QUALITY CONTROL AND ASSURANCE

1.4. RELEASABILITY

2. KEY PERSONNEL

2.0. ROLES AND RESPONSIBILITIES

2.1. CONTRACTING OFFICER

2.2. CONTRACT MANAGER

2.3. FUNCTIONAL DIRECTOR

2.4. PROGRAM MANAGER

2.5. QUALITY ASSURANCE MANAGER

2.6. CONTRACTING OFFICER REPRESENTATIVE

2.7. SUBJECT MATTER EXPERT

3. CONTRACT SURVEILLANCE

3.0. INTRODUCTION

3.1. SURVEILLANCE METHODS

3.2. FREQUENCY

3.3. SURVEILLANCE DOCUMENTS

3.4. SURVEILLANCE TYPES

3.5. SURVEILLANCE TEAM COORDINATION

3.6. ASSESSING PERFORMANCE

3.7. NONCONFORMANCE NAMING

3.8. ASSESSMENT PROCESS

3.9. ADDITIONAL REPORTS AND DOCUMENTS

3.10. CONTRACT PERFORMANCE ASSESSMENT REPORTING SYSTEM (CPARS)

3.11. WIDE AREA WORK FLOW (WAWF)

3.12. RELEASABILITY

4. RISK MANAGEMENT

5. ATTACHMENTS

5.1. CONTRACTOR DELIVERABLES

5.2. SERVICE SUMMARY AND PERFORMANCE MEASURES

5.3. NONCONFORMANCE TEMPLATE

5.4. AUDIT PLAN TEMPLATE AND STANDARDIZED STATEMENTS

5.5. AUDIT REPORT TEMPLATE AND STANDARDIZED STATEMENTS

5.6. ICPR TEMPLATE

5.7. MAR TEMPLATE

5.8. WAWF PROCESS GUIDE

5.9. MICT PROCESS GUIDE

5.10. RISK ANALYSIS GUIDE

5.11. CAPT GUIDE

5.12. CDRL

5.13. RANDOM SAMPLE GUIDE

5.14. ACRONYMS AND COMMON WEBSITES

SECTION 1

SCOPE

1.0. INTRODUCTION. The Quality Assurance Surveillance Plan (QASP) is applicable to personnel performing contract surveillance audits in accordance with (IAW) Federal Acquisition Regulation (FAR) Part 46, Quality Assur-ance; Department of Defense FAR Supplement (DFARS) Subpart 201.6 and PGI 201.602-2 , Contracting Officer Responsibilities; Air Force FAR Supplement (AFFARS) Mandatory Procedures (MP) 5346-103, Contracting Office Responsibilities, The Quality Assurance Program; Air Force Instruction (AFI) 63-145, Manufacturing and Quality Management; AFI 63-138, Acquisition of Services; and organizational policy.

1.1. PURPOSE. This QASP provides a systematic method to assess and evaluate the services received from the Contractor. It defines the policy, procedures, and associated methods used by Contracting Officer Representatives (COR) for planning, preparing, performing, analyzing, and reporting Contractor performance. This QASP is a liv-ing document and will be amended as required. In the event this QASP conflicts with the contract or task order, the contract or task order will take precedence.

1.2. AUTHORITY. The Federal Acquisition Regulation (FAR) provides the Government the authority to audit the Contractor. Clause 52-246-4, Inspection of Services – Fixed Price, and clause 52-246-5, Inspection of Services – Cost Reimbursement, provide the Government with the authority to inspect and test all services during the term of the contract. In addition, the Contractor shall provide and maintain an inspection system acceptable to the Govern-ment. If the Contractor fails to take the necessary action to correct any services that do not meet contract require-ments, the Government may perform the services and charge the Contractor for any direct cost incurred by the Gov-ernment. Further, the Government may perform the services and reduce any fee payable by an amount that is equi-table under the circumstances.

1.3. QUALITY CONTROL AND ASSURANCE. The contractor, not the government, is responsible for all man-agement and quality control actions to meet the terms of the contract.

1.4. RELEASABILITY. The Contractor may receive a courtesy copy of the QASP in order to further enhance communications with the Contractor, but it shall not be considered as part of the contract. This provides the Con-tractor with information on Air Force requirements, the level of performance expectations and how the Air Force will confirm the services are received. All surveillance records are considered FOR OFFICIAL USE ONLY docu-ments, and must be marked and safeguarded as such.

SECTION 2

KEY PERSONNEL

2.0. ROLES AND RESPONSIBILITIES.

2.1. CONTRACTING OFFICER. The Contracting Officer is located in ACC AMIC/PKC, (757) 764-4961, DSN 574-4961. The Contracting Officer is the only person authorized to contractually obligate the government. While the COR and SMEs may evaluate the contractor's performance and document noncompliance, only the Contracting Officer may take formal action against the contractor for unacceptable performance. The Contracting Officer en-sures the contract is administered in accordance with the Federal Acquisition Regulation and other applicable direc-tives. CO duties are detailed in FAR subpart 1.602-2.

2.2. CONTRACT MANAGER. Responsibilities of the Contract Manager (CM) are to maintain the official rec-ords on the contract, and assist the Contracting Officer.

2.3. FUNCTIONAL DIRECTOR. The Function Director (FD) is located in ACC AMIC/PMS, (757) 764-9169, DSN 574-9169. The FD is responsible for executing specified management and oversight responsibilities of the ac-quisition process, and providing oversight for delivery of acquired services. The FD reviews contract performance documentation from the CORs on a regular basis to ensure Contractor performance is compatible with contract mis-sion objectives. The FD ensures personnel are properly trained and technically qualified to perform COR duties.

2.4. PROGRAM MANAGER. The Program Manager (PM) is located in ACC AMIC/PMSP, (757) 764-9181, DSN 574-9181. The PM is responsible for the overall program management responsibility and mission accomplish-ment. The PM serves as the principal technical expert on the multi-functional team, and provides guidance and ap-provals for the technical effort within the existing requirements of the contract. The PM is the interface between the requirement owner(s) and the CO, and also manages the program budget within appropriated funds.

2.5. QUALITY ASSURANCE MANAGER. The Quality Assurance Manager (QAM) is located in ACC AMIC/DRQP, (757) 764-3808, DSN 574-3808. The QAM is responsible for establishing the surveillance program and associated policies. The QAM trains COR personnel on the requirements of the surveillance program, and over-sees the execution of the program. The QAM assists with requirements development and definition, performance work statement (PWS) modifications, deficiency resolution, and manages the overall risk and past performance (CPARS) programs.

2.6. CONTRACTING OFFICER’S REPRESENTATIVE. The COR is ultimately responsible for implementing all QA matters and procedures. The COR must maintain both technical competency and evaluation proficiency in Contractor surveillance procedures. The COR must also have enough knowledge of Contractor activities to properly review and disposition monthly invoices. The COR cannot assume duties until assigned by the FD, completed COR required training, and designated by the CO.

2.6.1. Primary COR. The Primary COR (also known as Flight Chief, QA (FCQA)) is fully trained and qualified in the quality assurance program. The Primary COR is responsible for assisting with the development of audit sched-ules, consolidating monthly COR Reports, and providing guidance and advice to the QAM, PM, and CO. The Pri-mary COR also assists CORs in drafting nonconformance notices.

2.6.2. Lead COR. Lead CORs are seasoned CORs who have reached formal and informal training milestones, at-tained superior proficiency, and demonstrated functional expertise amongst their peers. Lead CORs are responsible for training of new CORs, assisting QAM and Primary COR with development of schedules, reports, processes, and other tasks as needed.

2.6.3. Alternate COR. Alternate CORs are fully trained and qualified in a related functional specialty, such as, but not limited to: Metrology, Civil Engineering, or Logistics. Alternate CORs provide technical support, expertise, or augment CORs for Contractor evaluations during Air Force Inspector General Unit Effectiveness Inspections. Al-ternate CORs cannot assume duties until they are assigned by the FD, completed all required training, and have been designated by the CO.

2.6.4. COR Limitations. CORs are the "eyes and ears" of the QAM, PM, and the CO when it comes to monitoring actual contract performance. CORs WILL NOT CHANGE/MODIFY THE CONTRACT, AUTHORIZE, OR AP- PROVE ANYTHING NOT IN THE CONTRACT, OR FORMALLY INTERPRET THE CONTRACT. The CO resolves these types of issues, in coordination with CORs, QAM, PM, and other functional area/technical experts.

2.7. SUBJECT MATTER EXPERTS. Subject Matter Experts (SME) are third-party personnel from other gov-ernment organizations who provide technical support or expertise to the program. These personnel are not ap-pointed as a COR by the CO.

SECTION 3

CONTRACT SURVEILLANCE

3.0. INTRODUCTION. The COR is responsible for a wide range of surveillance requirements that effectively measure and evaluate Contractor performance. The Contractor, not the Government, is responsible for Quality Con-trol (QC) actions to meet the terms of the contract. However, the Government cannot rely solely on the Contractor’s quality program but must have a method to oversee the entire contracting process. The Government does this through the QASP and can inspect or test all services called for by the contract, to the extent practicable, at all times and places, during the term of the contract. The COR’s contribution is comprised of professional, non-adversarial relationships, which include positive and open communications with the CO, CM, and the Contractor. The founda-tion of this relationship is built upon regular, objective, fair, and consistent COR evaluations of Contractor perfor-mance against contract requirements, and the ability to discuss inspection results, trends, and items of mutual inter-est with the Contractor. The CORs use the methods in this QASP, the PWS, AFIs, and Technical Orders to achieve this goal.

3.1. SURVEILLANCE METHODS. The CORs will conduct and document surveillance as detailed in this QASP.

Three primary methods of surveillance are normally used: one hundred percent inspection, periodic surveillance, and customer complaints. Information obtained from these activities may be used by the Government to ascertain whether or not Contractor performance is compatible with contract and mission objectives and by the CO for actions relating to the Contractor.

3.1.1. One Hundred Percent Inspection. This method is primarily used for Contractor submissions, products, and deliverables (tangible items). The COR, with assistance from the PM if necessary, accepts or rejects these products based on the criteria and performance standards as outlined in the SS.

3.1.2. Periodic Surveillance. This type of surveillance consists of evaluating products/services not observed via one hundred percent inspection or by customer survey, and is implemented at a pre-defined frequency. An example of periodic surveillance is performing quarterly inspections of Test, Measurement, and Diagnostic Equipment (TMDE) in deferred status, or priority calibration services.

3.1.3. Customer Survey or Customer Complaint. Although usually not a primary method, this is a valuable sup-plement to other, more systematic methods of surveillance. For example, in a case where periodic surveillance indi-cates unsatisfactory service, customer complaints can be used as substantiating evidence. In certain situations, where customers complain consistently when the quality of performance is poor, customer surveys and customer complaints may be a primary surveillance method, and customer satisfaction an appropriate performance standard.

Information obtained from customer complaint activities may be used by the CO for actions relating to the Contrac-tor.

3.2. FREQUENCY. All functional areas within the PWS will be inspected at least once per year.

3.3. SURVEILLANCE DOCUMENTS. Deliverables and the Services Summary comprise the surveillance docu-ments used to monitor contractor performance.

3.3.1. Document Descriptions.

3.3.1.1. Deliverables. Deliverables (Attachment 1) required to be provided by the Contractor will be reviewed for timeliness, accuracy, and format. If a deliverable’s due date falls on a weekend or holiday, the Contractor shall sub-mit the deliverable on the last work day prior to the due date. If a conflict exists between the QASP and the PWS, the PWS takes precedence.

3.3.1.2. Service Summary. The SS (Attachment 2) is a list from the PWS of critical tasks, the performance objec-tive, and the performance measure that must be performed by the Contractor.

3.4. SURVEILLANCE TYPES. Three primary types of surveillance will be used: Initial Contract Performance Review (ICPR), Virtual Audit, and Onsite Audit.

3.4.1. ICPR. Initial evaluation of Contractor performance is a joint determination by a multi-functional team that may include CO, PM, QAM, CORs and SMEs to determine if the contractor has successfully started performance and completed transition, is fully operational, and is within the estimated cost, schedule, and performance parame-ters of the contract (see paragraph 3.9.1 for more information).

3.4.2. Virtual Audit. Virtual Audit consists of evaluating and/or validating information and data filed on Govern-ment or Contractor information systems, and any contract deliverable requirements list (CDRL) items or other re-ports, inventories, procedures, plans, or other Contractor provided deliverables. Additionally, results of third party/external inspections may be included.

3.4.2.1. Virtual Audits will be conducted monthly by accessing various systems the Contractor may use, such as:

Management Internal Control Toolset (MICT), Management Information Data System (MIDS), PMEL Automated Management System (PAMS), etc.

3.4.2.2. Virtual audits may focus on a single site, or take an enterprise approach by auditing all sites on a Task Or-der (TO). For example, all Root Cause Analysis, Trend Analysis, or Management Reviews for a TO may be audited to identify if the Contractor is taking an approach to determine and minimize risk at an enterprise level.

3.4.3. Onsite Audit. Onsite audits will be scheduled using a risk based approach. The Primary COR will evaluate all available data, to include: performance data, changes in personnel, known site issues, the AMIC risk tool, etc., to determine the frequency CORs will visit a site.

3.5. SURVEILLANCE TEAM COORDINATION.

3.5.1. Periodic Meetings. PM, QAM, CORs, and SMEs will meet periodically to discuss contract performance and surveillance efforts. Topics of discussion will include, but are not limited to, information from audits, nonconform-ance, known weak or problem areas, and any requirement owner complaints.

3.5.2. Weekly Meetings. At least weekly, the PM and QAM will review the previous weeks audits, discuss future audits, schedules, etc.

3.5.3. Post Audit Discussions. The first duty day following an onsite audit, the CORs will brief the QAM on audit findings, including subject evidence gathered during the audit, but not in the report. This additional information will be used to determine the level of risk assigned to the site.

3.6. ASSESSING PERFORMANCE. CORs must document Contractor performance, as well as the scope and pur-pose of any inspections. The COR will also assess and rate Contractor performance as it relates to contract require-ments. Performance assessment may take the form of assessment results from the surveillance reports, Contractor monthly reports, nonconformance notifications, and third party inspections. Each builds on the other and provides a thorough documented history of Contractor performance.

3.6.1. Inspections. CORs may perform inspection/surveillance at any time during contract performance IAW FAR 252.246-2 through 5, as applicable. A copy of all audit and inspection documentation will be maintained until con-tract closeout, unless otherwise approved by the CO.

3.6.2. Nonconformances and Deficiencies. Audit or inspection results that fall below contractual standards shall be identified as nonconformances or deficiencies.

3.6.3. Types of Nonconformance (Deficiency).

3.6.3.1. Minor Nonconformance: A minor nonconformance is a nonconformance, which by itself does not ad-versely impact mission, safety of personnel and/or equipment, performance (quality), schedule (delivery), or cost. A minor nonconformance may increase risk to the Government.

3.6.3.2. Major Nonconformance: A major nonconformance is a nonconformance that adversely impacts mission, safety of personnel and/or equipment, performance (quality), schedule (delivery), or cost. A major nonconformance also increases risk to the Government. An example of increasing risk would be significant number of recurring non-conformances, which indicates inadequate preventive measures or actions, thus lowering the Government’s confi-dence that quality services will be provided on time and at cost.

3.6.3.3. Nonconformance Notification. If a deficiency is found during inspection of the Contractor’s performance, the QAM/COR will determine the type of notification based on the category of the nonconformance (see Table 3.1).

3.6.3.3.1. First or Second Notices (Attachment 3). As shown in Table 3.1, First and Second Notices provide written notification from the Government to the Contractor of minor nonconformances. Initial minor nonconformances should be addressed with a First Notice to the applicable Contractor representative. The COR will prepare the no-tice, and submit to the QAM for review. The QAM will forward the notice to the PM for issuance. If the Contractor fails to resolve the issue as documented in the Notice, the COR should elevate the issue to the next level (Second Notice or CAR). Corrective/preventive actions initiated and/or completed by the Contractor will be verified by COR on the scheduled inspection and documented accordingly.

3.6.3.3.2. Corrective Action Requests (CAR) (Attachment 3). As shown in Table 3.1, a CAR provides written noti-fication from the Government to the Contractor of significant performance discrepancies. The COR will recom-mend to the PM/CO a CAR be issued when previous notifications failed to result in corrective actions, or when se-vere deficiencies exist. The CAR will be routed NLT ten duty days after the event to the CO for review and signa-ture, as the CO is the only person who has authority to issue a CAR to the Contractor. To satisfactorily close-out a CAR, the following criteria must be met: the Contractor must meet the suspense; the corrective action must have already begun; and the Contractor's actions must correct the deficiency and if applicable, be within specified stand-ards.

3.6.3.4. Reviewing the Corrective Action Plan (CAP). If the corrective action is deemed UNSATISFACTORY by the QAM, annotate the CAR accordingly and forward it to the PM/CO with a brief explanation. There are four pos-sible outcomes when reviewing a CAP: Accept and remain open, Accept and Close, Partial Acceptance, and Rejec-tion.

3.6.3.4.1. Accept/CAR Remains Open. If the Contractor provides an effective CAP in section D-2 of the CAR, but the implementation will take several weeks or months, the CAR will remain open until full implementation is achieved. The determination to close the CAR will be made after an evaluation of the corrective actions can be made to sufficiently determine the effectiveness. The QAM will recommend closure to the PM/CO and route the CAR and acceptance letter for signatures.

3.6.3.4.2. Accept/Close the CAR. If the CAP is deemed effective and implementation has been achieved, the QAM will review all data and determine if closure is warranted. Follow up will be performed during successive audits.

3.6.3.4.3. Partial Acceptance. Section D-2 of the CAP appears to provide an effective resolution with exceptions.

For example, the contractor’s plans for items (a-c) appear to provide sufficient depth, but item (d) does not. In sec-tion D-3, provide sufficient detail so the Contractor understands why (d) was rejected.

3.6.3.4.4. Rejected. If the Contractor’s CAP fails to provide sufficient detail to resolve the nonconformance, the response can be rejected. In section D-3 of the CAR, provide specific reason for the rejection.

3.6.3.5. The QAM will maintain a nonconformance log documenting all First and Second Notices, and CARs. The log will include the date sent to the Contractor, deficiency, due date for response and actual response date. The log can be found on SharePoint in the Nonconformance folder.

TABLE 3.1 – NONCONFORMANCE NOTIFICATIONS

Nonconformance Category Notification Type Minor

Step 1: First Notice Step 2: Second Notice

Major Corrective Action Request (CAR) https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FNonConformance&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d

3.7. NONCONFORMANCE NAMING. Nonconformance notifications will be labeled in the following format:

YYMM00xABCD-(1,2, or C), where ‘x’ equals the number of Notices or CARs the site received that month; 1 for First Notice, 2 for Second Notice; C for CAR. For example: 1707001ANDE-1, 1707001ANDR-2, or

1707001ARNO-C.

3.8. ASSESSMENT PROCESS. When planning audits, the COR shall determine the intensity, frequency, and surveillance areas based on past performance results, risk, contract requirements, and other directed (mandatory) requirements. The paragraphs below are provided as a guide for CORs to utilize when preparing to conduct an as-sessment, conducting the assessment, and reporting the results of that assessment to the FD, Contractor, and QAM.

3.8.1. Develop Audit Schedule. The audit schedule covers both onsite and virtual audits, and the frequency audits will occur. The audit schedule will be created by the Primary COR with coordination through the QAM and PM.

The audit schedule is located on SharePoint in the Audit Schedule folder and will be modified monthly based on current focus areas, potential risk, and previous audit results

3.8.2. Develop Audit Plan.

3.8.2.1. Review previous audits in the appropriate Task Order/site folder, located on SharePoint in the Audit Re-ports folder. Identify any previously identified First/Second Notices or CARs and ensure to follow-up on corrective actions as required. Be aware of past trends/issues.

3.8.2.2. Discuss overall Contractor performance with other CORs, QAM, PM, FD, and CO as required. Determine if they have any issues/concerns and add those to your audit. For example, evaluating a CAP, or following up on previously identified issues.

3.8.2.3. Review the Service Summary requirements and CDRLs (Attachment 2) that are due during the audit period.

Remember, we review all CDRLs for timeliness and accuracy. Coordinate with the PM or CO as required.

3.8.2.4. Identify any contract changes, such as contract modifications or changes in Contractor personnel (hiring, firing, resignations, duty changes) and adjust your audit accordingly. Consider changes in mission or workload.

3.8.2.5. Access PAMS, SharePoint, MIDS, and the COR Automated Planning Tool (CAPT) to gather as much data in determining the risk areas of the planned audit.

3.8.2.6. Review Contractor’s internal audits, management reviews, trend analysis, and any other internal documen-tation. Identify any findings and the status of the Contractor corrective actions. Follow-up/validate the status. This is often the most overlooked part of audit planning.

3.8.2.6.1. It is not always necessary to generate a First/Second Notice, or a CAR if the Contractor has self-identified a deficiency. It can be identified in the audit report for tracking and follow-up purposes. A decision to issue a no-tice or CAR will be made by QAM/Primary COR.

3.8.2.6.2. Review what the Contractor has reviewed. For example, if the Contractor has identified issues in a partic-ular Measurement Area or Section, CORs do not have to target the same area. Give the Contractor time to correct their deficiency and follow-up in later audits.

3.8.2.6.3. CORs must familiarize themselves with the Contractors MIDS, by reviewing each folder and learning the location of required products. Each MIDS is different, and being familiar with all MIDS will save time in audit planning.

3.8.2.6.4. Contract requirements encompass the entire contract, not just the PWS. CORs must be familiar with Con-tractor operations by reviewing QMs, checklists, work instructions, site operating procedures, etc.

3.8.2.6.5. Contracts are performance base, not AFI, TO, or Publication driven. Unless an AFI, TO, or Publication is mandatory as stated in Appendix C of the Contract, the Contractor may develop their own method or procedures to meet the requirement.

https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FAudit%20Schedule&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FAudit%20Reports&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FAudit%20Reports&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d

3.8.2.6.6. Maintain good documentation. CORs must document everything observed in the audit report. Thorough notes should be retained along with any objective evidence supporting the non-compliance. Consider the mind-set “If it isn’t documented, it didn’t happen”. Not all notes go into the report, but should be considered for additional information in CAPT.

3.8.2.6.7. CORs may identify potential issues outside of the audit plan. If this occurs, finish the planned audit, then focus on the potential issue (unless critical/imminent safety, environmental, or health issue – take action immedi-ately).

3.8.2.6.8. Deviations from the audit plan happen. If a deviation is needed, call or email the QAM or Primary COR to discuss alternate options.

3.8.2.6.9. Do not assume anything. If something does not seem correct, ask a fellow COR, Primary COR, or QAM.

Do not be afraid to ask questions.

3.8.3. Special Interest Item. Special Interest Items (SII) may be identified by the QAM for investigation. Inputs for SIIs may be solicited from the CO, PMs, or other members of the multifunction team. SIIs may be rotated quar-terly, or at longer intervals, such as after each site has had one inspection. Each SII is unique, so the QAM and Pri-mary COR will develop an audit strategy for the CORs to incorporate into their plan.

3.8.4. COR Audit Planning Tool. COR Audit Planning Tool (CAPT) is a risk management program developed for audit scheduling and planning purposes. CAPT analyzes data from multiple sources, and assigns an overall score for technical and program risk. Risk indicators should be utilized during all stages of audit planning.

3.8.5. Management Internal Control Toolset. Contractors are required to maintain self-assessment communica-tors (SAC) in the Management Internal Control Toolset (MICT). The contractor must report compliance or non-compliance, with objective evidence, in each assigned SAC. CORs must validate the contractor has accomplished the SAC, review findings, observations, estimate completion dates, and the status of corrective/preventative actions.

Table 3.2 lists the required SACs for AFEC PMELs.

TABLE 3.2 – AFEC Required MICT Self-Assessment Communicators MAJCOM/Directorate Abbreviation Due Date HAF/All Directorates AFI 21-113 AFI 21-113 Air Force Metrology and Calibration Program HAF/All Directorates A4L (23-111) Management of Government Property in Possession of Air Force HAF/All Directorates AFMAN 17-1203 Information Technology Asset Management HAF/All Directorates RNI Repair Network Integration (RNI) HAF/All Directorates None Stage 1-All Shops HAF/All Directorates AFI 21-101 Test, Measurement and Diagnostic Equipment (TMDE) AFMC Only AFMC/All Directorates ESD TO 00-25-234 – Electrostatic Discharge Control AFMC/All Directorates Chp 10 Tool &

Equipment Mgt AFI 21-102 Chp 10 Tool & Equipment Mgt AFMCSUP (For AFSC Only)

3.8.6. Audit Plan. After researching/developing the audit plan, use the current Audit Plan Template and Standard-ized Audit Plan Statements (Attachment 4) to produce the plan. The most current template and statements are lo-cated in the Template folder on SharePoint. Submit the audit plan to the QAM and Primary COR no later than five duty days prior to the audit start. The QAM will review and approve the plan, returning it to the COR for signature.

The COR will convert the final plan to PDF and digitally sign it. QAM or COR will will upload signed plans to the Audit Plan folder in SharePoint.

3.8.7. Performing Onsite Audit.

https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FProgram%20Specific%2FTemplates&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FProgram%20Specific%2FAudit%20Plans&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d

3.8.7.1. In/Out Brief Local Leadership, if requested. Provide a brief discussion concerning what your plan is or what was seen during the audit. A general overview will suffice. Ask if the Leadership has any concerns for the PM, CO or may-be applicable to the audit.

3.8.7.2. Email signed audit plan to the site manager, with a courtesy copy to the QAM, no earlier than the day be-fore audit start. Review audit plan with site manager or designated representative(s). The review should be brief, but effective. Discuss objectives of the audit, MAs to be covered, follow-ups, etc. Ensure the Contractor has no doubts about COR objectives for the audit.

3.8.7.3. Communication (both verbal and non-verbal), appearance, and disposition are key elements to a profes-sional COR. CO guidelines and COR “Do’s and Don’ts” must be adhered to at all times.

3.8.7.4. Remain professional at all times. Assess performance objectively, compare performance to actual contract requirements. Look only at the objective evidence (not “hear say” or your own opinions, unless you can corroborate those subjective opinions with objective evidence), and make solid decisions based on these facts.

3.8.7.5. Do not interfere with contractor operations or stop work on an item. Let the process play out. CORs shall not insert themselves into the process unless safety of personnel or equipment damage is imminent.

3.8.7.6. Fully explain expectations to the technician. Put the technician at ease. After the audit, explain to the tech-nician any observations, potential issues, and if further research is required.

3.8.7.7. Give the technician space to work during audits. Multiple CORs should not gather around a technician.

Exceptions are if COR is performing an EPE on QA, or if the QAM/Primary COR is performing an EPE on the

COR.

3.8.7.8. Do not be checklist driven. Maintain an open mind and don’t jump to conclusions. Gather data and ana-lyze the facts. Consult with other CORs. Be objective.

3.8.7.9. Maximize time onsite. It is not necessary to observe every step of the calibration. Once target MA has been observed, it is OK to end the audit item and move on. Additionally, a COR can observe more than one audit item at a time.

3.8.7.10. Assess performance as a whole instead of through individual findings and observations. Isolated, minor issues may seem inconsequential, however, a number of minor problems in a specific area may indicate a deeper, systemic problem. How does it affect the overall performance of the contract? What is the risk? When viewed to-gether do they indicate a moderate or high risk? If this is the case, a CAR may be warranted, rather than a first or second notice.

3.8.7.11. The goal is for the Contractor, through their internal QMS, is to identify issues and implement corrective actions. Is the problem already identified and in-work by the Contractor, is the issue something you found that should have been identified by the Contractor, or is it something else?

3.8.8. Audit Report.

3.8.8.1. Take good notes during your audit. These will be crucial when you go to write your report. In addition, the report should be updated daily. Don’t wait until the last day of the audit to draft the report. Drafts should be sent nightly to the whole COR team for inputs. If you wait to write your audit report, you run the risk of forgetting criti-cal information.

3.8.8.2. Observation may be used to describe Contractor performance that meets requirements, but may require at-tention in the future. An observation may not have a clear contract reference, or be directly tied to the contract QMS. Observations are Low Risk, non-critical discrepancies with minimal impact to contract performance. Multi-ple iterations of observations during an audit may lead to negative performance trends and increased risk. Contrac-tors are not required to track observations in their QMS.

3.8.8.3. Concern is used to describe Contractor nonconformance with a solid contractual reference. Concerns may be identified using PWS, AFI, TO, or Contractor QMS references. The Contractor must track concerns in their QMS. An example of a Concern is identifying a non-conformity while auditing an SS2 item.

3.8.8.4. Finding is used to describe audit results below standards and are identified as nonconformance. If multiple concerns are documented during an audit, the COR may roll the concerns up into a single Finding. Findings shall be further classified as a First or Second Notice, or a CAR.

3.8.8.5. Review the draft audit report with the Contractor and discuss any nonconformance identified during the audit. Remember, these are initial observations only, and may not make it into the final report. Inform the Contrac-tor if any items are left open, or require further research before a decision is made. The final decision on noncon-formance will be made by the QAM. If issues remain unresolved, or require further research, a draft report does not have to be given to the Contractor. In this case, a verbal out brief detailing the issues is sufficient.

3.8.8.6. Finalize the draft report and forward to the QAM NLT three duty days after audit completion. The QAM will review, edit or return for corrections if required, and sign the report. Signed reports are forwarded NLT 10 duty days after the audit to the PM for issue to the Contractor. QAM will post signed reports to the Audit Reports fold on SharePoint.

3.8.8.7. Reports shall be developed using the current Audit Report Template and Standardized Audit Statements (Attachment 5). The most current template and statements are located in the Template folder on SharePoint.

3.8.8.8. Reports shall be named in the following format: YYMMABCD-R (Draft), where ABCD is the first four letters of the base; for example: 1707ANDR-R (Draft).

3.8.8.9. Update audit results in CAPT NLT three duty days following audit completion.

3.9. ADDITIONAL REPORTS AND DOCUMENTATION.

3.9.1. Initial Contractor Performance Review. AFI 63-138, Chapter 6, paragraph 6.4, states the initial contractor performance review (ICPR) is a joint determination that the Contractor has successfully started performance, com-pleted transition, is fully operational, and is within the estimated cost, schedule, and performance parameters of the contract.

3.9.1.1. The ICPR shall take place within 30 days after the contractor assumes full performance responsibility (for your purposes - after contract start). For all contracts which are >$100M, the ICPR report shall include an assess-ment of schedule, management, technical, and cost performance. Negative variations in cost, schedule, staffing, and performance shall be reported with an assessment of the root causes and corrective action plan.

3.9.1.2. COR responsibility is to ensure whatever the Contractor put in their proposal and the Government included as part of the contract, is reviewed (most O&M contracts include the Contractors entire proposal as an Appendix to the PWS). Has the contractor started performance? (There are usually deliverables due to the Government within the first 30-days, e.g., CDRL) How is the contractor meeting their Transition Plan? Are they on track with schedule and milestones? Where do they stand in regards to their QCP? Have all key personnel been hired (if not, are opera-tions impeded?) These questions must be asked and reported. Report contract performance assessment values using the current template (Attachment 6). The most current template can be found in the Template folder on SharePoint.

3.9.1.3. Report contract performance assessment values in the following method: Green--No issues; Yellow--Is-sue(s) but contractor has an adequate mitigation or corrective action plan in place; Red--Issue(s) with inadequate or no contractor mitigation or corrective action plan. Any "red" assessment shall include the government proposed ac-tions for the failing Contractor. Additionally, any significant modifications to the contract made since contract award shall be included in the initial performance report as special interest items.

3.9.2. Primary COR Monthly Activity Report. Primary COR will provide the Monthly Activity Report (MAR) (Attachment 7) to the QAM NLT the 21st day of each month. The monthly activity report will include information https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FAudit%20Reports&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FProgram%20Specific%2FTemplates&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FProgram%20Specific%2FTemplates&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d from the previous month’s activities, to include: Audit Schedules and Reports (Onsite and Virtual); nonconform-ance log with nonconformance forms; service summary inspections results with metrics; and risk reports screenshots from CAPT. Additional comments related to performance or concerns identified, to include general program related issues.

3.9.3. QAM Monthly Reporting.

3.9.3.1. MAR and Executive Summary. The QAM will review the MAR, create the Executive Summary, and post both documents to Monthly Activity Report (MAR) and Executive Summary folders in SharePoint NLT the 28th of each month.

3.9.3.2. Monthly Virtual Audit Schedules. The QAM will develop a Virtual Audit schedule for the following month. The report will be by Contractor and Task Order. Once completed, it will be sent to the CO and PM, NLT the 21st day of the preceding month. QAM will post signed virtual schedules to the Audit Schedule SharePoint folder.

3.9.4. COR DELIVERABLES. DRQP is responsible for producing and delivering the documents in Table 3.3, within the listed due dates. The QAM shall make every effort to deliver the products in a timely manner to the MFT partners.

TABLE 3.3 – COR Deliverables Deliverables Delivered

To Owner Due Date

Audit Plan, Onsite QAM COR No Later Than (NLT) 5 duty days prior to audit

Audit Report, Monthly Desktop QAM CO FCQA NLT 15th of the next month

NLT 21st of the next month

Audit Notification Message Contractor

Commander

MFM

FCQA NLT 2 weeks prior to audit start

Audit Report, Onsite PM / CO QAM NLT 10 duty days from audit completion Audit Schedule, Monthly Virtual PM / CO QAM NLT 21st of month prior Audit Schedule, Onsite PM / CO FCQA NLT 21st of month prior CAR PM / CO QAM NLT 10 duty days after discovery Executive Summary DRQ QAM NLT 28th day of the month

Monthly Activity Report (MAR) QAM DRQ FCQA NLT 21st day of the month

NLT 28th day of the month

Notice (1st or 2nd) PM / CO COR QAM NLT 10 duty days after discovery

3.10. Annual Contract Performance Assessment Reporting System. The Contractor Performance Assessment Reporting System (CPARS) https://www.cpars.gov is the Department of Defense (DOD) Enterprise Solution for collection of Contractor Past Performance Information (PPI) as required by the Federal Acquisition Regulation (FAR). CPARS is a web-enabled application that collects and manages a library of automated Contractor report cards. An annual CPAR will be written on all contracts task orders that exceed $1M total contract value. The PM or COR will be designated as the Assessing Official Representative in CPARS; the written assessment in CPARS will begin annually from the start date of the contract and forwarded to the Contract Manager within 60 days of the contract start date anniversary. The COR will use objective evidence from previous, monthly Services Summary, inspection logs, First and Second notices, CARS, and correspondence letters for CPARs assessment

3.11. WIDE AREA WORK FLOW. Wide Area Work Flow (WAWF) is an automated web based system used by DoD components to electronically process requests for payment from Contractors performing on Government con-tracts. The COR will ensure the document entries are IAW the applicable contract (CLIN, quantity, unit of issue, price, total price, description of services, etc). See Attachment 8 for further instructions.

https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FMonthly%20Activity%20Report%20%28MAR%29&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FProgram%20Specific%2FExecutive%20Summary&FolderCTID=0x012000D3F3A9C914DE254A9C8DC59C6D7BB689&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://cs3.eis.af.mil/sites/AC-AQ-01-76/DRQP/Forms/AllItems.aspx?RootFolder=%2Fsites%2FAC%2DAQ%2D01%2D76%2FDRQP%2FAudit%20Schedule&View=%7bD1B329F3-03C4-4FAF-A9C7-BBA8DA9444D6%7d https://www.cpars.gov/

3.12. RELEASABILITY. ALL surveillance records are considered FOR OFFICIAL USE ONLY documents, and must be marked and safeguarded as such.

SECTION 4

RISK MANAGEMENT

4.0. RISK MANAGEMENT.

4.1. Risk management consists of risk planning, identification, analysis, mitigation planning, mitigation plan imple-mentation, tracking and documentation. Through risk management, the focus of Government surveillance is to di-rect resources toward high and moderate risk, with minimal resources directed toward low risk.

4.2. Risk management is factored into audit intensity, frequency, and surveillance. Every effort shall be made to adjust Government surveillance methods based on risk analysis/assessment. CORs will receive risk training from the HQ ACC AMIC Risk Manager. Additionally, training slides can be found on DRQ SharePoint. See Attachment 10 for the Risk Management Analysis Assessment Chart to assist in risk determination.

4.3. PREPARE FOR ANALYSIS.

4.3.1. Review Contract and identify all mandatory surveillance items (SS, CDRL, etc).

4.3.2. Review the AFEC PMEL Risk Management Plan (RMP). Identify High, Moderate, and Low risks docu-mented in the RMP.

4.3.3. Determine if the document RMP risk should be incorporated into the audit being planned. Emphasis should be on High and Moderate risks (more frequent surveillance), than Low risks (less often surveillance items).

4.3.4. Determine time frame for data collection (since beginning of contract, 12 months, 6 months, etc). Collected data from all available sources, Government, Contractor, or customer.

4.4. PERFORM RISK ANALYSIS.

4.4.1. Decide how to evaluate the data to determine the Likelihood/Probability of Risk and the Consequence/Impact of the risk (using number of observations, concerns, how often an onsite occurred, detected/undetected noncompli-ance, etc).

4.4.2. Review the Risk Analysis Process Assessment Guide (Attachment X) to using the data to determine the prob-ability and impact of the risk. If data is not available, the risk is Moderate Probability/Consequence until data is available. When data is not available for Safety of flight, loss of life, total mission failure, always rate the risk High.

4.4.3. Based on risk analysis results, document each risk High, Moderate, or Low as applicable.

4.5. MITIGATION STRATEGY/AUDIT EXECUTION.

4.5.1. Document a mitigation strategy for each risk: Assume, Control, Transfer, or Avoid. Next document surveil-lance intensity/frequency and implement the mitigation plan.

4.5.2. Document the risk analysis in the Active Risk Management (ARM) spreadsheet.

4.5.3. Perform Trend Analysis.

4.5.4. Perform another Risk analysis and make adjustments the strategy based on surveillance and TA results.

ATTACHMENT 1

CONTRACTOR DELIVERABLES

DELIVERABLES

Deliverable PWS Reference Delivered To Owner Due Date

Reimbursable Expense Documentation Report

(CDRL A001)

PWS A-2.2.1 PM PM No Later Than (NLT) 5 work days after the end of the month

Financial Management Report

(CDRL A002)

PWS A-2.2.2 PM PM NLT 15 business days after the end of the quarter

Technical Reports

(CDRL A003) PWS A-2.2.3 PM, CO,

COR PM Up to twelve per fiscal year, 10 days after Contractor publication

Quality Manual

(CDRL A004) PWS A-2.4.1 PM, COR PM

NLT 30 days after the start of the transition period or NLT 15 busi-ness days after the end of the fiscal year

Property Control Plan (CDRL A005) PWS A-6.2 PM PLG NLT 15 business days after the end of the fiscal year Quality Program Ac-tivity Summary

(CDRL A006)

PWS A-7.11.1 PM, COR PM NLT 10 business days of the subse-quent month

Environmental Protec-tion Plan

(CDRL A007)

PWS A-8.1.3 PM PM NLT 15 Business Days after the end of the fiscal year

Pollution Incident Spill Report

(CDRL A008)

PWS A-8.1.3.1 PM PM

NLT COB next business day and updates shall be provided as an ad-dendum to the CDRL

Environmental Health and Safety Plan

(CDRL A009)

PWS A-8.2.2 PM PM NLT 10 business days after revision

Mishap Report

(CDRL A010) PWS A-8.2.2.1,

8.2.2.2, 8.2.2.3 PM PM

NLT COB next business day and updates shall be provided as an ad-dendum to the CDRL

Quality control/assur-ance document and training records

PWS A-2.1.2.1.3 MIDS COR NLT 10 days of completion

ATTACHMENT 2

SERVICE SUMMARY (SS) AND PERFORMANCE MEASURES

SERVICE SUMMARY (SS) AND PERFORMANCE MEASURES

The Services Summary is a list of critical tasks that must be performed by the Contractor, the performance thresh-old, and the method of surveillance the COR will use to validate/inspect these tasks. These items must be assessed each month, with results documented in the audit report or monthly activity summary. If a conflict exists between this QASP and the applicable task order/PWS, the task order/PWS takes precedence

SERVICE SUMMARY 1

PERFORMANCE OBJECTIVE PERFORMANCE THRESHOLD METHOD OF SURVEILLANCE

Implement, maintain, and comply with a comprehensive Quality Man-agement System IAW TO 00-20- 14.

(PWS Section A, paragraph 2.4)

No major nonconformance is al-lowed. If a major nonconformance is discovered, the QMS will be con-sidered noncompliant.

Method: Sampling Who: COR What: Quality Manual, Manage-ment Review, Internal Audit (if applicable), Trend Analysis, Root Cause Analysis, MICT, others as required Where: Virtual, Onsite When: Monthly How: Random and 100% based on type, size, and scope of audit.

Digital review of all documents with follow-up as required onsite.

Purpose: To address the minimum inspection requirements for Virtual Audit items identified in the monthly Contract Surveillance Audit Schedules. These assessments will help provide focus for onsite audits based on risk indicators, and capture performance trends to aid in determining onsite visit frequency/intensity.

Inspection Level: Inspection levels will be performed at normal levels IAW ANSI Z 1.4 unless otherwise speci-fied or as contractor performance dictates.

References: Always understand the reference listed in the Master Schedule to ensure audit is IAW the PWS re-quirements.

1.0 Management Internal Control Toolset (MICT): Retrieve the appropriate Self-Assessment Communica-tors (SAC) from the flight MICT manager. Perform a random sample IAW ANSI Z 1.4 and inspect MICT SAC line items for accuracy, review supporting documentation, and any open observations.

2.0 Management Review (MR): TO 00-20-14, paragraph 10.4

2.1 Retrieve MR from contractor Management Information Data System (MIDS). Review for compliance.

Compare to previous MR and consider, was the current MR a copy and paste from previous MR?

2.2 Consider content of MR. Did it effectively communicate required information? Were all required topics ad-dressed with sufficient supporting evidence? Was a culture of Continuous Process Improvement (CPI) evident?