QASP_(Draft).pdf

Quality Assurance Surveillance Plan

RQ-4/U-2/T-38 Contract Aircrew Training and Courseware Development

FA4890-17-R-0013

COMPLIANCE WITH THIS DOCUMENT IS MANDATORY

NOTICE: This Document is available in digital format through ACC AMIC/DRQQ

THIS IS A DRAFT

COPY PROVIDED TO CONTRACTORS AS REQUESTED

THE PLAN SHOWS HOW THE GOVERNMENT WILL CONDUCT

CONTRACT SURVEILLANCE AND

IS NOT A CONTRACTUAL DOCUMENT

Pages: 31 19 Apr 2017

TABLE OF CONTENTS

Section Page

1.0 SCOPE 3

1.1 Purpose 3

1.2 Application 3

2.0 MANAGEMENT LOCATIONS AND KEY PERSONNEL 3

2.1 Contract Management Locations 3

2.2 Contract Management Team 3

2.3 Contractor Key Personnel 4

3.0 CONTRACT SURVEILLANCE 5

3.1 Surveillance Methods 5

3.2 Surveillance Documents 5

3.3 Assessing Performance 20

4.0 WIDE AREA WORKFLOW (WAWF) 24

4.1 Validating Receiving Reports 24

4.2 Certifying Receiving Reports 24

4.3 Rejecting Reveiving Reports 24

5.0 CONTRACTING OFFICER REPRESENTATIVE (COR) FOLDER 25

TABLES

1 Deliverables 6 2 Services Summary 9 3 Non-conformance Notifications 21 4 COR Folder Tabs and Contents 22

ATTACHMENTS

1 First or Second Notice Template 26 2 Corrective Action Request (CAR) Template 27 4 Contract Monitoring and Surveilance Report 28 5 Sample SME Nomination Letter 29 6 Sample SME Relief From Duties Letter 30 7 Guidelines For Courseware Deliverable Evaluation 31

1.0 SCOPE. The Quality Assurance Surveillance Plan (QASP) is applicable to personnel performing contract surveillance audits in accordance with (IAW) Federal Acquisition Regulation (FAR) Part 46, Quality Assurance; Department of Defense FAR Supplement (DFARS) Subpart and PGI 201.602-2 , Contracting Officer Responsibilities; Air Force FAR Supplement (AFFARS) Mandatory Procedures (MP) 5346-103, Contracting Office Responsibilities, The Quality Assurance Program; Air Force Instruction (AFI) 63-501, Air Force Acquisition Quality Program; and organizational policy. It is designed to provide the Contracting Officer Representative (COR) a systematic surveillance method for each service identified in the Performance Work Statement (PWS) for each contract. The owning Functional Director (FD) and COR develop the methods for administering and evaluating contract performance. The Program Manager (PM)/COR or Contracting Manager (CM), will review these methods, which are then approved by the Contracting Officer (CO).

1.1 Purpose. This document defines the surveillance policy, procedures, and associated methods used by CORs for planning, preparing, performing, analyzing, and reporting Contractor performance. This QASP provides a systematic surveillance method to evaluate the services received from the Contractor, and NOT to address the details of how the Contractor accomplishes the work.

1.2 Application: This QASP is used by all parties involved with assessing Contractor performance and is for the use of FDs, CO, CMs, CORs and SMEs.

2.0 MANAGEMENT LOCATIONS AND KEY PERSONNEL.

2.1 HQ ACC AMIC/PKB. The address for Contracting Management is:

129 Andrews Street, Suite 110 Langley AFB, VA, 23665

2.2 Contracting Management Team.

Contracting Officer (CO). CO duties are detailed in FAR subpart 1.602-2. The CO is the only Government agent authorized to award or modify contracts; therefore, the CO is the only person authorized to contractually obligate the Government. The CO is also the individual who will sign any notifications to Contractors regarding performance issues and accepts Government surveillance methods utilized for each TO. The CO approves/appoints all CORs assigned by the Functional Director/Functional Commander.

2.2.1 Contract Manager (CM) The CM is the Government agent within the contracting office who performs day-to-day administration of the contract. The CM/CA may also be the CO if the person has a warrant.

2.2.2 Quality Assurance Program Coorinator (QAPC). The QAPC is the Government Agent responsible for the quality assurance program/polices, ensuring contracts meet contracting requirements associated with the AMIC’s quality assurance program. Tracks and reviews all nonconformances, assisting CORs in drafting nonconformances notices (1st/2nd Notice/CAR) when requested. Supports the acquisition team in the development of contract quality assurance requirements specifically reviewing the PWS and QASP to ensure that requirements are clearly stated, enforceable, and measureable. Trains CORs and COR Management (e.g. COR Supervisor, Functional Commanders/Directors (FC/FD) on the contracting requirements associated with the quality assurance program and MAJCOM/DRU procedures.

2.2.3 The QAPC can be reached at DSN: 573-9372 or Commercial 757-764-9372

2.2.4 Program Manager (PM). The PM assists customers with requirements development and definition, PWS modifications, deficiency resolution, and portfolio management.

The PM can be reached at DSN: 574-4433 or Commercial 757-764-4433

2.2.5 Government Subject Matter Experts (SMEs). The SME assists the CO and COR in managing the overall QA program and supports customers in developing and adhering to TO-specific QASPs.

2.2.6 Functional Director (FD)/Functional Commander (FC). The FD/FC Identifies mission essential services, reviews contract performance documentation by the COR, on a regular basis to ensure performance is compatible with contract mission objectives and nominates COR(s) for contract oversight. The FD/FC receives contract specific training by the CO.

2.2.7 Contracting Officer Representative (COR). The COR is ultimately responsible for implementing all QA matters and procedures. COR(s) should be at a level where other duties don’t prevent proper oversight of Contractor performance. The COR must maintain both technical competency and evaluation proficiency in Contractor surveillance procedures. The COR must also have enough knowledge of Contractor activities to properly review and disposition monthly invoices. The COR cannot assume duties until assigned by the requiring agency (FD/FC), completed COR required training (CM led training), and approved by the CO .

2.2.5.1 Subject Matter Experts (SME). SMEs will conduct and document surveillance during each reporting period (Oct-Dec, Jan-Mar, Apr-Jun, Jul-Sep). A general summary of the surveillance will be recorded by the SME on the preprinted, Contract Monitoring and Surveillance Report (refer to Attachment 5). Local forms may be developed to document individual SME surveillance. All applicable items in the Quality Assurance Surveillance Report (refer to paragraph 3.2.2.) will be inspected during the period. The Quality Assurance Surveillance Report will be annotated to reflect the date and the items inspected, whether the inspection revealed satisfactory (i.e., acceptable) or unsatisfactory (i.e., defect) performance and any applicable comments (note: comments are required for unsatisfactory performance).

Applicable comments are intended to be a brief description of the results of the inspection.

2.3 Contractor Key Personnel.

2.3.1 Project Manager. The Contractor’s Project Manager is responsible for the overall execution of the task order. This individual will serve as the focal point for the CM/CO and/or PM assigned to the IDIQ contract and may interface with the customer, as needed.

3.0 CONTRACT SURVEILLANCE. The COR function is responsible for a wide range of surveillance requirements that effectively measure and evaluate Contractor performance. The Contractor, not the Government, is responsible for task order management and Quality Control (QC) actions to meet the terms of the contract. However, because of the non-commercial nature of the work performed by Contractors, the Government cannot rely solely on the Contractor’s quality program but must have a method to oversee the entire contracting process. The Government does this through the QASP and can inspect or test all services called for by the contract, to the extent practicable, at all times and places, during the term of the contract. The COR contribution is comprised of professional, non-adversarial relationships, which include positive and open communications with the CO, CM, SMEs, and the Contractor. The foundation of this relationship is built upon regular, objective, fair, and consistent COR evaluations of Contractor performance against contract requirements, and the ability to discuss inspection results, trends, and items of mutual interest with the Contractor. The COR uses the methods in this QASP and the PWS to achieve this goal.

3.1 Surveillance Methods. Three primary methods of surveillance are normally used: one hundred percent inspection, periodic audits and customer complaints. Other methods of surveillance must be fully documented by the COR and approved by the CO before implementation. Information obtained from these activities may be used by the FC/FD to ascertain whether or not Contractor performance is compatible with contract and mission objectives and by the CO for actions relating to the Contractor.

3.1.1 One Hundred Percent Inspection. This method will be used for Contractor submissions, products, and deliverables (tangible items). CORs will use the One Hundred Percent Inspection method to evaluate all reports delivered under paragraph 3.2. The COR, with assistance from the FD and CO if necessary, accepts or rejects these products based on the criteria and performance standards as outlined in the Services Summary. The COR also documents all results of inspections actions, and subsequent Contractor re-performance taskings, if applicable.

3.1.2 Periodic Audit. This type of surveillance consists of evaluating products/services not surveilled via one hundred percent inspection or by customer survey, and is implemented at a pre-defined frequency. An example of periodic audit is performing monthly inspections of class training conducted by the Contractor or weekly inspection of a Contractor’s process in developing training manuals.

3.1.3 Customer Survey or Customer Complaint. Although usually not a primary method, this is a valuable supplement to other, more systematic methods of surveillance. For example, in a case where random sampling indicates unsatisfactory service, customer complaints can be used as substantiating evidence. In certain situations, where customers complain consistently when the quality of performance is poor (e.g., dining facilities, custodial services), customer surveys and customer complaints may be a primary surveillance method, and customer satisfaction an appropriate performance standard. In all cases, complaints will be documented, preferably on a standard form. All customer complaints will be forwarded to the COR. Information obtained from customer complaint activities may be used by the CO for actions relating to the Contractor.

3.2 Surveillance Documents.

3.2.1 Deliverables. All deliverables will be reviewed for timeliness, accuracy and format. If a deliverable due date falls on a weekend or holiday, the Contractor shall submit the deliverable on the last work day prior to the due date.

3.2.2 Deliverables. All deliverables will be reviewed for timeliness, accuracy and format. If a deliverable due date falls on a weekend or holiday, the Contractor shall submit the deliverable on the last work day prior to the due date.

TABLE 1 – DELIVERABLES

Deliverables PWS Reference Delivered To Due Date

Courseware Delivery Schedule (CDS) PWS paragraph 1.5.4 T/SME NLT 45 days after contract award no later than 30 calendar days prior to the beginning of each fiscal year

Backup of all current courseware PWS paragraph 1.7.3 CW/SME Backup of all current courseware

Quality Control Plan PWS paragraph 1.5.2.8 CO & COR

The contractor shall submit a Quality Control Plan (QCP) concurrently to the CO with their proposal that addresses how they will provide quality training support and controls that will be used.

Academic Scheduling. PWS paragraph 1.7.4 T/SME Weekly

Conference/Meeting Minutes PWS paragraph 1.7.5 CO & COR Within 5 business days of the conference/ meeting

Strike Mitigation Plan PWS paragraph 4.9 CO & COR

The contractor shall submit a Strike Mitigation Plan to CO within ten days of contract award that will address how they will mitigate a strike.

Training Workload Management Plan PWS paragraph 1.4.2 CO & COR

The contractor shall develop and submit a plan concurrently to the CO with their proposal which outlines its management approaches, methods, innovations and policies for ensuring all required aircrews shall be trained in accordance with this PWS.

Instructional Systems Development Management Plan

PWS paragraph 1.5.2 CO & COR

The contractor shall develop, produce, and provide an ISD Management Plan to the Contracting Officer concurrently with the contractor's proposal.

Phase-In/Phase-Out Plan PWS paragraph 1.7.8 CO & COR

The contractor shall develop a Phase- In/Phase-Out-Plan. This plan shall be delivered to the CO concurrently with the contractor’s proposal.

Instructor Training and Certification Plan PWS paragraph 1.8.2.1. CO & COR The contractor shall develop, deliver, and implement an ITCP to the CO concurrently with their proposal.

Lesson Strategy Report PWS Attachment 1 CW/SME

Milestones with deadlines for the preparation and delivery of the Lesson Strategy Report shall be identified in the contractor's approved Courseware Delivery Schedule.

Objectives and Media Analysis Report PWS Attachment 2 CW/SME

Milestones with deadlines for the preparation and delivery of the OMAR shall be identified in the contractor's approved Courseware Delivery Schedule.

Course Syllabus PWS Attachment 4 T/SME

Each course syllabus is reviewed at least once every 24 months for required changes, updates and revisions.

Examination Package PWS Attachment 5 T/SME

A minimum of two examination packages, each with different questions, will be prepared for each syllabus-identified examination lesson.

Lesson Plan PWS Attachment 6 T/SME

Milestones with deadlines for the preparation and delivery of each Lesson Plan will be identified in the contractor's approved Courseware Delivery Schedule.

Lesson Presentation PWS Attachment 7 T/SME

Milestones with deadlines for the preparation and delivery of each Lesson Presentation (whether new, review or revision work) will be identified in the contractor's approved Courseware Delivery Schedule.

Lesson Study Guide PWS Attachment 8 T/SME

A Lesson Study Guide will be prepared for each presentation. This means that master file Lesson Study Guides are prepared on a work order basis in cycle with work done on presentations.

Student Handbook PWS Attachment 9 CW/SME

Milestones with deadlines for the preparation and delivery of Student Handbooks will be identified in the contractor's approved Courseware Delivery Schedule.

Instructional Media Package PWS Attachment 10 T/SME

Milestones with deadlines for the preparation and delivery of each Instructional Media Package (whether new, review or revision work) will be identified in the contractor's approved Courseware Delivery Schedule.

Training Evaluation Report PWS Attachment 11 CW/SME

Milestones with deadlines for the preparation and delivery of a Training Evaluation Report will be identified in the contractor's approved Courseware Delivery Schedule.

Contractor’s Progress, Status and Management Report

PWS Attachment 12 CO & COR A Contractor's Progress, Status and Management Report will be submitted monthly.

Instructor Guide PWS Attachment 13 CW/SME

Milestones with deadlines for the preparation and delivery of the Instructor Guide shall be identified in the contractor's approved Courseware Delivery Schedule.

Inventory of Government-Furnished Equipment for Contractor Availability

PWS Attachment 14 T/SME To be verified during Phase-In and inventoried at end of contract.

Contractor Full-Time Equivalent Reporting PWS Attachment 15 T/SME

Reporting inputs will be for the labor executed during the period of performance for each Government fiscal year (FY), which runs 1 October through 30 September.

U-2/T-38 & RQ-4 Academic Database PWS Paragraph 1.7.10 T/SME To be delivered to the Government when the Government request the Academic Database.

3.2.3. Services Summary. The Services Summary (SS) is a list of critical tasks that must be performed by the Contractor, the performance threshold, and the method of surveillance the COR, with input from the SME will use to validate/inspect these tasks. Inspection of each task will be documented in the COR folder. This checklist is to be completed by each site on a quarterly basis. The checklist is due to the COR five working days after the completion of the quarter. After each performance objective, mark “EX” for Exceeds minimum requirements, “SAT” for Satisfactory performance (meets requirements), “UNSAT” for Unsatisfactory performance, or “N/A” for not applicable. Any item marked “EX” or “UNSAT” must have written comments for justification. Specific comments on each objective are encouraged. To provide comments, specify the performance objective number in the comments section located on the second page of this attachment.

TABLE 2 – SERVICES SUMMARY PERFORMANCE MEASURE RATING (EX,

SAT, UNSAT

OR N/A)

(1) Contractor maintains qualified personnel.

Contractor effectively trains and certifies 100% of its instructors on time IAW its Instructor Training and Certification Plan and Transition Plan, and the instructors meet the standard qualifications addressed in this PWS. (Paragraph 4.4.2.1.)

(2) Contractor meets all security requirements.

Contractor ensures all employees adhere to 100% of the security requirements addressed in this PWS. (Paragraph 4.2.)

(3) Contractor responds to contingencies.

Contractor responds to requests for emergency and mission/event essential requirements in accordance with this PWS. (Paragraph 1.5.5.2.2., 1.5.5.2.3.)

(4) Contractor submits required reports.

Contractor performs delivery of reports, schedules and summaries as specified in this PWS. All other deliverables required by this PWS and the contractor’s proposal are delivered on time and are kept in the Master Files as defined in this PWS and in Appendix 4.

(5) Contractor performs U-2 and RQ-4 training device instructional events.

Contractor ensures training device events are attended by qualified instructors and conducted in accordance with this PWS. Contractor provides training to 100% of all required personnel within the timeframes specified in this PWS by accomplishing 100% of the training device sessions scheduled. Contractor exercises flexibility in device scheduling to provide maximum training opportunities. (Appendix 1)

(6) Contractor performs academic instruction.

Contractor ensures academic events are attended by qualified instructors and conducted IAW this PWS. Contractor provides training to 100% of all required personnel within the timeframes specified in this PWS by accomplishing 100% of the training sessions scheduled. Contractor exercises flexibility in class scheduling to provide maximum training opportunities. (Appendix 1)

(7) Contractor develops and delivers courseware.

Contractor develops complete courseware IAW its ISD Management Plan, which effectively conveys the objectives of training, and is delivered IAW the timelines required by this PWS and the approved Courseware Delivery Schedule and Transition Plan. (paragraph 1.5)

(8) Contractor maintains Master Files.

Contractor ensures all Master Files are maintained accurately and up-to-date IAW this PWS.

(Paragraph 1.7.2.).

(9) Contractor utilizes and maintains Academic Database.

Contractor shall utilize and maintain the current U- 2/T-38 & RQ-4 student academic training and courseware development tracking database provided at the U-2/T-38 & RQ-4. The contractor shall update and maintain the Excel database system to identify and track student progress and deficiencies, provide graduate critiques, and provide the capabilities defined in Paragraph 1.7.10.

SS PERFORMANCE OBJECTIVE # 1

CONTRACTOR MAINTAINS QUALIFIED PERSONNEL

1. Method of Surveillance: 100% Inspection.

2. Lot Size:

a. Initially, all CAT Instructor/Subject Matter Experts (I/SMEs).

b. Each new-hire subsequent to first report.

3. Performance Requirement--Acceptable/Defect:

a. Performance is acceptable when the contractor maintains sufficient qualified personnel to meet FTU and CT CAT mission requirements during the surveillance period.

b. A defect occurs whenever an academic or training device event/mission must be rescheduled due to non-availability of a qualified contract instructor.

4. Sampling Procedure:

a. All I/SMEs employed under this contract will be sampled to verify their qualifications.

b. All scheduled academic and training events that require a contract instructor will be attended by a qualified contract instructor.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Did contractor personnel meet hiring requirements? (Refer to PWS, para 4.4)

b. Did the contractor develop, deliver, and implement an Instructor Training and Certification Plan that will ensure contractor instructors meet the standards outlined in the PWS? (Refer to PWS, para 1.8.2.)

c. Did the contractor provide written notification to the TSME that an instructor is certified to instruct a lesson/training device event as specified?

(Refer to PWS, para 1.8.2.3)

d. Did the contractor conduct and schedule Continuation Training (CT) for contract instructors as required to meet minimum CT requirements for training devices? (Refer to PWS, para 1.8.1)

e. Did contractor personnel attend flight safety meetings, commander calls, pilot/mission developer meetings, and intelligence briefings? (Refer to SOW, para 1.8.1.2)

f. Did contractor personnel attend, as agreed by both the contractor and Contracting Officer, AF special schools? (Refer to PWS, para 1.8.1.3)

6. Results:

7. Remarks:

8. Date:

SS PERFORMANCE OBJECTIVE # 2

CONTRACTOR MEETS ALL SECURITY REQUIREMENTS

2. Lot Size:

a. Initially, all Contract Aircrew Training (CAT) personnel.

b. Each new-hire subsequent to the first report.

3. Performance Requirement--Acceptable/Defect:

a. Performance is acceptable when all contractor personnel adhere to security requirements.

b. A defect occurs whenever a security requirement is not met, and the contractor has not taken the required steps to fix the defect. In the case of security clearances, once the paperwork has been submitted, the contractor can not be held at fault due to delays in the government process.

4. Sampling Procedure: All contractor personnel will be monitored for compliance with requirements.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Did contractor personnel meet all security requirements as specified in the PWS? (Refer to PWS, para 4.2.2 and 4.2.4.)

b. Did contractor personnel comply with DD Form 254, Contract Security Classification Specification? (Refer to PWS, para 4.2.2)

c. Did all of the contractor personnel have the required security clearance during this period?

NOTE: If not, has the Site Manager turned in the paper work to update the clearance? (Refer to PWS, para 4.2.4 and 4.4.1.)

d. Did the contractor appropriately safeguard all government equipment and facilities provided for their use? (Refer to PWS, para 1.10)

SS PERFORMANCE OBJECTIVE # 3

CONTRACTOR RESPONDS TO CONTINGENCIES

2. Lot Size: The number of major block or system changes.

3. Performance Requirement--Acceptable/Defect:

a. Performance is acceptable when the contractor provides new or revised courseware in the proper format and within the time limits set forth in the PWS. (Refer to PWS, para 1.5.5.2.2.

1.5.5.2.3. and 1.5.5.2.4.)

b. A defect occurs whenever the contractor does not meet the requirements as specified in the PWS.

4. Sampling Procedure: All requests for new or revised courseware which the contractor was tasked to develop will be sampled.

5. Inspection Procedure

ITEM EX SAT UNSAT NA

a. Did contractor personnel provide new or revised courseware for major block or system changes in the proper format and time limits set forth in the PWS? (Refer to PWS, para 1.5.5.2.4)

b. Did contractor personnel respond to all courseware changes required by Urgent Action Directives in the proper format and time limits set forth in the PWS? (Refer to PWS, para 1.5.5.2.3)

SS PERFORMANCE OBJECTIVE # 4

CONTRACTOR SUBMITS REQUIRED REPORTS

2. Lot Size: The contractor will produce and deliver reports, schedules, and other deliverables relating to this contract. The contractor is responsible for coordinating the format and content of each deliverable with the Government CWSME IAW the approved formats. (Refer to PWS, para 1.5.2, 1.7.4, 1.5.4, 1.7.5 and Appendix 4) These deliverables are listed below.

1. Courseware Delivery Schedule; PWS para 1.5.4

2. Lesson Strategy Report; PWS Attachment 1

3. Lesson Overview; PWS Attachment 1

4. Instructional Media Package; PWS Attachment 10

5. Lesson Package; PWS para 4.2.31

6. Lesson Presentation; PWS Attachment 7

7. Instructor Guide; PWS pars 4.7.31

8. Lesson Study Guide; PWS para 4.7.31

9. Student Handbooks; PWS Attachment 9

10. Exam Package; PWS para 4.7.23

11. Training Evaluation Report; PWS Attachment 11

12. Contractor’s Progress, Status, and Management Report; PWS Attachment 12

13. Conference/Meeting Minutes; PWS para 1.7.5

14. Course Syllabus; PWS Attachment 4

3. Performance Requirement--Acceptable/Defect:

a. Performance is acceptable when the contractor produces and delivers the items specified in the PWS and in the proper format as defined in Appendix 4.

b. A defect occurs whenever a deliverable is not submitted on time or in the proper format.

4. Sampling Procedure:

a. All deliverables submitted under this contract will be sampled to verify their validity.

b. All courseware deliverables will be assigned a content and ISD grade by the SME

c. All deliverables that require contractor personnel will be written by qualified contractor personnel.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Did the contractor meet the required delivery schedules? (Refer to PWS, para 1.5.4)

b. Did all contract deliverables meet requirements for format and content? (Refer to PWS, Appendix 4)

c. Did the contractor coordinate the format and content of each deliverable with the CWSME? (Refer to PWS, Appendix 4)

SS PERFORMANCE OBJECTIVE # 5

CONTRACTOR PERFORMS TRAINING DEVICE INSTRUCTIONAL EVENTS

1. Method of Surveillance: Periodic Surveillance.

2. Lot Size: Number of academic events scheduled.

a. Performance is acceptable when the contractor performs training device instruction.

(Refer to PWS, para 1.4)

b. Defective training device instruction occurs whenever the contractor fails to show for scheduled academic instruction event or whenever instruction does not fulfill the training requirements specified in the PWS.

4. Sampling Procedure:

a. All I/SME contractor personnel will be sampled to verify their ability to perform training device instruction.

b. All scheduled academic instructional events that require a contractor instructor will be attended by a qualified contractor I/SME.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Did the contractor I/SME perform all training device events that required a contract instructor? (Refer to PWS, para 1.4)

b. Did the contractor I/SME review learning objectives with students in preparation for each training device? (Refer to PWS, para 1.4.1.2.3)

c. Did the contractor schedule and provide make-up instruction training devices that were not accomplished due to maintenance?

(Refer to PWS, para 1.4.1.2.4)

d. Did the contractor provide the maximum training opportunities for the class? (Refer to PWS, para 1.7.4)

6. Results

SS PERFORMANCE OBJECTIVE # 6

CONTRACTOR PERFORMS ACADEMIC INSTRUCTION

1. Method of Surveillance: Periodic Surveillance.

2. Lot Size: Number of academic events scheduled.

a. Performance is acceptable when the contractor performs academic instruction. (Refer to PWS, para 1.4 and Appendix 1)

b. Defective academic instruction occurs whenever the contractor fails to show for scheduled academic instruction event or whenever instruction does not fulfill the training requirements specified in the PWS.

4. Sampling Procedure:

a. All I/SME contractor personnel will be sampled to verify their ability to perform academic instruction.

b. All scheduled academic instructional events that require a contractor instructor will be attended by a qualified contractor I/SME.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Did the contractor I/SME perform all scheduled academic and training events that required a contract instructor? (Refer to PWS, para 1.4.1.1)

b. Did the contractor set-up the classrooms properly and issue any lesson-related student materials before instruction? (Refer to PWS, para 1.4.1.1.1 and 1.4.1.1.2)

c. Did the contractor I/SME review learning objectives with students in preparation for each examination? (Refer to PWS, para 1.4.1.1.9)

d. Did the contractor make I/SMEs accessible to students using self-paced/computer based instruction to answer questions? (Refer to PWS, para 1.4.1.1.4)

e. Did the contractor schedule and provide make-up instruction for students that missed class or that failed an examination? (Refer to PWS, para 1.4.1.1.5)

f. Did the contractor administer the tests properly and were the tests scored to include 100% correction? (Refer to PWS, para 1.4.1.1.10)

g. Did the contractor satisfactorily accomplish trend analysis and evaluation, and analyze all examination results for trends toward faulty questions and/or deficient instruction? (Refer to PWS, para 1.4.1.1.11)

6. Results

SS PERFORMANCE OBJECTIVE # 7

DEVELOPS AND DELIVERS COURSEWARE

2. Lot Size: The number of courseware deliverables in accordance with the Courseware Delivery Schedule. (Refer to PWS, para 1.5.)

3. Performance Requirement--Acceptable/Defect:

a. Performance is acceptable when the contractor submits the required courseware in the proper format and approved timelines. (Refer to PWS, para 1.5)

b. A defect occurs whenever courseware is not submitted on time or is not in the correct format as specified in the PWS.

4. Sampling Procedure:

a. All courseware submitted under this contract will be sampled to verify their validity.

b. The SME will assign a letter grade to all courseware at final acceptance using the standards in Attachment 8 to this Plan.

b. All courseware that requires contractor personnel will be written by qualified contractor personnel.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Did the contractor develop courseware as specified in the PWS?

(Refer to PWS, para 1.5.4)

b. Did the contractor maintain (review, revise, update, etc.) courseware as specified in the PWS? (Refer to PWS, para 1.5.5.2.1)

c. Did the contractor conduct and produce minutes for Courseware Review Meetings as specified in the PWS? (Refer to PWS, para 1.6.2)

d. Did the contractor coordinate courseware materials reproduction based upon the most current Master File? (Refer to PWS para 1.7.2)

SS PERFORMANCE OBJECTIVE # 8

MAINTAIN MASTER FILES

2. Lot Size: The number of files produced during the inspection period.

a. Performance is acceptable when the contractor maintains the required Master Files for each lesson/course accurately and up to date as specified in the PWS. (Refer to PWS, para 1.7.2)

b. Performance is acceptable when the contractor maintains the required publications and forms as specified in the PWS. (Refer to PWS, para 1.7.2)

c. A defect occurs whenever a Master File or require publications and forms are not accurate and up to date.

a. All Master Files for lessons/courses employed under this contract will be sampled to verify their accuracy.

b. All Master Files that require a correction and/or an update shall be done by contractor personnel.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Is the contractor maintaining a set of Master Files and mandatory publications and forms? (Refer to PWS, para 1.7.2)

b. Is the contractor keeping the Master Files and mandatory publications and forms accurate and up to date? (Refer to PWS, para

1.7.2 and 1.7.3)

SS PERFORMANCE OBJECTIVE # 9

DEVELOP AND MAINTAIN ACADEMIC DATABASE

2. Lot Size: The relational data created and maintained in the U-2 Academic Database.

a. Performance is acceptable when the contractor maintains the U-2 Academic Database as specified in the PWS. (Refer to PWS, para 1.7.10)

b. A defect occurs whenever data is inaccurate, inaccessible to authorized users, or unauthorized access is gained.

a. Database relational structure and all database information maintained by the contractor will be sampled to verify accuracy.

5. Inspection Procedure:

ITEM EX SAT UNSAT NA

a. Did the contractor meet or progress towards the database tasks as outlined in the PWS? (Refer to PWS, para 1.7.10)

b. Did the contractor develop new reports when requested by the government as specified in the PWS? (Refer to PWS, para 1.7.10 and Appendix 4)

c. Did the contractor maintain the database access structure as specified in the PWS? (Refer to PWS, para 1.7.10.2)

3.2.3 Surveillance Checklist (Attachment 3). The COR is responsible for developing standard procedures for Surveillance Checklists and may request SME assistance; the Surveillance Checklist serves as the guideline for all inspections. As a minimum, the Surveillance Checklist will include the items in the Services Summary and Deliverables Tables. The COR may also add other inspection activities/surveillance items to the Surveillance Checklist, with applicable PWS references, and obtain permission from the SME. These items may fluctuate throughout the course of the contract based on customer needs, identified risks, or issues related to Contractor performance. The COR will document all inspections conducted using the Surveillance Checklist and maintain them in the COR folder.

3.2.4 Contractor’s Quality Control Plan (QCP). A Contractor's internal actions, in the form of a QCP, are usually written to satisfy task order requirements for written procedures or to provide direction on how Contractor employees are to perform. Any deviations will be treated as deficiencies/non-conformances and handled IAW paragraph 3.3.2. The COR will review the QCP IAW requirements of the solicitation and subsequent awarded task order. The COR will forward any deficiencies/non-conformances to the SME and QAPC/CM/CO for appropriate action.

3.3 Assessing Performance. CORs must document Contractor performance on each contract, as well as the scope and purpose of any inspections. Documentation will be maintained in the COR’s monthly log. CORs will also assess and rate Contractor performance as it relates to contract requirements. Performance Assessment may take the form of notes in the monthly log, assessment results from the Surveillance Checklist, non-conformance notifications, quarterly Performance Assessment Reports (PAR) and annual Contract Performance Assessment Reports (CPAR). Each builds on the other and provides a thorough documented history of Contractor performance.

3.3.1 Inspections. The COR may perform inspection/surveillance at any time during contract performance IAW FAR 252.246-2 through 5, as applicable. The COR will retain all audit and inspection documentation until contract closeout, unless otherwise approved by the CO.

3.3.2 Non-conformances and Deficiencies. Audit or Inspection results that fall below contractual standards shall be identified as non-conformances or deficiencies. The COR will notify the contactor IAW para 3.3.2.2 and document all non-conformances in the monthly log, referencing the specific PWS/TO requirement that the Contractor did not meet, document the name of the individual contacted, and date and time of notification. If the Contractor fails to take corrective actions in a reasonable time-frame, the COR will notify the QAPC/CM/CO and SME for further actions.

3.3.2.1 Types of Non-conformance.

3.3.2.1.1 Minor Non-conformance: A minor non-conformance is a non-conformance, which by itself does not adversely impact mission, safety of personnel and/or equipment, performance (quality), schedule (delivery), or cost. A minor non-conformance normally does not increase risk to the Government.

3.3.2.1.2 Major Non-conformance: A major non-conformance is a non-conformance that adversely impacts mission, safety of personnel and/or equipment, performance (quality), schedule (delivery), or cost. A major non-conformance also increases risk to the Government.

An example of increasing risk would be significant number of recurring non-conformances, which indicates inadequate preventive measures or actions, thus lowering the Government’s confidence that quality services will be provided on time and at cost.

3.3.2.2 Non-conformance Notification. If a deficiency is found during inspection of the Contractor’s performance, the COR will determine the type of notification based on the category of the non-conformance (see Table 3) and pursue non-conformance resolution per paragraph 3.3.2.2.2.1.

3.3.2.2.1 First or Second Notices (see Attachment 1). As shown in Table 3, First and Second Notices provide written notification from the Government to the Contractor of minor non-conformances. Initial minor non-conformances should be addressed with a First Notice to the applicable Contractor representative. The COR will notify the CM/CO, QAPC, and the SME of the issue, and prepare the notice for CO signature. The COR will provide a final, signed copy to the CM/CO, QAPC and SME. The CO will issue First and Second notices to the contractor with a suspense date for the contractors reply. If the Contractor fails to respond by the specified due date, the COR will notify the CO/QAPC/SME for action and annotate the Contractor’s non-responsiveness in the quarterly PAR. If the Contractor fails to resolve the issue as documented in the Notice, the COR should elevate the issue to the next level (Second Notice or CAR).

Corrective/preventive actions initiated and/or completed by the Contractor will be verified by CORs on the scheduled inspection and documented accordingly.

3.3.2.2.2 Corrective Action Requests (CAR) (see Attachment 2). As shown in Table 3, a CAR provides written notification from the Government to the Contractor of significant performance discrepancies. The Primary COR will recommend to the CM/CO, QAPC, and SME that a CAR be issued when previous notifications failed to result in corrective actions or when severe deficiencies exist. If a CAR is deemed necessary, the QAPC will assist the primary COR in filling out the CAR. The draft CAR will be routed NLT ten duty days after the event to the CO for review and signature, as the CO is the only person who has authority to issue a CAR to the Contractor.

3.3.2.2.2.1 To satisfactorily close-out a CAR, the following criteria must be met: the Contractor must meet the suspense; the corrective action must have already begun; and the Contractor's actions must correct the deficiency and if applicable, be within specified standards.

3.3.2.2.2.2 If the corrective action is deemed UNSATISFACTORY by the FD and COR, annotate the CAR accordingly and forward it to the CO with a brief explanation, including an

TABLE 3 – NON-CONFORMANCE NOTIFICATIONS.

Non-conformance Category Notification Type Minor

Step 1: First Notice Step 2: Second Notice

Major Corrective Action Request (CAR) estimate of the Government's loss due to the deficient performance. The estimate may account for loss or destruction of property, supplies, or equipment, and hours lost to re-perform the task.

If the task cannot be re-performed, indicate the number of hours it ordinarily would have taken to do the task.

3.3.2.2.3 The COR will maintain a non-conformance log documenting all First and Second Notices and CARs. The log will include the date sent to the Contractor, issue, due date for response and actual response date. If a detailed response is required, include a link to the response and include the response in the applicable COR folder tab.

3.3.3 Quarterly Performance Report . The chief site SME will complete the quarterly report using the services summary Para 3.2.2, documenting Contractor performance for the preceding quarterly performance period (Jan – Mar, Apr. – Jun, Jul – Sep, and Oct. – Dec); Quarterly reports will be completed by the chief SME NLT the 15th Calendar day of each quarter (15 Apr, 15 Jul, 15 Oct, and 15 Jan)

3.3.4 Monthly COR Report

3.3.4.1 The Primary COR certifies the essential services accomplished by the contractor and documenting any issues or non-conformances that were identified during the month.

3.3.4.2 CMS (https://cms.acc.af.mil) will generate monthly emails to the primary COR(s) designated on the contract, monthly reports will be completed NLT the 10th calendar day of each month (or day requested by CO).

3.3.4.3 A primary COR will click on the embedded hyperlink in the email or Log into CMS, click on my requirements, then click on appropriate contract; click on QA Monthly Rpt

3.3.4.4 The monthly report will display questions for the previous month (ex. answering questions in the month of March will display a monthly report for February)

3.3.4.4.1 Click on Answer Questions

3.3.4.4.2 Click on meets, above, below standards or N/A (see Table 1) and provide justification for each answer.

3.3.4.4.3 After completing the Monthly COR Report, click submit.

3.3.4.4.4 CMS will only allow one submitted report per month.

TABLE 4 – MONTHLY PERFORMANCE RATING DEFINITIONS

Rating Definition

Exceeds Standards

Contractor performance exceeds contractual requirements and is beneficial to the government. To justify exceeds standards, state which PWS or SOW requirement(s) was exceeded (to include the paragraph number) along with how exceeding this requirement was a benefit to the government.

Meets Contractor performance meets contractual requirements. To justify meets standards, Standards specify the PWS paragraph/requirement and state that the contractor met the requirement.

Below Standards

Contractor performance does NOT meet some contractual requirements. To justify below standard, identify an event(s) that the Contractor had trouble overcoming and state how it impacted the Government. This rating should be supported by referencing any non-conformance notification sent to the Contractor.

Under Corrective Action, state what corrective action the contractor has taken or is planning to take based on the contractor’s response to the non-conformance notification.

N/A Contractor was not contractually required to perform specific task or Specific question is not applicable for evaluated period.

3.3.5 Annual Contract Performance Assessment Reporting System (CPARS) – The Contractor Performance Assessment Reporting System (CPARS) https://www.cpars.gov/ ) is the Department of Defense (DoD) Enterprise Solution for collection of Contractor Past Performance Information (PPI) as required by the Federal Acquisition Regulation (FAR).

CPARS is a web-enabled application that collects and manages a library of automated Contractor report cards. An annual CPAR will be written on all contracts that exceed $1M total contract value. The COR (or PM) will be designated as the Assessing Official Representative (AOR) in CPARS; the written assessment in CPARS will begin annually from the start date of the contract and forwarded to the Contract Manager within 60 days of the contract start date anniversary. The AOR will use objective evidence from previous quarterly PARS, monthly Services Summary, inspection logs, 1st/2nd notices, CARS, and correspondence letters for CPARs assessment.

The table below is an excerpt from the AMIC CPARS Roles and Responsibilities Policy Letter.

Performance Assessment Report Routine Timelines* CPAR Role: Timeframe Action in CPARS AOR +20 calendar days following the end of the Period of Performance (PoP)

Draft (Narrative and rating) report and send to AO

AO +40 calendar days following the end of the PoP

Validate narrative/ratingcontract information and send to Contractor Representative

Contractor Representative 60 calendar days Comment and return to AO AO (Contractor Concur or does not sign)

7 calendar days but not to exceed PoP end +107 days

Completes CPAR

AO (Contractor Non-Concur) RO (Contractor Non-Concur

14 calendar days but not to exceed PoP end +114 days

AO forwards to RO; RO reviews, updates and completes CPARS

*Focal Point will notify AOR, AO, or RO supervisor or division chief for all exceeded time frames

4.0 WIDE AREA WORK FLOW (WAWF). The COR will accept or reject all Receiving Reports submitted by the Contactor in WAWF within 7 days of submittal by the Contractor.

To access the receiving report, log into WAWF, select the Government, Acceptor, Acceptance Folder tab; then type in the contract and delivery order (task order) number. The Shipment Number column in WAWF is the Receiving Report number

4.1 Validating Receiving Reports. The COR will review and validate all WAWF receiving reports (Shipment Number in WAWF) ensuring that:

The Contractor submitted a combination invoice/receiving report The report reflects the correct contract and TO number Beginning and ending dates being billed are stated and accurate The correct CLIN(s) is(are) charged for billing period A description of the services is provided (Firm Fixed Price, Labor Hour, Travel, Materials) The correct amount is charged, to include applicable handling rates Invoiced travel was pre-approved Back-up documentation is attached that includes a Contractor invoice with current and cumulative hours (for FFP LOE contracts) and amounts, as applicable A breakout of any other charges, as agreed to by the CO or required by the TO, is attached in WAWF or has been provided to the COR

4.2 Certifying and Accepting Receiving Reports.

4.2.1 Certification of services is ALWAYS performed by the appointed COR. After validation of services is performed by the COR per paragraph 4.1 above, the COR will select the “CQA” (Contract Quality Assurance) button to verify that the invoice was accurate, this step will normally select the Acceptance: button also. If anyone other than the COR (typically the Resource Advisor (RA)) is accepting services for the task order, the COR MUST certify in writing to the acceptor that services have been inspected/validated, the acceptor must then state in the comments block (under Misc Info tab) that the COR has reviewed, validated and certified services/invoices. In all cases, the COR will retain documentation (electronic or hard copy) showing COR validation and certification, as well as final acceptance of Contractor invoices.

4.2.2 COR Expenditure Log. After certification of an invoice in WAWF, the Primary COR will update the expenditure log documenting/tracking certified invoice expenditures against funds allocated on the contract against each CLIN. The SME will provide a template, if requested. Funding belongs to the organization where the COR resides; the COR expenditure log allows the COR to ensure that funds are available for the work performed, to include travel and materials.

4.3 Rejecting Receiving Reports. If a receiving report is rejected in WAWF, the COR will go to the Miscellaneous Info tab to complete the comments block as follows:

List details regarding all unsatisfactory services or incorrect invoice information

AND

Instruct the Contractor to contact DFAS to have the invoice rejected back to the Contractor; this will allow the Contractor to make the same correction on both the receiving report and invoice, since combination invoices/receiving reports are required

4.4 The rejecter will then return to the Header tab and click the “Reject to Initiator” block.

Anyone other than the COR may reject receiving reports in WAWF, but only when services/invoices have been reviewed and rejected by the COR in writing per the paragraph above. After rejecting the receiving report, e-mail the CM/CO the contract number, receiving report number, and reason for rejection.

5.0 COR FOLDER. The Primary COR will establish and maintain a COR folder for each task order. The COR folder must contain the tabs and contents outlined below in Table 6. ALL surveillance records are considered FOR OFFICIAL USE ONLY documents, and must be marked and safeguarded as such. Documentation is kept for the life of the contract and provides necessary continuity should COR duties transition during the course of the contract. The FC/FD and COR will receive instructions from the CO on disposition or archival procedures upon contract completion.

TABLE 4 – COR FOLDER TABS & CONTENTS

Tab Title Items to Include

1 General Correspondence 2 COR/SME Appointment Letter(s) Assignment/nomination letter(s), appointment letter(s) and Government/Contractor relationships guidance 3 COR/SME Relief Letter(s) 4 COR Training Training certificates for SME-Led and CO-Led training 5 SME Training 6 Contract/Task Order, Modifications, PWS, Pricing Contract for current level of effort, any modifications and most current PWS/amendment on contract, and Contractor’s contract/task order pricing

7 Quality Assurance Surveillance Plan, Surveillance Checklist

Contract/task order QASP, approved surveillance checklist, and completed surveillance checklists

8 Contractor’s Quality Control Plan If applicable 9 Contractor’s Instructional Systems

Development Management Plan

10 Contractor’s Training and Certification Plan

11 First & Second Notices, Corrective Action Reports (CAR)

Non-conformance log, First and Second Notices and CARs issued to Contractor

12 Contractor’s Monthly Progress, Status, and Management Report

13 SME Quarterly Performance Reports 14 WAWF Invoices & Attachments If electronic, must be available upon request and for the life of the contract 15 COR Expenditure Log Updated with latest invoice and funding 16 DD Form 254 If applicable 17 Government Style Guide PWS Attachments

Attachment 1

First or Second Notice Template

(Date) MEMORANDUM FOR (Contractor, ATTN: Contractor POC)

FROM: (CO’s office symbol)

SUBJECT: First/Second Notice for Contract for (brief name for non-conformance) on

FA4890-11-D-000X)

1. Contracting Officer Representative: (Mr. John Doe)

2. Office: (COR Office symbol)

3. Date of First Notice: (if 2nd notice)

4. Date response due from Contractor: (dd-mmm-yyyy)

5. Requirement Reference: (PWS section and paragraph; Deliverable, Services Summary (SS))

6. Performance Requirement: (State what action the Contractor should have performed IAW the reference above)

7. Description of Non-Conformity: (provide objective evidence of the Contractor’s…