ATTACHMENT_19___N2ITSM_QASP.docx

Attachment 3, Certificate of Service (COS)

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

For

NORAD and USNORTHCOM (N-NC) Information Technology Service Management (N2ITSM) at

PETERSON AFB

Solicitation #: FA2517-15-R-7001

CRB #: 15-0006

The following members of the MFT have approved implementation of this Plan on behalf of the entire MFT:

21 CONS/LGCCA

Contracting Officer

N-NC/J6

Acquisition Program Manager

CHANGE CONTROL SUMMARY

Change. A change that is administrative in nature does not require the plan to be re-staffed. Changes that remove or add items periodically assessed or affect items assessed in frequency of assessment or thresholds must be re-staffed for signatures. Changes that affect roles of MFT members must also be re-staffed. Consolidate changes when possible, and identify as Change 1, Change 2, etc. Include a summary of the changes incorporated in the change table. Ensure change bars have been inserted.

Revision. A revision should be used to consolidate all changes into a new basic document with a new basic date. It should be used to clean up the document when it begins to become confusing with too many changes. Number the revision and include a summary of the revisions in the change table. Remove all change bars when the document is revised since the revised document takes all previous changes into account. Do not remove changes from the change table when the plan is revised since they may lend important historical value to the plan.

1. OVERVIEW

1.1. The Quality Assessment Surveillance Plan (QASP) provides the Contracting Officer’s Representative (COR) an effective and systematic surveillance method while defining the roles and responsibilities of the Multi-Functional Team (MFT) using a team concept.

1.2. Contracted Services. Contractor provides non-personal services to: manage, operate, sustain, and provide logistical and general support for the IT Services, as identified in the NORAD and USNORTHCOM IT Service Catalog, Appendix D, and service assets listed in the Hardware and Software Service Asset catalogs (Technical Library).

2. ROLES AND RESPONSIBILITIES

2.1. Multi-Functional Team (MFT). The goal of the MFT is to give all members/stakeholders the ability to maintain the highest quality representative services to our customers and propose/initiate improvements through the life of the contract. Under this concept continual contract assessment will be focused on ensuring the contract, as written, continues to meet the needs of the customer.

2.1.1. The MFT should strive to achieve efficiencies in the contract performance and improved customer support. The team is composed of stakeholders in the acquisition process. The goal of the team is to define the requirement in performance based terms, assure Contractor compliance with contract requirements and ensure the contract as written continues to meet the needs of the functional organization. In addition, another goal is implementation and execution of the acquisition strategy to ensure support of the mission.

2.1.2. Responsible for effective contract assessment and management. Core MFT members for this contract are the Program Manager (PM), Contracting Officer (CO), Contracting Officer Representatives (CORs), and the Contractor that provides the service.

2.1.2.1. General Duties and Responsibilities

2.1. Contracting Officer (CO)

2.1.1. Ensures requirements of the Federal Acquisition Regulations (FAR) and its supplements are met. Ensures compliance with the terms of the contract and safeguards the interests of the Government to include cost and property control. Ensures Contractors receive impartial, fair, and equitable treatment.

2.1.2. The CO is the only person with authority to direct the Contractor, make interpretations of and changes to the contract, and resolve disputes or conflicts with the Contractor.

2.1.3. Designates authority for inspection and/or acceptance of service to the CORs in writing. Issues termination of COR duties to assigned CORs.

2.1.4. Informs the Contractor of the names, duties, and limitations of authority for all personnel assigned to the contract.

2.1.5. Conducts contract-specific training for Primary and Alternate CORs.

2.1.6. Participates in the preparation and submission of annual execution reporting.

2.1.7. Serves as the business advisor to the MFT. Reviews all Performance Assessment Reports (PARs) prior to issuance; serves as an issuant authority for PARs.

2.1.8. Negotiates all requirement changes and Contractor change proposals.

2.1.9. Active contributor to the CPARS process.

2.2. COR Roles and Responsibilities

2.2.1. Reviews Contractor’s Monthly Summary Report (MSR) to assess performance.

2.2.2. Reviews and/or accepts Contractor-submitted invoices in Wide Area Work Flow (WAWF) system to ensure the Contractors are billing against the correct Contract Line Item Number (CLIN), and to ensure the correct hours for labor are billed against the contract.

2.2.2.1. COR must accept invoices submitted in WAWF each month for Firm Fixed-Price (FFP) contracts. The Primary CORs will establish and maintain a WAWF account in order to review and accept invoices.

2.2.2. Prepares the Monthly Performance Report (MPR) or Certificate of Service (COS) and inputs into the CORT tool.

2.2.3. Assists in development of annual CPARS evaluation and report.

2.2.4. Evaluates software maintenance status to ensure functional capability is maintained.

2.2.5. Monitors assigned subtasks to ensure they remain within cost, schedule, and performance, and reports to CO with any discrepancies identified.

3. PERFORMANCE ASSESSMENT

3.1. Strategy. To rely to the maximum extent on the Contractor’s Quality efforts to ensure requirements are met, CORs will assess the Contractor’s performance in a manner to sufficiently validate compliance with all contractual requirements.

3.2.1. Collecting Customer Feedback. CORs will solicit customer feedback regarding functionality and reliability of the contractor to manage, operate, sustain, and provide logistical and general support for the IT Services, as identified in the, NORAD and USNORTHCOM IT PWS, Service Catalog, Appendix D, and service assets listed in the Hardware and Software Service Asset catalogs (Technical Library).

3.2.2. Use of Metrics. Data should be used to the maximum extent to document and report Contractor success or failure in meeting customer requirements. If Contractor generated metrics are used, the COR must sufficiently validate the accuracy of the metrics to establish confidence in their use.

3.3. Performance Objectives (PO). The POs listed Table 1 identify the minimum performance assessment requirements that must be periodically accomplished.

3.3.1. The Government retains the right to assess all contract requirements to the extent practicable, but will make every attempt not to delay the Contractor's workflow. All assessments of the Contractor must be documented and reported.

Table 1 Performance Objectives

3.4. Performance Assessment Report (PAR). The PAR (see Attachment 2) is used to document Contractor assessments. PAR Section I is used to document assessment results. PAR Section II is used to document nonconformities (findings) identified during the assessment documented in Section I and forwarded to the Contractor for response. When a PAR Section II is used to document nonconformity, it is known as a PAR with Finding.

3.5. Types of Nonconformities. There are two types of Nonconformities, Contractor identified or COR identified.

3.5.1. Contractor Identified Nonconformities. CORs will review Contractor identified nonconformities to ensure corrective measures were successfully implemented, with all necessary actions taken to prevent recurrence. If documentation exists which substantiates the Contractor has identified the deficiency and timely corrective action is being taken to correct the deficiency, the deficiency may be excused from PAR with Finding. If the nonconformity has not yet been resolved but the Contractor has established a timely estimated completion date (ECD) for correction and resolution, CORs will monitor the Contractor’s resolution efforts until the nonconformity has been corrected.

3.5.1.1. A PAR with Finding for nonconformities when the Contractor has already self-identified may be considered if: the deficiency is identified as a critical item under the Service Summary; this is a repeat identification; Contractor fails to follow through with the agreed way ahead or timeframes; Contractor fails to follow their Quality Control Plan; or issue involves safety, environmental, or security issues.

3.5.2. COR Identified Nonconformities. Nonconformities should be documented as early as possible after discovery and CO concurrence on a PAR with Finding using PAR Section II. It is then issued to the Contractor with instructions on when and how to reply. When the Contractor’s reply is received and deemed to be acceptable, the COR will then close and file the PAR.

3.5.2.1. Notify the CO, MFT, and the Contractor of a suspected nonconformity.

3.5.2.2. Investigate to determine if the Contractor has self-identified the nonconformity within their quality system.

3.5.2.3. CO issues PAR.

3.6. Performance Assessment File. The Primary COR shall establish and maintain a Performance Assessment File (PAF). Multiple CORs on the same contract should share common documents for increased efficiency. The PAF should be electronic and paperless to the maximum extent such as using a shared database file (DoD CORT Tool). These files shall be passed on to the subsequent COR appointed to the contract and shall be retained as part of the official contract file. The file must contain:

Section 1 – Contract Documents. Includes the PWS, Attachments and Incentive/Award Fee/Term Plans.

Section 2 – Quality Documents. Includes this QASP and the Contractor’s Quality Control/Program Plan. CORs must be well versed in both of these documents.

Section 3 – Continuity Log. A record of actions taken in the performance of COR duties, providing a brief synopsis of contract inspection activities, meetings, and conversations with the Contractor regarding contract performance. This Log should be used to provide continuity between CORs and as documentation when needed. Items contained in this Log are at the discretion of the CORs but efforts should be made to ensure it is utilized when necessary.

Section 4 – Appointment and Designation Letters/Training Certificates. Each COR’s designation letter from the CO, nomination letter from the supervisor, and Phase I and Phase II training certificates.

Section 5 – Records. Includes PARs for contract requirements assessed as well as any other pertinent records that should be maintained.

Section 6 – Other Items (Optional). Optional use for any other records or data pertinent to this contract.

3.7. Performance Assessment Types. Performance Assessment is accomplished in the following ways:

3.7.1. COR Assessment. CORs assess and document the POs outlined in Table 1 and assess and document other requirements within their functional areas as needed. They should use data and nonconformities discovered by the COR to trace back the effectiveness of the Contractor’s quality system. All assessment results are documented in the monthly COS and efforts should be made to assess all PWS requirements as possible.

3.7.2. Contractor Self-Assessments. The Contractor assesses their own performance IAW their quality system requirements and maintains documentation of the results IAW the PWS and their Quality Plan. This internal assessment is the vehicle whereby the Contractor may generate Contractor identified nonconformities and devise corrections and subsequent actions to prevent recurrence.

4. Managing Contract Performance

4.1. Approach. To manage the contract to ensure the Contractor is meeting cost, schedule, and performance requirements, the MFT utilizes performance assessment outcomes as listed in Table 1 and/or other applicable indicators to continually implement improvements wherever possible. Primary methods used to manage contract performance include Customer Feedback for each software version release and achievement of intermediate milestones for Certification and Accreditation.

5. Reporting Performance Results

5.1. Certificate of Service (COS). CORs report all performance assessment through a Monthly Performance Report (MPR) or Certificate of Service (COS) and inputs into the CORT tool. Performance data reported in the COS provides feedback to the Contractor. Accuracy and completeness of each COS is critical in further reporting.

5.2. Contractor Performance Assessment Report (CPAR). The CPAR is an annual report of Contractor performance. It is based on the COSs and is an annual requirement. CPARs are used to provide past performance information to source selection teams for consideration in other acquisitions; therefore, accuracy, completeness, and ratings within the CPAR are critical.

6. Assessment Records

6.1. All assessment records are maintained in the PAF, other designated locations, or the contract file for the life of the contract, or as specified by the records disposition schedule. The CO may specify a periodic transfer of these records to the contracting office or at the conclusion of the contract for contract file inclusion. COR must contact the CO for the final disposition instructions upon contract completion.

FA2517-15-R-7001

ATTACHMENT 19 – N2ITSM QASP

A1.1. PAR with Finding Generation Process (as it applies to the COR) A1.1.1. COR discovers nonconformity with services. COR researches to confirm deficiency and discusses with Contractor.

A1.1.1.1. Investigates to determine if the Contractor has self-identified the nonconformity within their quality system and is taking action to correct the nonconformity and implement preventive actions. If so, the CO may decide not to issue a PAR with Finding and monitor the Contractor’s corrective and preventive actions.

A1.1.1.2. COR drafts a PAR with Finding and sends to the CO for review and discussion.

A1.1.2. CO returns with instructions on how to proceed. CO is final determination authority for issuance of the PAR.

A1.1.2.1. If PAR is justified, Finding category and Rating determined.

A1.1.2.2. Proceed with issue to Contractor upon CO concurrence.

A1.1.3. SCOR (with written issuing authority) will forward the PAR with Finding to the Contractor's Site Program Manager and others as required and copy the CO.

A1.1.3. Contractor replies by the suspense date on the PAR with Finding.

A1.1.4. COR reviews response and determines if preventative actions are acceptable, and coordinates Contractor response with the CO prior to accepting or rejecting Contractor response.

A1.1.5. If acceptable, mark blocks 17-20 to close the PAR with Finding. Send closure reply to the Contractor.

A1.1.6. IAW FAR 52.246-4, Inspection of Services (Aug 1996), if the response is unacceptable, the Contractor is notified and further recourse is taken as appropriate.

A1.1.7. PAR with Finding remains open until resolved.

Attachment 1, PAR Generation Process

PERFORMANCE ASSESSMENT REPORT (PAR)

For Check Blocks, enable and lock Forms Toolbar

Section I. Performance Assessment

|_| No Finding |_| Minor Nonconformity |_| Major Nonconformity

7. PERFORMANCE ASSESSMENT—DESCRIBE CONTRACTOR PERFORMANCE

Section II. Nonconformities Identified

13. BRIEF DESCRIPTION OF THE FINDING (Include PWS reference narrative)

14. FINDING IMPACT

15. INSTRUCTIONS TO THE CONTRACTOR: If a suspense date is entered in block 12, respond with a corrective action plan that details the corrective action of the cited deficiency, the cause of the deficiency, and actions taken to prevent recurrence. If date was not entered in Block 12, the Contractor is not required to provide a response.

16. COR ASSESSMENT TO CONTRACTOR’S CORRECTIVE ACTION PLAN

|_| Accepted |_| Rejected

Continuation Sheet

INSTRUCTIONS. The PAR is used to document contract performance assessments, to report nonconformities to the Contractor and to request correction actions taken by the Contractor to prevent recurrence. Instructions are included for blocks not self-explanatory.

SECTION I - PERFORMANCE ASSESSMENT. Use document assessments of Contractor performance with PWS requirements. At a minimum, document assessments of Performance Objectives (PO) and Assessment Items (AI) from the Performance Plan.

Block 4, Finding Category: Check one block only.

No Finding: Performance meets PWS requirements.

Major Finding: A conclusion that identifies a condition that has a significant adverse effect on the quality of the activity under review such as:

1. Failure to meet a Performance Threshold.

2. Significant mission degradation in any Contractor operated function.

3. A situation that is likely to result in hazardous or unsafe conditions for individuals (or any other government resource) coming into contact with the work.

4. Failure to provide adequate corrective action to prevent reoccurrence of government identified findings.

5. Failure of the Contractor to provide corrective action to self-identified nonconformities within a reasonable time frame dependent on deficiency complexity.

6. Any failure to adhere to security regulations resulting in a security incident.

Minor Finding: A departure from established standards having little bearing on service provided. If the same minor finding is repeatedly identified, it may be an indication that a Major Finding has occurred because the Contractor’s quality system has not acted to prevent recurrence. In this case, the Minor Finding may be elevated to the level of a Major Finding and a new PAR initiated.

Block 5, Contract Requirement: Identify contract requirement, i.e., Civil Engineering.

Block 7, Performance Assessment: Enter the particular details and results of the assessment. Detailed results of assessments are required to report and manage contract performance. Avoid Pass/Fail descriptions of contract performance when possible.

SECTION II - NONCONFORMITIES IDENTIFIED. Used to document nonconformities identified during an assessment. Section II is used to notify the Contractor of the finding and request corrective action.

Block 10, Control Number: Assigned by the issuing authority. Track all findings with a Control Number containing the first two or three letters of the name of the site or contract - the last two digits of the fiscal year - and a three digit number starting with 001 and progressing upward through the fiscal year. For example, the 1st PAR finding for Peterson for FY10 is: PE-10-001. When the issues a PAR with Findings, the will use the same format and add a “Q” designator after the last two digits of the fiscal year. For example, PE-10Q-001.

Block 11, Issuing Authority/Name/Grade: Enter the Grade/Name. The issuing authorities for a PAR with Finding are the Contract Functional Authority and the CO. The Contract Functional Authority may delegate this responsibility to the SCOR. Document in the meeting minutes the CO agreement/concurrence with the issuance of a PAR with Finding.

Block 12, Response Suspense Date: Enter the date the response is due from the Contractor for the cited nonconformity. Normally, this is two calendar weeks unless extenuating circumstances exist. The response suspense date is for initial response from the Contractor and not necessarily the complete corrective action plan.

Block 13, Brief Description of the Finding (Include PWS reference): Clearly and briefly, state the details of the finding including the PWS reference and quote. For example:

PWS para 3.4 reads: “The Contractor shall maintain an MSDS in the work area for all hazardous materials used in the work center.” The Contractor did not have the MSDS for “Simple Green” in the building 100 work area.

Block 14, Finding Impact: Enter the impact the finding has, or could reasonably have, on mission accomplishment.

Block 16, COR Assessment of Contractor’s Corrective Action Plan: Indicate if the Contractor has corrected the nonconformity and if the preventive actions appear to be sufficient to prevent recurrence.

PAR Continuation Sheet. The PAR continuation sheet may be used to expand upon information in Section I or Section II. The continuation sheet may also be used to document recurring assessments during the contract year. For example, if the performance plan requires the COR to assess a certain PWS requirement every 30 days, a table of those checks could be developed and inserted into the continuation sheet. This allows the COR to have only one PAR per year generated for each recurring assessment requirement. If a deficiency is identified, complete a separate PAR for that particular finding. For example:

PWS Reference

Attachment 2, PAR

Attachment 3

FROM: N-NC/J6

SUBJECT: NORAD and USNORTHCOM (N-NC) Information Technology Service Management (N2ITSM) Solicitation #: FA2517-15-R-7001

SECTION I: SERVICE SUMMARY (SS) ITEMS:

Performance Objectives

(SS-1) Service Operation and Stability

(SS-2) Service Transition and Support

(SS-3) Service Optimization - Provide timely technical support, accurate technical and analytical advice, guidance and products as required by individual Subtasks.

(SS-4) Continuous Service Improvement - Meet technical qualifications and certifications in accordance with DoDI 8570IAM and IAT Levels to meet contractual requirements in PWS.

SECTION III: CPARS PERFORMANCE INPUTS

1. STRENGTHS & WEAKNESSES: For the CPARS rating areas below, provide an explanation of any performance that was exceeded with the specific benefit to the Government with applicable PWS paragraph/s identified. For any areas in which task order performance was not met, please identify the impact of the substandard performance and applicable PWS paragraph/s not met.

NOTE: Please do not leave any of the below CPARS categories blank. The COR is required to provide an evaluation of each area. If there was no over and above performance noted and performance was satisfactory, please indicate such. SECURITY is not rated for CPARS purposes; however, it is evaluated for compliance purposes.

Technical Performance:

EX

VG

SAT

MAR

UNSAT

Schedule:

EX

VG

SAT

MAR

UNSAT

Cost Management:

EX

VG

SAT

MAR

UNSAT

Business Relations:

EX

VG

SAT

MAR

UNSAT

Management of Key Personnel:

EX

VG

SAT

MAR

UNSAT

Regulatory Compliance:

Pass

Fail

Security:

2. ADDITIONAL COMMENTS:

SECTION IV: PERFORMANCE ASSSESSMENT REPORT (PAR) SUMMARY: (if any)

Opened Date Closed

SECTION V: COR LIST:

Rank/Name/ Office Symbol Position/ Function Phone DSN prefix:

(XXX)

Comp Date

Primary

Alternate

I certify that the services required by this task order have been performed and that any non-performance of task order requirements has been identified in Section II under the appropriate category. Further, I have reviewed the Contractor’s monthly cost reports and have found them to be acceptable. Any discrepancies have been noted in Section II under Cost Management.

//SIGNED//

image1.jpeg