ePS_RFP_Attach_11_Quality_Assurance_Surveillance_Plan_signed.pdf

N00039-17-R-0002 electronic Procurement System

Request for Proposal

Attachment 11: Quality Assurance Surveillance Plan

ACAT 1

Dated

6 March 2017

Distribution Notice D: Authorized to the Department of Defense (DoD) and DoD Contractors only.

This document is for administrative or operational use, as of the date of approval for this document.

Refer to the Program Manager (PM), PMW 220 Navy Enterprise Business Solutions (Navy EBS), Program Executive Office Enterprise Information System (PEO EIS), Washington Navy Yard, 1325 th

St. SE, Bldg. 196, Washington, D.C. 20374 for additional document requests.

Handling and Destruction Notice: Comply with distribution statement and destroy by any method that will prevent disclosure of contents or reconstruction of the document.

Quality Assurance Surveillance Plan

(QASP)

File name:

ePS_RFP_Attach_11_Quality_Assurance_

Surveillance_Plan.docx

Page:

2/29

Version:

Version 1.0

Date of Creation

6 Mar 2017

Release Authorization

Submitted by:

CDR R. M. Garrison Date ePS Deputy Assistant Program Manager

Program Management – Warfare (PMW) - 220 Navy

Enterprise Business Systems (Navy eBS)

Approved by:

S. E. SEAY Date

Program Manager

PMW 220

Surveillance_Plan.docx

Page:

3/29

Version:

Version 1.0

Document History

VERSION DATE

CHANGE

AUTHORIZATION

RESPONSIBLE

INDIVIDUAL

1.0 6 March 2017 Initial Document Terry Coley

Surveillance_Plan.docx

Page:

4/29

Version:

Version 1.0

Table of Contents

1 __ Purpose

2 __ Authority

3 __ Scope

3.1 Contract Scope

3.2 Contract Quality Assurance

3.3 Government Quality Assurance Surveillance

4 __ Roles and Responsibilities

4.1 Assistant Program Manager

4.2 Procuring Contracting Officer

4.3 Contracting Officer Representative

5 __ Methods of Quality Assurance

5.1 Surveillance Folder

5.2 Evaluation Areas of Contractor Performance Assessment Report

5.3 Contract Manpower Reporting Application

5.4 Performance Rating Definitions

5.5 Performance Standards

5.6 Timely Feedback to Contractor

5.7 Documents / Reporting

5.8 Government Surveillance Metrics

5.9 Remedies for Nonconformance and Monitoring Forms

5.10 Contract Performance Assessment Reporting System

5.11 Disputes

List of Tables

Table 5-1: Evaluation Rating Definitions Table 5-2: System Attributes and System Performance Measures Table 5-3: Surveillance Matrix (ePS Focus Area)

Table 5-4: Surveillance Matrix (Surveyed Product)

Table of Appendixes

Appendix A: Performance Assessment Report Appendix B: Corrective Action Report Appendix C: Customer Complaint Record

Surveillance_Plan.docx

Page:

5/29

Version:

Version 1.0

1 PURPOSE

This Quality Assurance Surveillance Plan (QASP) is a Government developed and applied document that describes the systematic quality assurance methods and guidelines used by the Procuring

Contracting Officer (PCO) and the Contracting Officer Representative (COR) to measure and monitor Contractor performance.

Specifically, this QASP measures the Contractor’s efforts toward meeting the requirements of the

Performance Work Statement (PWS) and the contractual requirements for the Department of the

Navy (DON) Electronic Procurement System (ePS).

This QASP is designed to define roles and responsibilities, identify the performance objectives, define the methodologies used to monitor and evaluate the Contractor’s performance, describe quality assurance reporting, and describe the analysis of quality assurance monitoring results. The QASP describes the procedures to ensure that the Contractor provides quality products in response to requirements and performs in accordance with the basic PWS and performance standards.

The associated performance-based contract has a hybrid of Cost-Plus-Fixed-Fee (CPFF), Firm-Fixed-

Price (FFP), Fixed-Price Incentive Firm (FPIF), and Cost Reimbursement (no fee) Contract Line Item

Numbers (CLINs). The period of performance is an initial 15-month base ordering period followed by nine option ordering periods, including an initial 14-month option and concluding with a six-month option period.

2 AUTHORITY

This surveillance plan is developed in accordance with the relevant clauses for inspection of services and/or inspection of supplies in Federal Acquisition Regulation (FAR) Part 46, Quality Assurance, which establish the procedures for monitoring, reporting, and evaluating the performance of the

Contractor. The Government acceptance of supplies or services is to be executed by the PCO or designated COR. The frequency and method of the Government’s inspection, review responsibilities, and reporting requirements follow.

3 SCOPE

3.1 Contract Scope

The PWS for ePS establishes the requirements for a System Integrator (SI) and Commercial Cloud

Service Provider (CCSP). This modernization effort will replace the DON’s legacy Contract Writing

Systems (CWS) and other ancillary procurement systems or portals, to include: the Standard

Procurement System (SPS) (i.e., the Navy substantiation of Procurement Desktop Defense (PD2)), Integrated Technical Item Management and Procurement (ITIMP), SeaPort-e, and PRISM. These procurement systems write and manage the majority of the contracts for the DON’s 10 Heads of

Contracting Activity (HCAs). Consequently, ePS is targeted to consolidate the functional capabilities of multiple legacy systems into an integrated solution.

Surveillance_Plan.docx

Page:

6/29

Version:

Version 1.0

3.2 Contract Quality Assurance

Per the Defense Federal Acquisition Regulation Supplement (DFARS) 246.4, Government Contract

Quality Assurance, and the contract PWS Section 4.7.2, Quality Control Plan (QCP), the

Government will monitor and record the Contractor’s performance as stated in this QASP. Section

4.7.2 requires that the Contractor’s QCP (Contract Data Requirements List (CDRL) A001, Quality

Control Plan) must provide the Government insight into performance criteria for each project phase including a definition of the performance requirements, performance objectives, acceptable quality levels, and surveillance methods.

The PCO may notify the Contractor of required modifications to the plan during the period of performance, as corrective actions are identified.

3.3 Government Quality Assurance Surveillance

The Government will monitor Contractor performance under this contract in accordance with this

QASP and the associated reports as Appendices A, B, and C.

Requirements for performance include CDRLs, work products, meetings, monthly Project

Management Reviews (PMRs), and technical reviews to provide progress of the contracted effort.

The Government COR will monitor performance and review performance reports furnished by the

Contractor to determine how the Contractor is performing against communicated and documented performance objectives. The Government will make decisions based on performance measurement metric data and will notify the Contractor of those decisions. The Contractor will be responsible for making required changes in processes and practices to ensure performance is managed effectively.

4 ROLES AND RESPONSIBILITIES

The following Government personnel positions will oversee and implementation surveillance activities:

4.1 Assistant Program Manager

The ePS Assistant Program Manager (APM) provides primary program oversight. While the APM may serve as a direct conduit to provide Government guidance and feedback to the Contractor on project matters, the APM is not empowered to make any contractual commitments or any contract changes on behalf of the Government. The APM will recommend a COR to the PCO, ensure the

COR is trained before performing any delegated duties, and will support COR performance assessment activities.

4.2 Procuring Contracting Officer

The Space and Naval Warfare Systems Command (SPAWAR) will assign a PCO, duly appointed with the authority to enter into, administer, or terminate contract actions such as basic awards or modifications, and make related determinations and findings on behalf of the Government. See

FAR 1.602, Contracting Officers, for authority and responsibilities.

Surveillance_Plan.docx

Page:

7/29

Version:

Version 1.0

The PCO will ensure performance of all necessary actions for effective contracting and ensure compliance with the contract terms and conditions. The PCO serves as the official Government agent with authority to modify and interpret the contract with the Contractor. The PCO is also the only Government official who can authorize contract actions that may result from review of

Contractor performance. The PCO will also ensure that the Contractor receives impartial, fair, and equitable treatment.

The PCO relies on the COR to provide information and recommendations regarding Contractor performance. While the PCO may delegate many of the day-to-day contractual administration duties to the COR, certain contractual actions remain as the sole purview of the PCO. These actions include negotiation and issuance of contract modifications, issuance of Contract Deficiency Reports, issuance of “show cause” letters, contract termination, contract closeout, and Contractor claims and disputes resolution. The PCO will designate a COR, in writing to act as the authorized representative to assist in the administration of the contract. The PCO will ensure the COR is trained before performing any duties that support COR performance assessment activities.

The PCO is ultimately responsible for the final determination of adequacy regarding Contractor performance.

4.3 Contracting Officer Representative

The COR is responsible for providing continuous technical oversight of Contractor performance.

The COR is not empowered to make any contractual commitments or authorize any contractual changes on behalf of the Government. The COR will alert the PCO for potential resolution or action concerning any changes that may affect the price, terms, or conditions of the contract. The PCO delegation memorandum is the source of authority for the COR.

See FAR 1.604, Contracting Officer’s Representative (COR), and DFARS 201.602-2, Responsibilities, for definition of responsibilities.

5 METHODS OF QUALITY ASSURANCE

5.1 Surveillance Folder

A Surveillance Folder will be used by the primary COR assigned to perform surveillance of

Contractor performance for Contractor-assigned contract requirements. The folder has the following divisions:

a. COR appointment letter, PCO letter of designation, and Training Record

b. PWS and other pertinent parts of the contract

c. Surveillance Plan

d. Surveillance Reports

e. Correspondence: customer and user comments

Surveillance_Plan.docx

Page:

8/29

Version:

Version 1.0

f. Data and technical reports

5.2 Evaluation Areas of Contractor Performance Assessment Report

The COR will perform an assessment quarterly and file an annual assessment of the following evaluation factors:

a. Quality of Product or Service (Technical Performance) – assess the Contractor’s effort to transform operational needs and requirements into an integrated solution.

Areas of focus may include the planning and management of program tasks, quality of support provided throughout all phases of contract execution, and management of a totally integrated effort of all program management concerns to meet cost, performance, and schedule objectives. The Contractor will be evaluated monthly as to the quality of the CDRLs, work products, and metrics in tables 1, 2, and 3. Results will be documented in the COR file. The technical performance includes the

Contractor’s contribution in meetings and reviews; the quality of the Contractor's technical reports; the Contractor's productivity level, and the overall quality of the technical support provided.

b. Schedule and Timeliness of Deliveries – assess the Contractor’s adherence to the required delivery schedule by evaluating the Contractor’s efforts during the assessment period that contribute to or effect the schedule variance. This evaluation will take into consideration the significance of scheduled events, and will also assess the effectiveness of corrective actions taken by the Contractor. The Contractor should be responsive to Government tasking and submit their CDRLs, work products and reports as required by the contract. The Contractor shall conduct scheduled reviews on time and shall submit material for the review per the agreed upon process and associated timelines as required by the contract.

c. Cost/Resource Control – assess the Contractor’s effectiveness in forecasting, managing, and controlling cost; assess the causes and Contractor-proposed solutions for the existence of cost overruns; address underruns in the same manner where performance is adversely impacted; and assess the extent to which the Contractor demonstrates cost responsibility through the efficient use of resources. The

Contractor shall use the funding authorized to provide support throughout the period of performance. The Contractor should not exceed the material, travel, or other cost-reimbursable CLINs without prior PCO approval. The Contractor will be evaluated on the successful control of resources dedicated to PWS tasks. Monthly, the COR will compare the Contractor burn rate of labor, and funds usage, with those allocated for the effort on the CLIN. The PCO will be notified of any discrepancies, and when funds reach 25% remaining.

d. Business Relations or Management – assess the turnaround, completeness, and quality of problem identification, corrective action plans, the cooperativeness of the

Contractor, and customer satisfaction; assess the Contractor’s success with

Surveillance_Plan.docx

Page:

9/29

Version:

Version 1.0 management of subcontracts; assess the Contractor’s ability to identify and apply resources required to meet schedule requirements, assign responsibility for tasks/actions required by the PWS and communicate appropriate information to affected program elements in a timely manner; assess the Contractor’s risk mitigation plans; and identify any other management areas that are applicable to the PWS.

Contractor management should put adequate systems in place to most efficiently perform the contract and protect Government interests. For example, the Contractor should have approved Government property management, quality control, subcontracting, and accounting systems in place.

e. Small Business Participation – assess the Contractor-submitted Electronic

Subcontracting Report System (www.eSRS.gov) report; assess compliance with all terms and conditions in the contract/order relating to small business participation

(including FAR 52.219-8, Utilization of Small Business Concerns and FAR 52.219-9, Small Business Subcontracting Plan (when required)); assess any small business participation goals which are stated separately in the contract; and assess achievement on each individual goal stated within the contract or subcontracting plan including good faith effort if the goal was not achieved.

f. Other Areas – assess additional evaluation areas within the terms and conditions of the contract or unique to the PWS.

5.3 Contract Manpower Reporting Application

The COR is responsible for validating Contractor input into the Contract Manpower Reporting

Application (CMRA) website at https://cmra.army.mil. The COR must obtain a user name and password, and validate that the Contractor has completely filled in the required information on the

CMRA web site by October 31 of the first year of the contract, and annually by October 31 thereafter for the duration of the contract, or sooner if the contract is closed out. The COR will perform this validation during the month of November each year. The COR will report the CMRA results (due in

December of each year) in the November Monthly COR Report.

https://cmra.army.mil/

Surveillance_Plan.docx

Page:

10/29

Version:

Version 1.0

5.4 Performance Rating Definitions

The Government ratings and narrative must reflect FAR Part 42.1503, Procedures, Table 42-1

Evaluation Ratings Definitions, as depicted in Table 5-1:

Table 5-1: Evaluation Rating Definitions

RATING DEFINITION CONTRACTOR NOTE

Exceptional – Performance meets contractual requirements and exceeds many to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with few minor problems for which corrective actions taken by the Contractor were highly effective.

To justify an Exceptional rating, identify multiple significant events and state how they were of benefit to the Government. A singular benefit, however, could be of such magnitude that it alone constitutes an Exceptional rating. Also, there should not have been any significant weaknesses identified.

Very Good – Performance meets contractual requirements and exceeds some to the Government’s benefit. The contractual performance of the element or sub-element being evaluated was accomplished with some minor problems for which corrective actions taken by the Contractor were effective.

To justify a Very Good rating, identify a significant event and state how it was a benefit to the Government.

There should not be any significant weaknesses identified.

Satisfactory – Performance meets contractual requirements. The contractual performance of the element or sub-element contains some minor problems for which corrective actions taken by the

Contractor appear or were satisfactory.

To justify a Satisfactory rating, there should have been only minor problems, or major problems the Contractor recovered from without impact to the contract. There should not have been significant weaknesses identified.

A fundamental principle of assigning ratings is that contractors will not be evaluated with a rating lower than

Satisfactory solely for not performing.

Marginal – Performance does not meet some contractual requirements. The contractual performance of the element or sub-element being evaluated reflects a serious problem for which the

C o n t r a c t o r has not yet identified corrective actions. The Contractor proposed actions appear only marginally effective or were not fully implemented.

To justify Marginal performance, identify a significant event in each category that the Contractor had trouble overcoming and state how it impacted the G overnment.

A Marginal rating should be supported by referencing the management tool that notified the Contractor of the contractual deficiency (e.g., management, quality, safety, or environmental).

Unsatisfactory – Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the element or sub-element contains a serious problem(s) for which Contractor corrective actions appear or were ineffective.

To justify an Unsatisfactory rating, identify multiple significant events in each category that the Contractor had trouble overcoming and state how it impacted the

Government. A singular problem, however, could be of such serious magnitude that it alone constitutes an unsatisfactory rating. An Unsatisfactory rating should be sup por t ed by referencing the management tools used to notify the Contractor of the contractual deficiencies

(e.g., management, quality, safety or environmental deficiency reports or letters).

Surveillance_Plan.docx

Page:

11/29

Version:

Version 1.0

RATING DEFINITION CONTRACTOR NOTE

NOTE 1: Plus or minus signs may be used to indicate an improving (+) or worsening (-) trend insufficient to change the evaluation status.

NOTE 2: N/A (not applicable) should be used if the ratings are not going to be applied to a particular area for evaluation.

5.5 Performance Standards

Performance standards define the desired service levels. The Government performs surveillance to determine if the Contractor exceeds, meets, or does not meet these standards.

The Performance Requirements Matrix: System Performance Measures and System Attributes and

(Table 2) include the ePS system attributes and system performance measures delineated in the ePS

System Requirements Specification (SRS) and Capabilities Requirements Document (CRD). The

Government will use these standards as well as the quarterly assessments by the COR, to measure

Contractor performance.

Table 5-2: System Performance Measures and System Attributes

SYSTEM

ATTRIBUTES

AND

PERFORMANCE

MEASURES

OVERVIEW

THRESHOLD /

ACCEPTABLE

QUALITY LEVEL

OBJECTIVE

METHOD OF

SURVEILLANCE

SRS 5.2.1

System

Responsiveness

The ability of the system to provide timely transactional and query process results to the user.

Transactional

(OLTP):

95% within 5.0 sec

Query (OLAP):

95% within 60.0 sec

Transactional

(OLTP):

95% within 2.0 sec

Query (OLAP):

95% within 15.0 sec

Random Sampling

Analysis

SRS 5.2.2

Data Restoration

The ability of the system to provide restored capability from a catastrophic loss or data disruptions from system operations to include corruption, destruction, loss, and compromise of the eps components.

Recovery Time

Objective (RTO) = 24

HRS

Recovery Point

Objective (RPO) = 8

HRS

RTO = 8 HRS

RPO = 8 HRS

Random Sampling

Analysis

SRS 5.2.4.1

Reliability

Mean time between failure

(MTBF) is measured by total operating time /number of failures

Threshold = 720 HRS Objective = 1,080

HRS

Random Sampling

Analysis

SRS 5.2.4.2

Availability

Availability is measured by uptime / (uptime + scheduled downtime + unscheduled downtime)

Threshold = 96%

Including Scheduled

Downtime

99.5% Discounting

Scheduled Downtime

Threshold =

Objective

Random Sampling

Analysis

Surveillance_Plan.docx

Page:

12/29

Version:

Version 1.0

SYSTEM

ATTRIBUTES

AND

PERFORMANCE

MEASURES

OVERVIEW

THRESHOLD /

ACCEPTABLE

QUALITY LEVEL

OBJECTIVE

METHOD OF

SURVEILLANCE

SRS 5.2.4.3

Maintainability

Maintainability is measured by the amount of time it takes a system or system element to be repaired and is expressed as mean time to repair (MTTR).

MTTR = total corrective maintenance time/ total number of corrective maintenance actions.

MTTR = 6.7HRS MTTR = 2.7HRS Random Sampling

Analysis

SRS 5.1.1

Data Error

Rejection

Percent of erroneous data identified and rejected prior to data storage.

90% at IOC

95% at FOC directive issuance

99.9% Random Sampling, Analysis

SRS 5.1.2

Document Preview

Rendering Time

Percent of procurement instruments (e.g., awards, modifications, and orders) forms rendered, measured as a function of time, starting with the user selecting document preview (acceptable loss).

90% within 10 seconds of user selecting document preview

90% within 10 seconds of user selecting document preview

Random Sampling, Analysis

SRS 5.1.3

Contract

Instrument

Generation

Percent of procurement instruments (e.g., awards, modifications, and orders) generated as a function of time.

90% within 30 seconds of PCO signature

99.9% within 30 seconds of PCO signature

Random Sampling, SRS 5.1.4

Entitlement Data

Transmission

Percent of transmission of contract entitlement data to the appropriate entitlement system from the ePS.

90% within 4 hours of award

95% within 4 hours of award Random Sampling, Analysis

SRS 5.1.5

Obligation Data

Transmission

Percent of contract actions finalized in the ePS that transmit obligation data electronically to the appropriate financial management system.

90% within 4 hours of award

95% within 4 hours of award

SRS 5.1.6

Contract Data

Compliance

Percent of contract actions that are successfully distributed to electronic document access (EDA) upon execution (i.e., PCO signature). This presumes that all pre-release validation checks have successfully occurred prior to contract

95% of eligible contract actions successfully meet all federal, DoD and

DON data validation requirements as assessed through centralized data

99.9% of eligible contract actions successfully meet all federal, DoD and

DON data validation requirements as assessed through centralized data

Surveillance_Plan.docx

Page:

13/29

Version:

Version 1.0

SYSTEM

ATTRIBUTES

AND

PERFORMANCE

MEASURES

OVERVIEW

THRESHOLD /

ACCEPTABLE

QUALITY LEVEL

OBJECTIVE

METHOD OF

SURVEILLANCE

execution and that the entire document chain (i.e., base and all modifications) was generated within the ePS.

validation services(s).

validation services(s).

5.6 Timely Feedback to Contractor

The COR will provide timely feedback to Contractors, in writing regarding unacceptable performance. Prompt feedback is essential so that the Contractor can develop and implement a corrective action plan. The Contractor's corrective action must be reported to the PCO for tracking purposes.

5.7 Documents / Reporting

A copy of the Quarterly Contractor Performance Review shall be provided to the PCO. The COR will report any issues and concerns to the PCO, which cannot be resolved at the working level, or impact contractual requirements for resolution and contractual action as they occur.

5.8 Government Surveillance Metrics

The Government will use generally accepted surveillance methods. These methods include, but are not limited to:

a. 100% Inspection – the Government will evaluate 100% of designated tasks. This surveillance method is appropriate for tasks where 100% inspection is required and is practical.

b. Partial Inspection – the Government will evaluate less than 100% of designated tasks and on a selective basis. This surveillance method is appropriate for tasks that occur infrequently and where 100% inspection is neither required nor practical.

c. Random Sampling – the Government will evaluate less than 100% of designated tasks on a random basis. This surveillance method is appropriate for tasks that occur infrequently and where 100% inspection is neither required nor practical.

d. Analysis– the Government will conduct operational monitoring of the program, and examine reports generated by automated tools and logs and websites.

e. Performance Review Meetings – the Contractor shall attend Contract Performance Reviews as required. These meetings will focus on overall contract performance. Specific dates, locations, agenda, and attendance requirements will be specified by the PCO at least 30 days prior to each meeting.

Performance objectives are used to define the desired outcomes. Performance standards are used to define the level of service required under the contract to successfully meet the performance objective.

Surveillance_Plan.docx

Page:

14/29

Version:

Version 1.0

These objectives and thresholds are contained in the Surveillance Matrices in Tables 2 and 3 along with the systems attributes/functions, acceptance quality, and surveillance methodology. These define how, when, and what will be assessed in measuring performance. The Government performs surveillance, using this QASP, to determine the quality of Contractor performance. The PWS should be used to form the foundation of the COR inspection checklist.

Surveillance_Plan.docx

Page:

15/29

Version:

Version 1.0

Table 5-3 is the ePS Focus Areas Surveillance Matrix. The Contractor’s ability to execute the items reflected in this table will have a significant impact on the success of ePS for the duration of the contract.

Table 5-3: Surveillance Matrix (ePS Focus Area)

FOCUS AREA OVERVIEW

THRESHOLD /

ACCEPTABLE

QUALITY LEVEL

OBJECTIVE

METHOD OF

SURVEILLANCE

PWS 4.3.5

Gap Closure

Design and implement a solution to address the identified gaps between the proposed out of the box COTS product capabilities certified in Attachment 15

(Gate Form) and the requirements set in the System Requirements Specification

(SRS).

Closure of 100% of the identified gaps reflected in

Attachment 15 by the time Release 2 is deployed.

Accomplish 100% of gap closures in accordance with the Contractor developed;

Government approved Gap

Solution

Methodology submitted as

CDRL A013, Gap

Closure.

Analysis

Program Management

Reviews

PWS 4.1.10 and

4.3.12

Auditability

As a business system with financial implications it is critical that the ePS program be compliant with all federal level guidance that dictates the separation of financial responsibilities and allows for full transparency and compartmentalized access to all transactions with financial implications.

The ePS system must achieve a state of compliance with all

Financial

Improvement and

Audit Readiness

(FIAR) Federal

Information System

Controls Audit

Manual (FISCAM), and Risk

Management

Framework (RMF)

The ePS system must enable the

Government to achieve compliance, at the system level, with all FIAR, FISCAM, and

RMF

requirements.

Random Sampling

Analysis

Program Management

Reviews

Surveillance_Plan.docx

Page:

16/29

Version:

Version 1.0

FOCUS AREA OVERVIEW

THRESHOLD /

ACCEPTABLE

QUALITY LEVEL

OBJECTIVE

METHOD OF

SURVEILLANCE

requirements prior to

Limited Deployment.

PWS 4.3.13

Cybersecurity

Focuses on identifying, implementing, and testing the prescribed security controls and in accordance with the

Risk Management Framework (RMF) for DoD Information Technology.

Cybersecurity controls on ePS must securely manage system and application authentication and access, contain role-based data access schemas, protect security-critical system components and securely interface with add-on security tools and features.

Fully execute

Contractor responsibilities identified in

CDRL A002,

DoD Risk

Management

Framework, and other

Cybersecurity related CDRLs

(A003, SEMP;

A020, Test Plan;

and C001, Status

Report).

Partial Inspection

Analysis

Program Management

Reviews

PWS 4.4.3

Training

Delivery and

PWS 4.7.7

Training

Management

End-user training includes development and delivery of training to ePS end users, end- user training will be delivered through Vendor Training, Initial –Led Training (ILT), eLearning

Computer Based Training (CBT).

100% of Government

Tier I and II personnel trained throughout the duration of contract.

Fully execute

Contractor training plan described in

CDRLs A010, Course Conduct

Information

Package; B003, Training Conduct

Support

Document; B005, Instructional

Performance

Analysis

Surveillance_Plan.docx

Page:

17/29

Version:

Version 1.0

FOCUS AREA OVERVIEW

THRESHOLD /

ACCEPTABLE

QUALITY LEVEL

OBJECTIVE

METHOD OF

SURVEILLANCE

Requirements

Document; and

C002, Presentation

Materials.

PWS 4.5.2 and

4.6.1.2

Service Desk

Operations, Escalation

&Metrics

The Contractor will provide effective assistance to ePS users as part of a streamlined process through which support requests can be managed efficiently.

The Contractor shall provide Service Desk availability during normal, local time and duty and duty hours with after-hours on-call availability.

Fully execute strategy / plan described in

CDRL A039,

Service Desk

Support and

Strategy Plan.

Random Sampling

Analysis

Surveillance_Plan

Page:

18/29

Version:

Version 2.0

3 Mar 2017

Table 5-4 is the Surveillance Matrix (Surveyed Products). This table reflects all CDRLs associated with the contract the CDRLs and standards that must be delivered by the Contractor. This matrix details the method of surveillance the COR will use to validate and inspect the CDRLs. Inspection of each element will be documented in the COR file.

Table 5-4: Surveillance Matrix (Surveyed Product)

SURVEYED PRODUCTS ENTRY CRITERIA METHOD OF SURVEILLANCE

1. CDRLs A001 – A046

2. CDRLs B001 – B008

3. CDRLs C001 – C024

Delivered on time in the format required per DD-1423

100% inspection of initial submission of CDRLs.

For each CDRL the COR and

Government team will provide:

Comment Matrix

Documented conversations and/or correspondence regarding issues

COR and Government team will review:

Meeting Minutes

Action Items

Corrective Action/Customer

Feedback Reports and Records:

1. ePS Corrective Action Report

2. ePS Customer

Complaint Record

3. ePS Performance Assessment

Report

Report(s)/Records(s) forms submitted by COR or a Customer

100% inspection of submitted reports and Customer Complaint Records.

The COR and Government team will r e v i e w Report/Record form with corrective actions; plans/tasks will be tracked until completely resolved.

5.9 Remedies for Nonconformance and Monitoring Forms

In accordance with FAR 46.407, Nonconforming Supplies or Services, the PCO should reject supplies or services not conforming in all respects to contract requirements. If inspections indicate an unacceptable product or performance, the COR will notify the Contractor POC of the deficiencies for correction. The PCO ordinarily must give the Contractor an opportunity to correct or replace nonconforming supplies or services when this can be accomplished within the required delivery schedule.

If deficiencies are not corrected within the required time frame, the COR should notify the PCO for action. If the Contractor disagrees with the noted discrepancy and an agreement cannot be reached, the COR will notify the PCO for a final decision.

Surveillance_Plan

Page:

19/29

Version:

Version 2.0

Three forms will be utilized to record nonconformance:

1. Performance Assessment Report (PAR): the PAR (Attachment A) will be used to report all minor discrepancies and will be generated by the COR and sent to the Contractor for corrective action. Contractors will be given 10 days to correct PARs unless they have requested an extension. If three minor discrepancies are found, combine them, and process a

Corrective Action Report (CAR).

2. Corrective Action Report (CAR) - The CAR (Attachment B) will be used to report a situation when there are three or more minor discrepancies, or one or more major discrepancies documented. The CAR will be generated by the COR and forwarded to the PCO for action.

A copy will be provided to the Contractor representative by the PCO. Most CARs will be corrected within 30 calendar days. Depending on the complexity of the finding, the

Contractor may be allowed a maximum of six months to correct a CAR. The COR will request that the Contractor provide the status of CARs on a monthly basis. The COR will keep an electronic log of all CARs and will keep a hard copy of CARs in the contract surveillance file.

3. Customer Complaint Record - the COR must furnish instructions for this form (Attachment C) to the customer receiving the Contractor service including the contract requirements for the service or product received. The customer must retain supporting evidence of complaint or feedback. After the COR validates feedback or complaint demonstrating Contractor noncompliance, the COR processes a PAR or CAR to the PCO. All Customer Complaint

Record forms will be validated by the COR prior to notifying the Contractor.

5.10 Contract Performance Assessment Reporting System

Contract Performance Assessment Reporting System (CPARS (www.cpars.gov)) will be prepared at least annually, documenting Contractor performance for this specific contract (see FAR 42.1502, Policy). More frequent updates may be made as required to document a significant improvement or degradation of the level of contract performance.

5.11 Disputes

Every attempt will be made to resolve any disputes arising under this contact/plan using FAR 33.214, Alternative Dispute Resolution. If the parties cannot reach a resolution, the PCO will render a final decision.

Surveillance_Plan

Page:

20/29

Version:

Version 2.0

APPENDIX A: PERFORMANCE ASSESSMENT REPORT

Performance Assessment Report (PAR) Instructions

Block 1. Quality Assurance Personnel (QAP) enters contract or task order number.

Block 2. QAP enters contractor name.

Block 3. QAP enters type of services.

Block 4. QAP signs and dates.

Block 5. QAP enters telephone number.

Block 6. QAP assigns suspense date for Contractor Representative validation if a deficiency box was checked in Block 7.

Part I. Contractor Performance

Block 7. QAP checks all boxes that apply.

Block 8. QAP enters Services Summary or Statement of Work (SOW) paragraph item reviewed.

Block 9. QAP enters brief description of deficiency and tracking number if the deficiency box was checked in Block 7. Detailed description will be on Corrective Action Report (CAR).

Block 10. QAP enters a detailed performance assessment if the deficiency box in Block 7 was not checked.

Parts II and III will be used only if a deficiency was documented in Part I.

Part II. Contractor Validation

Block 11. Contractor Representative checks one box in response to QAP cited deficiency. If

Concurrence is indicated, Contractor Representative continues to Block 12. If Non-Concurrence is indicated, continue to Block 13 and state reason(s) for Non-Concurrence with QAP cited deficiency and return PAR to QAP.

Block 12. Contractor Representative enters Estimated Completion Date of Corrective Action and continues to Block 13.

Surveillance_Plan

Page:

21/29

Version:

Version 2.0

Block 13. Contractor Representative details Corrective Action and the Action Taken to Prevent

Recurrence or Reason for Non-Concurrence of QAP cited deficiency.

Part III. Deficiency Corrected

Block 14. QAP checks appropriate block, signs, and dates.

Block 15. (Required Entry) QAP remarks regarding the corrective action and the action taken to prevent recurrence. If Non-Concur was checked, this block should be very detailed to explain why.

Block 16. Contractor Representative enters remarks if further discussion is needed about QAP concurrence with Contractor’s Corrective Action and Action Taken to Prevent Recurrence.

Continuation Sheet. The Continuation Sheet is to be used to expand on any information that will not fit in the applicable numbered block.

PERFORMANCE ASSESSMENT REPORT (PAR)

(If more space is needed, use reverse and identify by number)

1. CONTRACT/TASK ORDER NUMBER 2. CONTRACTOR 3. TYPE OF SERVICES

4. QUALITY ASSURANCE PERSONNEL (QAP) SIGNATURE AND DATE 5. QAP PHONE 6. SUSPENSE DATE

I. PERFORMANCE

7. DEFICIENCY (CHECK ALL BOXES THAT APPLY)

NEW

REPEAT

NO DEFICIENCY NOTED

8. SERVICES SUMMARY or SOW PARAGRAPH ITEM REVIEWED

9. BRIEF DESCRIPTION OF DEFICIENCY (IF DEFICIENCY BOX WAS

CHECKED)

10. DETAILED PERFORMANCE ASSESSMENT

II. CONTRACTOR VALIDATION

11. CONTRACTOR REPRESENTATIVE CONCUR NON-

CONCUR

12. CORRECTIVE ACTION ESTIMATED COMPLETION DATE

13. CONTRACTOR REPRESENTATIVE CORRECTIVE ACTION AND PREVENTION OF RECURRENCE OR REASON FOR NON-

CONCURRENCE OF QAP CITED DEFICIENCY

III ACTION CORRECTED

14. CONCUR NON-CONCUR QAP SIGNATURE AND DATE

15. QAP REMARKS (REQUIRED)

16. CONTRACTOR REPRESENTATIVE REMARKS

Performance Assessment Report (PAR) Template, Feb 05 (SAF/AQCP)

PERFORMANCE ASSESSMENT REPORT (PAR)

Continuation Sheet

(Number to correspond with applicable Item Number on reverse)

Performance Assessment Report (PAR) Template, Feb 05 (SAF/AQCP)

Surveillance_Plan

Page:

24/29

Version:

Version 2.0

APPENDIX B: CORRECTIVE ACTION REPORT

Corrective Action Report (CAR) Instructions

Block 1. Enter Contractor Name.

Block 2. Enter Contract Number.

Block 3. Enter Contract Name or Type of Services.

Block 4. Enter Functional Area of the Contract.

Block 5. Enter the assigned suspense date given the contractor to provide a response to the CAR. A date must be entered for a Major CAR. A date is optional at the discretion of the Quality Assurance

Personnel (QAP) initiating the CAR if the finding is Minor.

Block 6. All CARs will be tracked with a Control Number. The functional commander/functional director is the Issuing Authority for his/her contract Quality Assurance Evaluators (QAPs) generating

CARs. By providing the control number to the QAP originating the CAR, the Issuing Authority demonstrates concurrence with the finding. The control number will be made up of the first two letters of the name of the site or contract, the last two digits of the calendar year, and a three digit number starting with 001 and progressing upward throughout the calendar year (e.g., WP04-001).

Block 7.

1. Check the block that indicates whether the identified deficiency is assigned as a Major or Minor finding. See Performance Plan (PP), paragraph 6.1.4, for the definitions of Major and Minor findings and associated explanations.

2. Finding: Clearly state the details of the finding followed by a reference to the stated contractual requirement. The reference must state the portion of the contract, part, section, paragraph and subparagraph and must make a complete brief quotation of the contract reference.

3. State the impact that the finding has or could have on the accomplishment of the mission that the contract provides.

4. When determining how long to give the contractor to respond to the identified finding, the normal is 10 working days. The suspense date should reflect this unless the finding requires a greater amount of time to come to solution. Major findings require a contractor response. Minor findings may require a contractor response at the discretion of the QAP initiating the CAR and/or the Issuing Authority.

Blocks 8 and 9. Contract QAPs initiating a CAR must sign in Block 8 and the Issuing Authority signs in Block 9. Each annotates the date at the time of signature.

Block 10. Upon review of the contractor’s reply, the originator of the CAR will enter comments regarding acceptance or rejection of the contractor’s response. This block may also contain any comments regarding follow-on inspections conducted or needed at a later date to validate that the finding has not reoccurred.

Surveillance_Plan

Page:

25/29

Version:

Version 2.0

Block 11. The QAP selects Accept or Reject after reviewing the contractor’s response.

Block 12. The QAP enters a close date after advising the Issuing Authority that the contractor’s response is acceptable and the Issuing Authority concurs. The Issuing Authority will then forward the closed CAR to the contractor and report accordingly in the monthly Certificate Of Service (COS) or other applicable contracting officer services report. This routing remains the same if the contractor response is unacceptable and the CAR remains open until an acceptable response is received except that the Issuing Authority forwards the open CAR back to the contractor for additional information.

Continuation Sheet. The Continuation Sheet is to be used to expand on any information that will not fit in the applicable numbered block.

CORRECTIVE ACTION REPORT (CAR)

(If more space is needed, use reverse and identify by number)

1. CONTRACTOR 2. CONTRACT NUMBER 3. TYPE OF SERVICES

4. FUNCTIONAL AREA 5. SUSPENSE DATE 6. CONTROL NUMBER

7. DEFICIENCY MAJOR MINOR

FINDING:

FINDING IMPACT:

Please respond with a written corrective action plan that details the corrective action of the cited deficiency, the cause of the deficiency, and actions taken to prevent recurrence by Suspense Date in Block 5. If date was not entered in Block 5, the contractor is not required to provide a response.

8. QUALITY ASSURANCE PERSONNEL (QAP)

TYPED NAME AND GRADE SIGNATURE AND DATE

9. ISSUING AUTHORITY

TYPED NAME AND GRADE SIGNATURE AND DATE

10. QAP RESPONSE TO CONTRACTOR CORRECTIVE ACTION AND ACTION TAKEN TO PREVENT RECURRENCE

11. QAP DETERMINATION

ACCEPTED REJECTED

12. CLOSE DATE

Corrective Action Report (CAR) Template, Feb 05 (SAF/AQCP)

CORRECTIVE ACTION REPORT (CAR)

Continuation Sheet

(Number to correspond with applicable Item Number on reverse)

Corrective Action Report (CAR) Template, Feb 05 (SAF/AQCP)

APPENDIX C: CUSTOMER COMPLAINT RECORD

Form on next page.

CUSTOMER COMPLAINT RECORD

DATE/TIME OF COMPLAINT

SOURCE OF COMPLAINT

ORGANIZATION BUILDING NUMBER INDIVIDUAL PHONE NUMBER

NATURE OF COMPLAINT

CONTRACT REFERENCE

VALIDATION

DATE/TIME CONTRACTOR INFORMED OF COMPLAINT

ACTION TAKEN BY CONTRACTOR

RECEIVED/VALIDATED BY

Customer Complaint Record Template, Feb 05 (SAF/AQCP) COMPLAINT-1