Attach 21_Compare Doc.pdf

Attachment 21 Solicitation #: W900KK-20-R-0011 Rev Date: 7 Feb27 Mar 2020

ITSM – Quality Assurance Surveillance Plan Page 1 of 19

QUALITY ASSURANCE SURVEILLANCE PLAN

Version 1 – dd mmm yyyy

National Cyber Range Complex (NCRC) Event Planning, Operations, and Support (EPOS)

NCRC Information Technology Service Management (ITSM) Small Business (SB) Task Order XXXX

Contractor’s name: {to be determined}

1. PURPOSE.

This Quality Assurance Surveillance Plan (QASP) is a Government developed document used to ensure that the Government receives quality services, and pays only for conforming services actually provided.

All tasks are defined in the NCRC EPOS Small Business (SB) Task Order for NCRC Information Technology Service Management PWS, sections 3 & 4. Government-furnished property and services for those items specified in the PWS are exempt from this QASP. The Contractor shall perform to the standards specified in relevant sections of the PWS.

This Quality Assurance Surveillance Plan (QASP) identifies the methods and procedures the Government will use to ensure it receives the services to the desired standard, as identified in the

PWS.

This Quality Assurance Surveillance Plan (QASP) provides a systematic method to evaluate performance for the stated contract. This QASP explains the following:

What will be monitored.

How monitoring will take place.

Who will conduct the monitoring.

How monitoring efforts and results will be documented.

This QASP does not detail how the contractor accomplishes the work. Rather, the QASP is created with the premise that the contractor is responsible for management and quality control actions to meet the terms of the contract. It is the Government’s responsibility to be objective, fair, and consistent in evaluating performance. In addition, the QASP should recognize that unforeseen and uncontrollable situations may occur.

This QASP is a “living document” and the Government may review and revise it on a regular basis. However, the Government shall coordinate changes with the contractor. Updates shall ensure that the QASP remains a valid, useful, and enforceable document. Copies of the original QASP and revisions shall be provided to the contractor and Government officials implementing surveillance activities.

ITSM – Quality Assurance Surveillance Plan Page 2 of 19

1.1. Intent

The intent of this QASP is to:

1) Ensure Contractor performance meets or exceeds contract terms, conditions and specifications

2) Hold the Contractor accountable for quality control

3) Encourage the Contractor to take appropriate steps to control and improve quality

4) Ensure all sections of FAR 46 are applied as required and applicable

Accordingly, the Government will perform surveillance on the contract in accordance with this QASP, but reserves the right to monitor the contract in any manner necessary, at any time necessary, and at all places necessary to ensure that the services rendered conform to contract requirements. The Government also reserves the right to perform quality assurance at the Subcontractor level, and at the contractor's place of business. Sub-contract service providers shall be accountable to the Contractor who, in turn is responsible to the Government. Resolution of non-conforming services discovered at the Subcontractor level will be addressed with the prime Contractor. The Government will not exercise any supervisory controls over the contract service providers performing the services herein.

2. Description of Services.

2.1. Scope of Work.

The objective of this contract is to provide Information Technology Service Management (ITSM) services to include: Enterprise Configuration Management (CM); Enterprise Trouble Ticketing/Problem Reporting; Enterprise Property and Asset Management; and Enterprise Software License Management. The services support the Test Resource Management Center’s (TRMC) mission to provide realistic cybersecurity environments for Test and Evaluation (T&E) of major Department of Defense (DoD) acquisition programs and to enable the conduct of realistic training and certification events for the DoD Cyber Mission Force (CMF). The NCRC facility may also support non-DoD customers such as Department of Homeland Security (DHS), industry, academia and international partners

2.2. Type of Contract.

This will be a Multiple Award Task Order contract under a Multiple Award ID/IQ.

3. Government Roles and Responsibilities.

The following personnel shall oversee and coordinate surveillance activities.

a. Contracting Officer (CO) - The CO shall ensure performance of all necessary actions for effective contracting, ensure compliance with the contract terms, and shall safeguard the interests of the United States in the contractual relationship. The CO shall also assure that the contractor receives impartial, fair, and equitable treatment under this contract. The CO is ultimately responsible for the final determination of the adequacy of the contractor’s performance.

The CO is the only person authorized to direct changes in any of the requirements under this contract. In the event the Contractor effects any such change at the direction of any person other

ITSM – Quality Assurance Surveillance Plan Page 3 of 19 than the CO, the change will be considered to have been made without authority, and solely at the risk of the Contractor. No changes in, or deviation from the terms and conditions shall be affected, without a written modification to the contract, executed by the CO authorizing such changes.

Assigned CO: Jean BorowskiDuane A. St. Peter Organization or Agency: U.S. Army Contracting Command – Orlando Bravo Division OPB Telephone: (407-380-4261) 384-3889 Email: jean.m.borowskiduane.a.stpeter.civ@mail.mil

b. Contract Specialist (CS) - The CS acts as an acquisition consultant and serves as liaison between the requesting program office, the contractor, and the supporting contracting office.

Assigned CS: Robert (Bob) BairdDavid B. Hector Organization or Agency: U.S. Army Contracting Command – Orlando Bravo Division OPB Telephone: (407-208-3485) 384-3721 Email: robert.d.baird10david.b.hector.civ@mail.mil

c. Contracting Officer’s Representative (COR) - The COR is responsible for technical administration of the contract and shall assure proper Government surveillance of the contractor’s performance. The COR shall keep a quality assurance file. At the conclusion of the contract or when requested by the CO, the COR shall provide documentation to the CO. The COR is not empowered to make any contractual commitments or to authorize any contractual changes on the Government’s behalf. The contractor shall refer any changes they deem may affect contract price, terms, or conditions to the CO for action.

The COR will monitor all technical and performance aspects of the contract, and assist the CO in contract administration. The COR is authorized to perform the following functions:

• Assure that the Contractor performs the technical requirements of the contract

• Perform inspections necessary in connection with contract performance

• Maintain written and oral communications with the Contractor concerning technical and performance aspects of the contract

• Issue written interpretations of technical requirements, including Government drawings, designs, and specifications

• Monitor Contractor's performance and documents and notifies both the CO and Contractor of all deficiencies

• Coordinate availability of Government furnished property

Some personnel may be assigned as Alternate CORs (ACOR), but other than a statement to that effect and what actions the CO expects that ACOR to take in the absence of the primary COR, there is no difference between a COR and an ACOR, or other surveillance personnel assigned to conduct contract oversight. Throughout the remainder of this QASP, The term “COR’ will be used to refer to all government surveillance personnel assigned to monitor the contractor’s performance on this contract.

ITSM – Quality Assurance Surveillance Plan Page 4 of 19

A letter of designation issued to the COR, a copy of which is sent to the Contractor, states the responsibilities and limitations of the COR, especially with regard to changes in cost, price, or delivery dates. The COR is not authorized to change any of the terms and conditions of the resulting order.

The Contracting Officer shall use the DoD Joint Appointment Module (JAM) to electronically track COR nominations, appointments, terminations, and training certifications for each COR appointed to conduct oversight on this contract. The COR will upload their surveillance documents, such as COR Status Reports (Smartform), COR trip reports, correspondence with the contractor, and other miscellaneous documents into the Surveillance and Performance Monitoring (SPM) module within the online contract file. JAM and SPM are available within the Procurement Integrated Enterprise Environment (PIEE) suite of tools at: https://wawf.eb.mil/.

Assigned COR: Rolando Lopez Organization or Agency: Program Executive Office Simulation, Training and Instrumentation Command (PEO STRI) Telephone: 407-384-5241 Email: rolando.lopez39.civ@mail.mil

d. Alternate Contracting Officer’s Representative (ACOR). The Alternate COR is an employee of the U.S. Government provided by the Requiring Activities (RA) and appointed by the Contracting Officer to provide additional surveillance personnel in monitoring and documenting the contractor’s performance. These alternate surveillance support personnel may serve as on-site representatives of the COR in performance of actual contract surveillance, if they meet all COR training and experience requirements and are appointed by the contracting officer as Alternate CORs (ACOR) in the Joint Appointment Module (JAM). This term will apply to other government surveillance personnel as defined in Army Regulation 70-13, paragraph 2-2 g.

Assigned Alternate COR (ACOR): {to be determined} Organization or Agency: {to be determined} Telephone: {to be determined} Email: {to be determined}

e. Quality Assurance Representative (QAR) - The QAR is responsible for technical surveillance of the contract and shall assess the contractor’s performance against the scope of requirements.

The QAR shall review appropriate contractor reports and conduct management of contract to ensure the contractor performs within cost, schedule, and performance metrics. On a monthly basis, the QAR will provide a report to the COR addressing the above stated metrics. The QAR is not empowered to make any contractual commitments or to authorize any contractual changes on the Government’s behalf. The contractor shall refer any changes they deem may affect contract price, terms, or conditions to the CO for action.

Assigned QAR: Francisca (Cisca) Vuong Organization or Agency: Test Resource Management Center (TRMC) Telephone: 407-384-3908 Email: francisca.r.vuong.civ@mail.mil

ITSM – Quality Assurance Surveillance Plan Page 5 of 19

Assigned Alternate QAR (AQAR): Dr. Robert Tamburello Organization or Agency: TRMC Telephone: 571-372-2753 Email: robert.n.tamburello.civ@mail.mil

3.1 CONTRACTOR REPRESENTATIVES:

The following employees of the contractor serve as the contractor’s Program Manager and Task Manager for this contract.

a. Program Director: {to be determined} Telephone: {to be determined} Email: {to be determined}

b. Program Manager: {to be determined} Telephone: {to be determined} Email: {to be determined}

c. Contracts Manager: {to be determined} Telephone: {to be determined} Email: {to be determined}

4. PERFORMANCE STANDARDS.

Performance standards define desired services. The Government performs surveillance to determine if the contractor exceeds, meets or does not meet these standards.

The Performance Work Statement (PWS) for the NCRC Information Technology Service Management (ITSM) contract delineates the tasks required by the Government. These tasks shall be subject to the following performance standards listed in the table below. Method of Surveillance codes indicated in the table are are explained in Section 5 following the table.

In the table below, the following abbreviations apply:

“IDIQ PWS” refers to the Small Business (SB) Indefinite Delivery/Indefinite Quantity (ID/IQ) Performance Work Statement (PWS) for National Cyber Range Complex (NCRC) Event Planning, Operations, and Support (EPOS)

“ITSM PWS” refers to the Performance Work Statement (PWS) for National Cyber

Range Complex (NCRC) Event Planning, Operations, and Support (EPOS) NCRC Information Technology Service Management (ITSM) Small Business Task Order

XXXX

ITSM – Quality Assurance Surveillance Plan Page 6 of 19

Table 1 - Performance Standards Summary Matrix

Line No

Task Document Section

Performance Standard Method of Surveillance

Frequency Evaluation Criteria

T1 Organizational Conflict of Interest (OCI)

IDIQ PWS

3.17

ITSM PWS

4.16

Ensure that the contractor communicates actual or potential OCI issues to the PCO and COR/ACOR. Ensure that the contractor enters into Non- Disclosure and Propritary Information Agreements (NDA/PIA) as required to accomplish required tasks.

C, D, E Status and Program Progress Reports, Potential/Actu al OCI event occurrence

Does the contractor communicate actual or potential OCI issues to the PCO and COR/ACOR 100% of the time?

T2 Enterprise Trouble Ticketing/Proble m Reporting/Help Desk

ITSM PWS

4.23.2

Ensure that the Contractor proposes an Enterprise Trouble Ticketing/Problem Reporting System (TTPRS). Ensure that the contractor implements the TTPRS system along with any associated help desk capability.

A, C, D, E, F

Periodic Observation, Audits, Status and Program Progress Reports

Does the contractor meet the timeline in implementing the approved enterprise TTPRS?

T3 Enterprise SW License Management

ITSM PWS

4.23.4

Ensure that the Contractor investigates and proposes an NCRC Enterprise-wide SW License Management (SLM) framework. Ensure that the contractor implements the approved SLM framework.

A, C, D, E, F

Periodic Observation, Audits, Status and Program Progress Reports

Does the Contractor meet the timeline in implementing the approved SLM framework?

T4 ITSM Training ITSM PWS 4.23.6

Ensure that the contractor delivers training, when and where requested.

A, C, D, E, F

Periodic Observation, Audits, Status and Program Progress Reports

Does the contractor meet the timeline in delivering the approved training material/courseware?

ITSM – Quality Assurance Surveillance Plan Page 7 of 19

Line No

Task Document Section

Performance Standard Method of Surveillance

Frequency Evaluation Criteria

T5 All CDRLs Document Summary List

Ensure that required CDRLs are submitted and contain required content in accordance with each

CDRL

C, D, E, F Progress, Status, and Management Reports. Other submissions as defined in

ITSM PWS

All deliverables shall comply with the requirements defined in the DD Form 1423.

ITSM – Quality Assurance Surveillance Plan Page 8 of 19

5. Methods of QUALITY ASSURANCE Surveillance.

Various methods exist to monitor performance. The QAR shall use the surveillance methods listed below in the administration of this QASP.

Regardless of the surveillance method, the QAR shall always contact the contractor's task manager or on-site representative when a defect is identified and inform the manager of the specifics of the problem. The QAR, with assistance, shall be responsible for monitoring the contractor’s performance in meeting a specific performance standard.

A. Direct Observation. (Can be performed periodically or through 100% surveillance.)

B. Management Information Systems (MIS). (Evaluates outputs through the use of management information reports. Best used for general surveillance and may need to be supplemented by periodic inspections.)

C. Periodic Inspection. (Uses a comprehensive evaluation of selected outputs.

Inspections will be as required.)

D. Progress or status meetings.

E. Analysis of contractor's progress reports. (Evaluate cost, schedule, etc.)

F. Performance reporting. (CDRLs and monthly COR reports)

5.1. Surveillance Schedule. A Surveillance Schedule (Appendix C) will be created each month.

The schedule is “FOR OFFICIAL USE ONLY” and is not releasable to anyone other than authorized Government personnel. Each Month’s Surveillance Schedule will be completed at the beginning of each month during the Period of Performance (PoP) to schedule the upcoming month surveillance activity. Completed Surveillance Schedules will be filed in the online Surveillance and Performance Monitoring (SPM) module located within (https://wawf.eb.mil) file for this contract, either as an attachment to COR Status Report (CSR) or uploaded separately in the Miscellaneous Documents Section.

5.2. Surveillance Checklists. A Surveillance Checklist will be created each month, using the template contained in Appendix D, populated with applicable items drawn from Table 1. The Checklist is “FOR OFFICIAL USE ONLY” and is not releasable to anyone other than authorized Government personnel. Each Month’s Surveillance Checklist will be completed with the Surveillance Schedule. Completed Surveillance Checklist will be filed in the online Surveillance and Performance Monitoring (SPM) module located within (https://wawf.eb.mil) file for this contract, either as an attachment to COR Status Report (CSR) or uploaded separately in the Miscellaneous Documents Section.

6. RATINGS.

ITSM – Quality Assurance Surveillance Plan Page 9 of 19

Metrics and methods are designed to determine if performance exceeds, meets, or does not meet a given standard and acceptable quality level. A rating scale shall be used to determine a positive, neutral, or negative outcome.

Surveillance results may be used as the basis for actions (to include payment deductions) against the contractor. In such cases, the Inspection of Services clause in the Contract becomes the basis for the CO’s actions. Additionally, surveillance results may be used to prepare Performance Assessment Reports through the Contract Performance Assessment Reporting System (CPARS).

7. DOCUMENTING PERFORMANCE.

COR Status Reports, will be submitted on a monthly basis to the Contracting Officer via the Surveillance and Performance Monitoring (SPM) module using the “Smart Form” COR Status Report (CSR). CSR reports will be due NLT the 10th day of the month following the month of surveillance. COR Status Reports will provide a synopsis of the Contractors performance for the inclusive dates of the report. The synopsis must contain a summary of surveillances performed, Customer Feedback if any, CNRs if any, an analysis of the Contractors performance, recommended adjustments to surveillance schedules, and any other pertinent contract information IAW FAR 46.104, and the COR Appointment Letter. Completed Contract Nonconformance Reports (CNRs), if any, will be included as attachments to COR Status Reports.

Documented surveillance data (COR Status Reports, Completed Customer Feedback and Contract Nonconformance Report (CNRs), if any) will be used to support CPARS ratings.

CPARS will be completed in accordance with AFARS, Subpart 5142.15 – Contractor Performance Information.

The official COR/Quality Assurance Surveillance Files will be maintained in the on-line Joint Appointment Module (JAM) and Surveillance and Performance Monitoring (SPM) module IAW DoDI 5000.72, and DFARS PGI 201.602-2.

7.1 Acceptable Performance.

The Government shall document positive performance. Any report may become a part of the supporting documentation for fixed fee payments or other actions. Additionally, surveillance results may be used to prepare Performance Assessment Reports in CPARS.

7.2 Unacceptable Performance.

Unacceptable performance (non-conformance) occurs when the Contractor fails to meet the terms, conditions and/or specifications of the Contract/Task Order. All instances of non-conformance must be documented by the COR. The COR will provide the details of the non-conformance using the Contract Non-conformance Report (CNR), Appendix B to the CO, who will notify the Contractor of each non-conformance occurrence, unless the CO has authorized the COR to submit directly to the contractor, and copy the CO. Resolution of any non-conformance discovered at the Subcontractor level will be addressed with the prime Contractor.

ITSM – Quality Assurance Surveillance Plan Page 10 of 19

Non-conformances are classified as either: Level I (Minor); Level II (Major); or Level III (Critical).

7.2.1. Level I (Minor). Level I non-conformance is not likely to materially reduce the usability of services for their intended purpose or is a departure from established standards having little bearing on completing the contract requirement. Minor non-conformances can usually be corrected on the spot and can be issued to the Contractor through either a verbal or written CNR.

However, the Government is required to document verbal Level I CNRs. The COR may issue verbal Level I CNRs directly to the appropriate level of Contractor management. The COR will notify the CO as soon as practical when a verbal Level I CNR is issued and document them in their Monthly Reports. If the non-conformance cannot be corrected on the spot, the Contractor shall be given a suspense date to correct the deficiency. The CO will issue and sign written Level I CNRs.

7.2.2. Level II (Major). Level II non-conformance is likely to result in failure of the services to meet contract requirements. Also, repeated discrepancies, a large number of documented Level I CNRs and trends indicating failure of the Contractor’s quality control program can be classified as Level II CNRs. The CO issues all Level II CNRs in writing using the form in Appendix B.

The COR will notify the CO as soon as practical when a major non-conformance is identified.

7.2.3. Level III (Critical). Level III non-conformances result in hazardous or unsafe conditions due to the manner in which the services are performed; prevent or impact a vital agency mission as a result of performance; or are for repeated or uncorrected Level II CNRs. The Contracting Office Director or Deputy Director issues written Level III CNRs. The COR is responsible for notifying the CO upon discovery of a critical non-conformance.

7.3 Documenting CNRs. The COR will provide the details of the non-conformance using the CNR form shown in Appendix B. If the Requiring activity prefers to issue a locally developed CNR, documentation must include as a minimum: the Contract or Task Order number; reference to the specific contract requirement; the specific discrepancy to the requirement; where it was discovered; the date and time it was discovered; Contractor representative who was notified; and the suspense date for Contractor response/corrective action.

7.4 Tracking and Resolving CNRs. All CNRs, verbal and written, regardless of issuer will be included in the COR’s Monthly Report. The COR and CO shall track CNRs to ensure the Contractor identifies and takes appropriate corrective action to the root cause and the specific non-conformance is corrected within the required timeline. The CO will make a final written determination of the Government’s remedies if the non-conformance is not corrected by the Contractor after receiving notice and a reasonable opportunity to correct the work.

Any documentation detailing non-conformance may become a part of the supporting documentation for contract payment deductions, fixed fee deductions, award fee nonpayment, or other actions deemed necessary by the CO. Additionally, surveillance results may be used to prepare Performance Assessment Reports in CPARS.

ITSM – Quality Assurance Surveillance Plan Page 11 of 19

8. FREQUENCY OF MEASUREMENT.

a. Frequency of Measurement.

During contract performance, the COR shall take periodic measurements as specified in the Frequency column of the Performance Standards Summary Matrix, and shall analyze whether the negotiated frequency of measurement is appropriate for the work being performed.

b. Frequency of Performance Assessment Meetings.

At a minimum, the QAR/COR shall meet with the contractor on monthly teleconference sessions and at the Government’s or the contractor’s facility to assess performance and shall provide a written assessment. Appendix A and B provide sample report formats.

Prepared by:

{to be determined}

QAR

{title to be determined}

ITSM – Quality Assurance Surveillance Plan Page 12 of 19

Appendix A Performance Report

1. CONTRACT NUMBER:

2. Prepared by: (Name of COR) <insert name>

3. Date and time of observation

4. Description of Observation:

<Examples of items to include in a report are:

- Method of surveillance

- How frequently you conducted surveillance

- Surveillance Results

- Number of Observations

- Prepared by: <Enter COR’s name>

Signature of Quality Assurance Representative Date

ITSM – Quality Assurance Surveillance Plan Page 13 of 19

Appendix B Contract Nonconformance Report (CNR)

The QAR may initiate a CNR at any time contractor performance is determined to be impacting cost, schedule or performance standards.

DA FORM

5479_Contractor Nonconformance Report.pdf

ITSM – Quality Assurance Surveillance Plan Page 14 of 19

5479_Contractor Nonconformance Report.pdf

Field Code Changed

ITSM – Quality Assurance Surveillance Plan Page 15 of 19

Appendix C

Contract Surveillance Schedule

ITSM – Quality Assurance Surveillance Plan Page 16 of 19

ITSM – Quality Assurance Surveillance Plan Page 17 of 19

If utilized, you will need to develop a monthly schedule for conducting surveillance. The date you are going to conduct surveillance would go in the far left column. You would list the PWS, paragraph number for the task you are going to surveil on the top line.

5475_Surveillance Sechedule.pdf

5475_Surveillance Sechedule.pdf

ITSM – Quality Assurance Surveillance Plan Page 18 of 19

Appendix D Contract Surveillance Checklist Template

COR SURVEILLANCE CHECKLIST Template COR Inspection Checklist & Data Report (Fixed Firm Price Contract) Period Of

Performance:

Contractor: From:

Contract Number: To:

Service Provided:

Meeting Contract Requirements? COR Name:

Date: YES No

Comments:

Comments summary, to include, positive trends, negative trends, concerns, customer comments, and any additional COR comments:

To populate the checklist, find in the PWS: "The Contractor Shall...”, Then, make that requirement a checklist item by converting the requirement to a question: “Did the contractor…" Each "Yes" can be graded as a 1, and each NO graded as a 0. Once the checklist is complete, the COR may generate a quantifiable performance rating using these numbers as metrics.

ITSM – Quality Assurance Surveillance Plan Page 19 of 19

Revision History Date Version Major Enhancement(s)