Attachment J-5_SWIFT Quality Assurance Surveillance Plan.pdf

Page | 1

ATTACHMENT J-3

QUALITY ASSURANCE SURVEILLANCE PLAN (QASP)

2/25/22

U.S. Department of Homeland Security

Scalable Ways to Implement Flexible Tasks (SWIFT)

Contract Number: TBD Contractor’s name: TBD (hereafter referred to as the Contractor)

Page | 2

INTRODUCTION

This Quality Assurance Surveillance Plan (QASP) is pursuant to the requirements listed in the Scalable Ways to Implement Flexible Tasks (SWIFT) Performance Work Statement (PWS). This plan sets the procedures and guidelines the Immigration and Customs Enforcement (ICE) Office of the Chief Information Officer (OCIO) will use in ensuring the required performance standards or services levels are achieved by the Contractor. Other work required under this contract may be monitored and other contractual remedies taken by the Government as needed.

The subject QASP will be utilized by the Government to monitor performance at the Indefinite Delivery-Indefinite Quantity (IDIQ) contract level and by DHS Components at the Task Order level.

EXECUTIVE SUMMARY

This QASP provides a systematic method to evaluate performance for the stated contract and explains the following:

What will be monitored?

How monitoring will take place?

Who will conduct the monitoring?

How monitoring efforts and results will be documented?

This QASP does not detail how the Contractor will accomplish the work. Rather, the QASP is created with the premise that the Contractor is responsible for management and quality control actions necessary to meet the terms of the contract. It is the Government’s responsibility to be objective, fair, and consistent in evaluating performance. In addition, the QASP recognizes that unforeseen and uncontrollable situations may occur.

This QASP is a “living document” and the Government may review and revise it on a regular basis. However, the Government will coordinate changes with the Contractor. Updates shall ensure that the QASP remains a valid, useful, and enforceable document. Copies of the original QASP and revisions shall be provided to the Contractor and Government officials implementing surveillance activities.

The following FAR clauses may apply depending on contract type:

37.6 Performance-Based Acquisition

46.4 Government Contract Quality Assurance

52.246-4 Inspection of Services – Fixed-Price, 52.246-6 Inspection of Services – Time-and-Material and Labor-Hour

PURPOSE

This QASP describes the procedures the ICE will use to monitor and evaluate the Contractor’s performance.

The primary concern of ICE is with the determined quality of the services provided by the Contractor.

Therefore, the QASP focuses on the Government’s measure of the Contractor’s performance on service level agreements (SLAs). It is intended that the QASP be a tool to guide the Contracting Officer’s Representative (COR) in assessing Contractor performance. In some cases, specific metrics will be used to measure Contractor performance; in other cases, subjective judgment, and evaluation by ICE /OCIO personnel will be the determining criteria. This plan describes the methodology utilized to conduct both quantitative and qualitative evaluation of Contractor performance under the contract.

Page | 3

The QASP provides a means for evaluating whether the Contractor is meeting the performance standards/quality levels identified in the PWS

QUALITY MANAGEMENT STRATEGY

The Contractor is responsible for the quality and consistency of all services provided. The Contractor measures that quality through execution of its own Quality Management Plan. ICE will regularly review quality of delivered services through surveillance of performance of the requirements stated in the Performance Work Statement, Service Level Agreement (SLA) performance, reviews of deliverables, and reviews of stated operational practices.

The contractor SLA performance will generally be monitored on a monthly basis. The COR and designates will surveille stated monthly performance metrics, the trends, and report any discrepancies in findings.

The Contractor PM shall participate in quarterly “self-assessments “with the COR and associated PM designates. Quarterly performance reviews will be conducted informally and act as a “progress report” on the annual assessment of Contractor performance. Quarterly assessments will detail adherence to the submitted Quality Management Plan (QMP) deliverable and performance in stated Contractor Performance Assessment Reporting System (CPARS) categories.

The COR will enter annual performance assessments in the CPARS system that encompass monthly SLA performance metric analysis, details of Contractor performance and support, and a qualitative review of performance that encompasses quarterly assessments of performance.

The COR will monitor performance and review deliverables furnished by the Contractor to determine how the Contractor is performing against communicated performance objectives of the QMP. The COR will surveille determinations regarding incentives and disincentives based on the QASP and notify the Contractor of findings.

The Contractor will be responsible for adjusting SLA performance levels, incentives, disincentives, and deliverables as necessary, based on COR review. The ICE on-site representatives will monitor and report to the COR on Contractor practices, adherence to industry standards, performance, and support.

QASP RELATION TO THE QUALITY MANAGEMENT PLAN

The Contractor’s QMP is a formal contract deliverable. ICE expects the implementation of the Contractor’s QMP requirements will be sufficient in meeting the performance details of the PWS. While the QMP represents the way the Contractor will pursue quality and timeliness of services, as defined in the PWS, the QASP represents the way ICE will evaluate the Contractor’s performance. The Contractor’s QMP and the QASP should be complementary and ultimately ensure successful Contractor performance.

REVISIONS TO THE QASP

The QASP will be used in ICE ’s administration of the contract and remains subject to revision at any time by the Government throughout the contract performance period. Revisions to this surveillance plan are the responsibility of the Contracting Officer (CO) or designee with support, input, and coordination with the ICE program designees. An initial revision encompassing surveillance details related to the Contractor’s submitted QMP may be made within 60 days of the initial QMP submission. While the Contractor may be engaged, determinations of necessary revisions or changes may be made unilaterally at the discretion of the Government.

Page | 4

As the performance period progresses, surveillance may be altered as determined necessary to improve performance in specific areas. ICE may also alter surveillance as key priority changes are made by DHS

OCIO.

ROLES AND RESPONSIBILITIES

As the purpose of QASP is to ensure that the Government receives satisfactory services and that the Contractor is meeting contractual requirements, the roles and responsibilities of the Contractor and Government involved in the QASP are described below.

CONTRACTOR ROLES AND RESPONSIBILITIES

The Contractor is responsible for delivering acceptable service levels, as detailed in the QASP and per industry standards for any actions and activities not covered in the QASP. The Contractor is responsible for implementing its QMP, which is a contract deliverable. The QMP describes the Contractor’s methods for ensuring all services provided under the contract meet established performance standards. The Contractor is responsible for producing, maintaining, and providing for audit, quality assurance/control records and reports and all records associated with the investigation and resolution of COR-identified performance issues.

Program Manager (PM) – The PM is responsible for all Contractor work performed under the contract and as stated in the PWS. The PM shall meet with the COR and OCIO Enterprise Platform Services Program Manager regularly. The PM shall participate in quarterly performance reviews and engage Contractor staff as appropriate. The PM is responsible for Quality Management. Quality Management includes providing appropriate visibility as to the quality of work products and the maintenance of standards, processes, and procedures at the program and project level across the Enterprise Platform Services Branch.

The following employees of the Contractor serve as the Contractor’s representatives for this contract:

a. Program Manager: <upon award, enter name> Telephone: <enter number> Email: <enter address>

b. Other Contractor Personnel: Upon award, enter the following lines for each additional individual.

Delete these lines if not applicable:

<enter title>: <upon award, enter name> Telephone: <enter number> Email: <enter address>

GOVERNMENT ROLES AND RESPONSIBILITIES

The Government is responsible for clear communication of established ICE initiatives, standards for performance, and acceptance of deliverables. The Government also shall revise the QASP to surveille in concert with stated QMP practices.

Contracting Officer (CO) - The CO shall ensure performance of all necessary actions for effective contracting, ensure compliance with the contract terms, and shall safeguard the interests of ICE regarding the contractual relationship. The CO shall also assure that the Contractor receives impartial, fair, and equitable treatment under this contract. The CO is ultimately responsible for the final determination of the adequacy of the Contractor’s performance.

Page | 5

Contracting Officer’s Representative (COR) - The COR is responsible for technical administration of the contract and shall assure proper Government surveillance of the Contractor’s performance. The COR shall revise the QASP to meet the specifics of the submitted QMP and also keep a quality assurance file including (but not limited to) monthly SLA results, quarterly quality review details and annual CPARS. At the conclusion of the contract or when requested by the CO, the COR shall provide documentation to the CO. The COR is not empowered to make any contractual commitments or to authorize any contractual changes on the Government’s behalf. The Contractor shall refer any changes they deem may affect contract price, terms, or conditions to the CO for action.

Government Program Manager – The OCIO Enterprise Platform Services Program Manager shall meet regularly with the COR and Contractor PM to keep open communication of customer and executive feedback and keep the Contractor PM abreast of ICE and OCIO initiatives. The OCIO Enterprise Platform Services Program Manager may engage or designate additional Government Project Managers as necessary to participate in related meetings.

The following Government personnel represent the Government for this contract:

Assigned CO: TBD Organization or Agency: DHS ICE OAQ Telephone: <enter telephone number> Email: <enter email address>

Assigned COR: TBD Organization or Agency: DHS ICE OCIO

Assigned Program Manager: TBD Organization or Agency: DHS ICE OCIO SDD

PERFORMANCE STANDARDS

Performance standards define desired services. The Government performs surveillance to determine if the Contractor exceeds, meets, or does not meet these standards.

The Performance Work Statement (PWS) (Attachment J-2) and the QASP include performance standards and other applicable contract requirements. The Government shall use these standards to assess Contractor performance and shall compare Contractor performance to the Acceptable Quality Levels (AQL) for each Performance Objective.

INCENTIVES

The Government shall NOT use incentives and disincentives for the Enterprise SLAs. Incentives or disincentives shall be based on exceeding, meeting, or not meeting performance standards. Information about incentives can be found in the QASP.

Page | 6

METHODS OF QUALITY ASSURANCE SURVEILLANCE

Various methods exist to monitor performance. The COR shall use the surveillance methods listed below in the administration of this QASP. Regardless of the surveillance method, the COR shall always contact the Contractor's task manager or on-site representative when a defect is identified and inform the manager of the specifics of the problem. The COR shall be responsible for monitoring the Contractor’s performance in meeting a specific performance standard/AQL. Final determination of methods of surveillance will be determined upon review of the Contractor’s QMP deliverable.

a) Direct Observation – (Can be performed periodically or through 100% surveillance by the COR and or designates.)

- Performance Standard: All performance standards listed in the QASP and other applicable contract requirements.

b) Management Information Systems (MIS). (Evaluate outputs through the use of monthly contract deliverable management information reports and other compliance output from other entities.)

- Performance Standard: All SLAs listed in the QASP and other applicable contract requirements.

c) Periodic Inspection. (Uses a comprehensive evaluation of selected outputs. Inspections may be scheduled [daily, weekly, monthly, quarterly, or annually] or unscheduled, as required.)

d) Validated User/Customer complaints. (Relies on the user of the service to identify deficiencies.

Complaints are then investigated and validated.)

e) Random Sampling. (Designed to evaluate the outputs of the award requirement by randomly selecting and inspecting a statistically significant sample, such as sanitization service of level 1 hardware.)

f) Periodic Sampling. (Variation of random sampling. However, sample is only taken when a deficiency is suspected. May be a follow-up to MIS analysis. Sample results are applicable only for the specific work inspected. Since the sample is not entirely random, it cannot be applied to total activity performance.)

g) Progress or Status Meetings. (Status Meetings include at a minimum a Monthly SLA review and daily status briefing/meetings with key contract personnel, support personnel, and onsite federal representatives; regular Progress Meetings; ad hoc meetings directed by the customer and Contractor that may address the execution of a separate Task Order, tech uplift of supporting equipment, security related matters, and any other topics of concern.)

h) Performance reporting. (Evaluates metrics for a specific time period as listed in the QASP.)

- Performance Standard: All SLAs listed in the QASP and other applicable contract requirements

Page | 7

i) User Survey. (Combines elements of validated user complaints and random sampling. Random survey is conducted to solicit user satisfaction. Appropriate for high quantity activities that have historically been satisfactory. May also generate periodic and 100% inspections.)

Surveillance results may be used as the basis for revisions to contract deliverables and to substantiate CPARS review ratings. In such cases, the Inspection of Services clause in the Contract becomes the basis for the CO’s actions.

Table 1 lists the DCCO performance objectives and corresponding acceptable quality levels (AQLs)

Table 1: Performance Requirements Summary: Performance Objectives and Acceptable Quality Levels

Service Output

Performance Objective/PWS Task Acceptable Quality Level (AQL)

Method of Inspection

Positive Incentive

Negative Incentive

1 Information Security Awareness Training for the measured reporting period/ PWS Section 7

95% COR review of

PALMS

compliant report

CPARS

Rating

CPARS

Rating

2 Provide roll out deployments successfully using deployment scripts or automation builds/ PWS Sections 2.0/ 2.1/ 2.5

90% COR review of Contractor provided SLA outputs detailing metrics

CPARS

Rating

CPARS

Rating

3 Collaboration and Innovation. Contractor teams will be evaluated based on innovative approaches they introduce to accomplish the work assigned, and their willingness and demonstrated effort to work collaboratively across the entire ecosystem to share knowledge, experience, and lessons learned. The government will evaluate contractors based on examples of innovative approaches that are introduced. / PWS Sections 2.0 /2.1/2.2 /2.3 /2.4

95% COR review feedback reviews from Program Manager

CPARS

Rating

CPARS

Rating

4 Process discipline and continuous improvement.

Contractor teams will be assessed based on their discipline in implementing lean software development processes, their conformance to established Agile and DevSecOps best practices as required by ICE frameworks / PWS Section 2.0

95% COR review feedback reviews from Program Manager

CPARS

Rating

CPARS

Rating

Page | 8

5 Replace legacy solutions with RPA bots. Increase automation for efficiency and cost savings/ PWS Section 2.2

90% COR review of Contractor provided SLA outputs detailing metrics

CPARS

Rating

6 Provide the latest technologies associated with digital graphic design and ensure that all related design support is compatible with Enterprise Platform Services and web standards and industry best practices/PWS Section 2.4

90% COR and Program Manager review

CPARS

Rating

CPARS

Rating

7 Provide the technical expertise and support identifying, coordinating, and implementing new technologies into the Enterprise Platform Service environment/PWS Section 2.4

95% COR and

Manager review

CPARS

Rating

CPARS

Rating

8 Provide solution and system testing support for all solutions and custom scripts developed by the vendor/PWS Section 2.4

95% COR and Program Manager review

CPARS

Rating

CPARS

Rating

9 Provide requirements review and create and execute test plans based on the requirements gathered/PWS Section 2.4

95% COR and Program Manager review

CPARS

Rating

CPARS

Rating

10 Provide and validate developed applications meet the stated requirements/PWS Section 2.4

95% COR and Program Manager review

CPARS

Rating

CPARS

Rating

11 Provide development and O&M support for the Enterprise Platform Service and any applications at Task Order level/PWS Section 2.0

95% COR and Program Manager review

CPARS

Rating

CPARS

Rating

12 Provide support for application development, existing platform environments and prospective workflow automations while the agency expands the operational model to include additional ICE programs and offices, evaluates the adoption of new and emerging technologies, as well as new software and alternative hosting options.

/PWS Section 2.1

Manager review

CPARS

Rating

CPARS

Rating

13 Provide and prepare a monthly progress report. Initial reports are due to the COR thirty (30) days after Task Order kickoff and every thirty (30) days thereafter until the last month of performance; the final delivery will occur ten (10) days before the end of the final option period and will summarize performance during the period of performance and provide the status of any planned transition activity/PWS Section 4.3

Manager review

CPARS

Rating

Page | 9

RATINGS

The Contractor’s SLA measurement outputs will be used to determine if performance exceeds, meets, or does not meet the expectations of the ICE. The Contractor’s overall success rate in meeting the determined AQLs will have significance in completion of the Contractor’s annual CPARS rating.

DOCUMENTING PERFORMANCE

The Government shall document performance using CPARS.

When exemplary or unacceptable performance occurs, the COR shall inform the Contractor in writing. The COR shall document the discussion and place it in the COR file.

When unacceptable performance occurs, conflicting with requirements of the contract, the COR will prepare a Contract Discrepancy Report (CDR) and present it to the Contractor's program manager and or on-site representative. A CDR template is attached to this QASP. The CDR and any other documentation (including but not limited to letters to the file) will document deficient Contractor performance and specify any determined need for corrective actions.

The Contractor shall acknowledge receipt in writing. If the Contractor is required to prepare a corrective action plan to document how the Contractor shall correct the unacceptable performance and avoid a recurrence, the COR will specify how long after receipt the Contractor has to present this corrective action plan to the COR. The Government shall review the Contractor's corrective action plan to determine acceptability.

CDRs and any other related documentation detailing exemplary or unacceptable performance shall become a part of the supporting documentation for contract quarterly incentive/disincentive allocations, or other contractual actions deemed necessary by the CO.

Page | 10

Table 2. CPARS Ratings

Definition

Notes

(a) Exceptiona l

Performance meets contractual requirements and exceeds many to the Government’s benefit. The contractual performance of the element or sub- element being evaluated was accomplished with few minor problems for which corrective actions taken by the

Contractor were highly effective.

To justify an Exceptional rating, identify multiple significant events and state how they were of benefit to the Government.

A singular benefit, however, could be of such magnitude that it alone constitutes an Exceptional rating. Also, there should have been NO significant weaknesses identified.

(b) Very Good requirements and exceeds some to the Government’s benefit. The contractual performance of the element or sub- element being evaluated was accomplished with some minor problems for which corrective actions taken by the

Contractor were effective.

To justify a Very Good rating, identify a significant event and state how it was a benefit to the Government. There should have been no significant weaknesses identified.

(c) Satisfactor y requirements. The contractual performance of the element or sub-element contains some minor problems for which corrective actions taken by the Contractor appear or were satisfactory.

To justify a Satisfactory rating, there should have been only minor problems, or major problems the Contractor recovered from without impact to the contract/order. There should have been NO significant weaknesses identified. A fundamental principle of assigning ratings is that contractors will not be evaluated with a rating lower than Satisfactory solely for not performing beyond the requirements of the contract/order.

Page | 11

(d) Marginal

Performance does not meet some contractual requirements. The contractual performance of the element or sub- element being evaluated reflects a serious problem for which the Contractor has not yet identified corrective actions. The Contractor’s proposed actions appear only marginally effective or were not fully implemented

To justify Marginal performance, identify a significant event in each category that the Contractor had trouble overcoming and state how it impacted the Government. A Marginal rating should be supported by referencing the management tool that notified the Contractor of the contractual deficiency (e.g., management, quality, safety, or environmental deficiency report or letter).

(e) Unsatisfact ory

Performance does not meet most contractual requirements and recovery is not likely in a timely manner. The contractual performance of the element or sub-element contains a serious problem(s) for which the

Contractor’s corrective actions appear or were ineffective.

To justify an Unsatisfactory rating, identify multiple significant events in each category that the Contractor had trouble overcoming and state how it impacted the Government. A singular problem, however, could be of such serious magnitude that it alone constitutes an unsatisfactory rating. An

Unsatisfactory rating should be supported by referencing the management tools used to notify the Contractor of the contractual deficiencies (e.g., management, quality, safety, or environmental deficiency reports, or letters).

Page | 12

CONTRACT DISCREPANCY REPORT (CDR)

1. CONTRACT NUMBER: <insert number>

2. TO: (Contractor Program Manager, Task Manager, or on-site representative) <insert name>

3. FROM: COR <insert name of COR>

4. DATE AND TIME DISCREPANCY OBSERVED: <insert date and time>

5. DISCREPANCY OR PROBLEM:

<Describe in detail. Identify any attachments.>

6. CORRECTIVE ACTION PLAN:

A written corrective action plan < is / is not > required.

< If a written corrective action plan is required include the following. > The written Corrective Action Plan will be provided to the undersigned not later than < # days after receipt of this

CDR. >

Prepared by: <Enter COR’s name>

Contracting Officer’s Representative Date

Received by:

Contractor Program Manager, Task Manager, or Date on-site representative

< The COR may initiate a CDR at any time, including whenever the number of monthly recorded defects for a performance standard exceeds the allowable number of defects;

anytime unacceptable performance is determined critical in nature and requires formal corrective action; and whenever an unfavorable trend is detected in Contractor performance.>