IEP2_0 Initial TO DRAFT QASP.pdf

Internal Revenue Service

Integrated Enterprise Portal (IEP) Performance Management

Quality Assurance Surveillance Plan (QASP)

Version 0.76

March 2023

Integrated Enterprise Portal Quality Assurance Surveillance Plan

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 ii March 2023

Review and Approval This Quality Assurance Surveillance Plan (QASP) documents how the Internal Revenue Service (IRS) will monitor Service Level Objectives (SLO) performance under selected Managed Services contracts. The IRS Enterprise Operations Web Infrastructure Services Division (WISD) has established a multi-tiered framework to support Integrated Enterprise Portal (IEP) 2.0 contract performance surveillance. As part of this framework, IEP 2.0 Quality Management (QM) will utilize Contractor-supplied data to monitor performance periodically and will provide the results of this surveillance to the Contractor.

The IRS WISD IEP 2.0 will augment this process with in-depth reviews (audits), as necessary.

Review of this QASP signifies formal notification of the periodic surveillance audit to monitor IEP 2.0 Contractor technical performance. If an in-depth review (audit) is necessary, the IRS will issue separate formal notification to the Contractor of its intent to perform the audit, no later than 30 calendar days before it is scheduled to begin. All material support on part to assist the IRS in its audit or in-depth deep dive reviews would be treated as a Project Change Request.

Review and Approval:

The following stakeholders have approved this IEP 2.0 Quality Assurance Surveillance Plan, vxxx:

EO ACIO, Date

WISD Program Manager, WISD Transition Manager, Date

WISD Quality Management Date

IEP 2.0 Contracting Officer, Date

IEP 2.0 Contracting Officer Representative Date

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 iii March 2023

Table of Contents

1. Introduction

1.1 Scope

1.2 Authority

1.3 Goals and Objectives

2. Roles and Responsibilities

2.1 QASP Update

2.2 Internal Revenue Service

2.3 Contractor

3. Approach

3.1 Overall Approach

3.2 Surveillance Methods

3.3 Evaluation Methodology

4. Required Performance Metrics

5. Reporting Flow

Appendix A: Metric Specifications Example

Appendix B: Forms

Appendix C: References

Appendix D: Acronyms

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 iv March 2023

Table of Tables

Table 2-1 Government Roles and Responsibilities Table 2-2 Contractor Roles and Responsibilities Table 4-1 Performance Requirement Summary (PRS)

Table of Figures Figure 5-1 Surveillance Reporting Flow

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 v March 2023

Record of Changes

Date Reference

(Table, Figure, or

Paragraph)

A=Add M=Modify

D=Delete

Brief Description of Change

12/20/2021 0.1 M Created baseline document

03/04/2022 0.2 A Updated to include SLOs and Incorporated general management concepts from RFI responses

08/17/2022 0.5 D, M Updated to include adjudicated responses for IPT review and feedback..

01/26/2023 0.6 A, M Updated to include new and modified SLOs

02/13/2023 0.6 M Updated the signatory. The contract number

‘TIRNO-23-xxxxxxxx’ to ‘TBD’

03/16/2023 0.7 M Incorporated WISD and OLS comments

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 1 March 2023

1. Introduction This Quality Assurance Surveillance Plan (QASP) documents the approach the Internal Revenue Service (IRS) will take to monitor portal Contractor technical performance required by the Performance Work Statement in effect at the time of the surveillance activity.

1.1 Scope

This QASP documents how the IRS will monitor Contractor Performance Service Level Objective (SLO) performance and deliverables furnished by the contract holder under this Integrated Enterprise Portal 2.0 contract TBD. The QASP will be tailored to the characteristics and requirements of the particular contract by the addition of the germane Performance Requirement Summary table in Section 4 of this document. SLOs not contained in this document will not be considered within scope of the QASP implementation. This Plan will be updated periodically to reflect the results of a joint review, by IRS and appropriate Contractor personnel, of SLOs, SLO definitions, and SLO monitoring approaches.

1.2 Authority

Authority for issuance of this QASP is the Federal Acquisition Regulation (FAR) Subparts

37.6 Performance Based Acquisition, 46 Quality Assurance, and 46.4 Government Contract Quality Assurance, as well as contract Section E – Inspection and Acceptance, which provides for inspections and acceptance of the work and services called for in-service contracts, to be executed by the Contracting Officer (CO) or a duly authorized representative. This QASP will be implemented by the IRS Web Infrastructure Services Division (WISD) Quality Management (QM) function.

1.3 Goals and Objectives

The QASP activity is a critical component for Contractor performance measurement. The QASP is a Government document used to monitor the Contractor’s Quality Assurance Program (QAP), which the Contractor must have in place, or shall establish prior to contract award, to attain, sustain and advance exceptional delivery of services to the IRS.

The Contractor’s QAP will include its description of people, tools, and processes that it will use to ensure compliance with all contract requirements and its proposed methods for providing continuous improvement in service delivery. The Contractor’s QAP is expected to include a detailed set of plans (for example, Quality Control (QC) Plan, Configuration Management and Change Control Plan, Asset Management Plan, Risk Management Plan and/or other plans at the Contractor’s discretion) that the Contractor will use to deliver quality Portal Services.

The QASP does not define or dictate how the Contractor should accomplish the work required under contract provisions; instead, the QASP measures the degree of

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 2 March 2023 achievement of the Contractor’s execution of its management and QC actions necessary to meet the performance standards documented herein. Thus, the internal continuous performance monitoring performed by the Contractor is not part of the QASP implementation activity but does yield data that are leveraged during QASP activities.

1.3.1 Goal

The overall goal of the QASP activity is to confirm that the IEP 2.0 Contractor’s delivery and execution of products or services meets the technical performance measures, or SLOs, included in the contract and documented in this QASP.

1.3.2 Objectives

Specific QASP objectives include:

• Determine Contractor performance is meeting all contract SLOs. Contractor performance data may be collected through audits, examinations, customer feedback and measurements gathered from reports.

• Provide data to the CO and Contracting Officer Representative (COR) to assess performance and other contract provisions.

• Identify portal program risks arising from Contractor performance in meeting contract SLOs.

• Identify Contractor corrective action(s) needed to meet SLOs or to address identified risks or issues.

2. Roles and Responsibilities The IRS bears primary responsibility for implementing this QASP, with support from the Contractor as described in FAR 46.105 Contractor responsibilities.

2.1 QASP Update

The QASP can be changed or updated. It is intended to be a “living” document that should be revised or modified as circumstances warrant. The Government may initiate changes to the QASP at any time during contract performance; and will inform the contractor. The contractor is allowed to recommend change and or updates to the QASP and must submit these recommendations in writing to the Office of Information Technology Acquisition (OITA) contracting official (see contract G.2) and a copy to the contracting office representative. Such recommendations are at the discretion of the contracting officer and technical manager to accept or reject. Any negotiations of the terms and conditions stated in this document will occur if SLOs are included in task orders issued, or through formal modification of contract TBD. Such changes shall not entitle the Contractor to any equitable adjustments or to any other compensation for performance in a prior period.

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 3 March 2023

2.2 Internal Revenue Service

Table 2-1 describes Government roles and responsibilities regarding QASP implementation.

Table 2-1 Government Roles and Responsibilities

Role Responsibilities

Enterprise Operations and WISD Management

• Establishes and approves the QASP

• Provides resources to implement the QASP

• Determines achievement of other contract provisions

• Analyzes Quality Assurance Surveillance Review (QASR) and similar feedback and findings, as appropriate

WISD Quality Management

(QM)

• Develops initial draft of the QASP and approves updates

• Conducts QASR Interviews business customer(s) Reviews Contractor-provided reports Conducts liaison with COR(s) Documents findings Provides formal written feedback, including findings Participates in project status reviews

• Performs in-depth quality review activities at Contractor location, as necessary and if authorized

• Debriefs Contractor (with CO and/or COR) on the surveillance review results

Contracting Officer

• Approves the QASP

• Monitors contract compliance, contract administration, and cost control

• Reviews QASR and similar feedback and findings, as appropriate

• Provides contractual direction to WISD and the contractor, and acts on recommendations submitted to the office.

• Debriefs Contractor (with WISD QM and/or COR) on the surveillance review results

• Ensures that non-conformances are identified and establishes the significance of a nonconformance when considering the acceptability of supplies or services which do not meet contract requirements.

Contracting Officer’s Representative

• Works with QM to develop initial QASP and approves QASP updates

• Approves the QASP Chairs Contractor deliverables

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 4 March 2023

Role Responsibilities review, Project Status Reviews (PSRs), and similar performance and status reviews with IRS stakeholders and Contractor

• Reviews QASR and similar feedback and findings, as appropriate

• Forwards QASR and similar reports to appropriate business customer (Online Services (OLS), etc.)

• Debriefs Contractor (with CO and/or WISD QM) on surveillance review results

2.3 Contractor

Table 2-2 describes Contractor roles and responsibilities regarding QASP implementation.

Table 2-2 Contractor Roles and Responsibilities

Role Key Responsibilities (see FAR 46.105 for all responsibilities)

Contractor Program Manager

• Controlling the quality of supplies or services

• Reviews the QASP, makes any recommendations to the CO/COR.

• Collects, analyzes, documents, and produces performance data

• Submits data reports to CO/COR

• Supports IRS QASRs and similar activities

• Identifies and implements corrective actions

• Submits requests to COR for new reports or measures

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 5 March 2023

3. Approach WISD has established a multi-tiered framework to facilitate quality assurance (QA) monitoring of portal contractor performance. WISD QM will leverage Contractor-supplied data to implement this framework. The following sections describe this strategy in more detail.

3.1 Overall Approach

The WISD QA monitoring framework described in this QASP integrates and leverages ongoing monitoring activities. WISD QM performs two types of audits, QA surveillance reviews on a regular basis and in-depth reviews (“deep dives”) on an exception basis, when implementing the QA monitoring framework. These surveillance reviews shall not unduly burden the Contractor and shall comply with FAR 52.246-4 Inspection of Services

– Fixed-Price. The Contracting Officer must be notified if the Contractor believes any action may affect contract price, terms, or conditions.

WISD QM’s basic approach is to: establish an agreed and understood quality baseline;

monitor Contractor performance against that baseline; identify any performance gaps on a timely basis; and confirm that corrective actions are applied in a timely fashion.

• WISD QM staff and other interested IRS stakeholders meet periodically with

Contractor program personnel to review SLOs in the QASP and the monitoring and reporting systems to enable continuous and accurate intake, analysis, and reporting of performance data.

• The Contractor implements continuous performance monitoring as agreed during the QA Program Baseline Review (PBR), uses internal Quality Control (QC) resources to meet SLO performance thresholds defined by the Acceptable Quality Levels (AQLs), and submits deliverables periodically to report status against the quality baseline.

Although these monitoring activities themselves are not included in the scope of the IRS’ QA surveillance, they yield measurement data and deliverables that are assessed during QASP implementation.

• The IRS COR conducts Project Status Reviews (PSR) with other relevant IRS stakeholders, including WISD QM and internal IRS SLO “owners”, to assess SLO status against the established baseline, identify emerging risks and issues, and determine potential “deep dive” opportunities.

• The WISD QM function audits Contractor performance by checking and synthesizing reported SLO data to develop quality surveillance feedback, including findings and potential corrective action requests, for submission to the CO/COR. These audit results are submitted using the form in Appendix B.

• The IRS reserves the right to perform in-depth reviews of Contractor performance (“deep dives”) as necessary to assess perceived or actual performance issues. These in-depth reviews may occur at any time during the evaluation period and may address performance issues arising in the current or any prior evaluation period. If an in-depth review (audit) is necessary, the IRS will issue separate formal notification to the

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 6 March 2023

Contractor of its intent to perform the review, no later than 30 calendar days before it is scheduled to begin. To support such deep dives, the IRS may request additional data from the Contractor, including but not limited to raw data, audit results, evidence of certification, QC program data, etc. The IRS reserves the right to perform these deep dives at Contractor or subcontractor facilities.

• The government representative(s) will monitor performance and review performance reports furnished by the contractor to determine how the contractor is performing against communicated performance objectives. The contractor is encouraged to provide impactful improvements to the SLOs at any time during the contract period of performance. The government will make determination regarding incentives based on performance measurement metric data and notify the contractor of those decisions. The contractor will be responsible for making required changes in processes and practices to ensure performance is managed effectively. Agreements on improved SLOs and performance measures will be documented in subsequent versions of this

QASP.

3.2 Surveillance Methods

The Federal Acquisition Regulation (46.401 General) states that quality assurance surveillance plans should be prepared in conjunction with the preparation of the statement of work. These plans should specify all work requiring surveillance and the method of surveillance employed [3]. The WISD QM has at its disposal several methods with which to perform the QA monitoring function described above:

• Contractor Self-Reporting. Uses contractor-supplied performance data collected by the tracking and monitoring tools and/or contractor status reports that identify problems, issues, and/or risks.

• Trend Analysis. Employed systematically over time to monitor contractor performance.

• Quantitative Analysis. This refers to the use of statistical and other quantitative techniques to analyze raw performance data to characterize performance and/or identify trends, causal relationships, or root causes of performance issues. Examples would include analyzing results of customer survey and problem tickets against relevant SLOs. This method is appropriate for in-depth analysis of areas of interest.

• Periodic Inspection. Per the Office of Management and Budget (Chapter 5, Quality Assurance Plan and Surveillance, P. 18): “This method, sometimes called ‘planned sampling,’ consists of the evaluation of tasks selected on other than a 100 percent or random basis. It may be appropriate for tasks that occur infrequently, and where 100 percent inspection is neither required nor practicable. A predetermined plan for inspecting part of the work is established using subjective judgment and analysis of agency resources to decide what work to inspect and how frequently to inspect it.”

(See Appendix C, Reference number 1)

• Random Sampling. Per the Office of Management and Budget (Chapter 5, Quality Assurance Plan and Surveillance, P. 19): “This is usually the most appropriate method for recurring tasks. With random sampling, services are sampled to determine if the

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 7 March 2023 level of performance is acceptable. Random sampling works best when the number of instances of the services being performed is very large and a statistically valid sample can be obtained.” (See Appendix C, Reference numbers 1)

• 100 Percent Inspection. Per the Office of Management and Budget (Chapter 5, Quality Assurance Plan and Surveillance, P. 19): “This is usually only the most appropriate method for infrequent tasks or tasks with stringent performance requirements, e.g., where safety or health is a concern. With this method, performance is inspected/evaluated at each occurrence. One hundred percent inspection is too expensive to be used in most cases.” (See Appendix C, Reference numbers 1.)

• Customer/User/Stakeholder Feedback. This method is an independent evaluation of an activity or process to assess compliance with SLOs. This surveillance method supplements the other surveillance methods. Customer Feedback may be necessary for certain types of tasks that do not lend themselves to random sampling of 100% inspection. Customer feedback may be obtained either from the results of formal customer satisfaction surveys or from random customer complaints or complements. Customer complaints, to be considered valid, must set forth clearly and in writing the detailed nature of the complaint, must be signed or emailed and forwarded to the project manager or COR who will assess the validity of the complaint and retain it in a documentation file.

3.3 Evaluation Methodology

WISD QM will use the cited observation methods to determine whether AQLs have been met over the evaluation time frame. The evaluation period will be annually. The specific evaluation methodology is as follows:

• WISD QM has established a Surveillance Activity Workbook (SAW) to capture quality surveillance data.

• For each applicable SLO/performance metric, WISD QM will determine whether the associated AQL has been met. WISD QM will base this determination on SLO data contained in Contractor deliverables, using the specification in Appendix A as the baseline definition of the SLO, as well as any additional data gathered by Government personnel. The Met — Not Met determination will be captured in the SAW. In addition, any quality issues or questions noted for the reporting period will be captured as comments in the SAW.

• Based on the evaluation results, the IRS may determine that additional review of Contractor performance is warranted. In this case, WISD QM will use available process documentation for In-depth Review to plan and conduct a “deep dive” analysis.

Results of this analysis will be provided to WISD management, the CO, and the COR before discussion with the Contractor.

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 8 March 2023

4. Required Performance Metrics Error! Reference source not found.Table 2-1 lists Contractor performance standards and identifies the surveillance methods that may be used to monitor performance against those standards. Although only a Primary Monitoring Approach(es) is (are) listed, the IRS reserves the right to engage, as necessary, in any of the monitoring approaches described in Section 3.2 to obtain full insight into Contractor performance.

Enterprise Integrated Portal Quality Assurance Surveillance Plan

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 9 March 2023

Table 4-1 Performance Requirement Summary (PRS)

PWS

Ref.

IEP Req.

ID

ELM ID

Service

Area in the PWS

Proposed Performance Measure/Standard

Acceptable

Quality

Level (AQL)

Contractor

Reporting

Document / Contractor

Reporting Frequency

Primary

Monitoring

Approach

Penalty for not meeting

AQL

3.1

PSPIS-SLO-

S.3.1.30 2565832

IEP Service and Program

Integration

Services

Successful publication of stakeholder satisfaction survey.

100.00%

• SLO

Compliance

Report

• Quarterly

Contractor

Self-reporting

Negative

CPARS and

Award Term impact

3.1

PSPIS-SLO-

S.3.1.31 2565833

IEP Service and Program

Integration

Services

Inclusion of all stakeholders (identified and approved by IRS) in stakeholder satisfaction survey.

100.00%

• SLO

Compliance

Report

• Quarterly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.1

PSPIS-SLO-

S.3.1.29 2565835

IEP Service and

Program Integration

Services

Portal Service and Program Integration

Services Reporting:

• Managed Service Provider reporting provides sufficient, accurate data for the Government to make informed decisions regarding programmatic, strategic, and financial planning.

• Managed Service Provider communicates, acknowledges, and resolves issues and/or complaints promptly.

• Financial reporting is accurate and enables the COTR and CO to monitor the contract and ensure that expenditures are kept within budget constraints.

90.00%

• SLO

Compliance

Report

• Annually

Contractor

Self-reporting;

Trend

Analysis

Award Term impact

3.1

PSPIS-SLO-

S.3.1.32

2565834

IEP Service and

Program Integration

Services

Communications Plan adequately describes Managed Service Provider's stakeholder engagement approach to ensure customer satisfaction.

100.00%

• SLO

Compliance Report

• Annually

Contractor delivered plan

Negative

CPARS and Award Term impact

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 10 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.1

PSPIS-SLO-

S.3.1.33 1370038

IEP Service and

Program Integration

Services

The Managed Service Provider routinely report Service Level Objectives metrics.

99.99%

• SLO

Compliance

Report

• Monthly

IRS

Inspection

Negative CPARS and

Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.3

2565836 Application

Hosting

Incident Management: Incident Response Time (Production) for:

• Priority 1 incidents (within 15 minutes); and

• Priority 2 incidents (within 1 hour), as per IRM 2.148.2

95.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.4

2565838 Application Hosting

Incident Management: Incident

Notification Time (Production):

• Priority 1 incidents (within 15 minutes); and

• Priority 2 incidents (within 1 hour), as per IRM 2.148.2

95.00%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.5

2565839 Application

Hosting

Incident Management: Incident Target Resolution Time (Production) for:

• Priority 1 incidents (within 4 hours);

and

• Priority 2 incidents (within 8 hours), as per IRM 2.148.2

95.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.9

2565840 Application

Hosting

Incident Management: Incident

Response Time (Production) for:

• Priority 3 incidents (within 4 hours)

• Priority 4 incidents (within 1 day), as per IRM 2.148.2

95.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.10

2565841 Application

Hosting

Incident Management: Incident

Notification Time (Production) for:

• Priority 3 incidents (within 3 hours)

• Priority 4 incidents (within 8 hours)

95.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative

CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 11 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.3.3

PWHS-

SLO-

S.3.3.3.11

2565842 Application

Hosting

Incident Management: Incident Target

Resolution Time (Production) for:

• Priority 3 incidents (within 16 hours)

• Priority 4 incidents (within 32 hours)

95.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.12

2565843 Application Hosting

Incident Management: Incident Response Time (Non-Production) for:

• Priority 3 incidents (within 8 hours)

• Priority 4 incidents (within 16 hours)

95.00%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.13

2565844 Application

Hosting

Incident Management: Incident

Notification Time (Non-Production) for:

• Priority 3 incidents (within 12 hours)

• Priority 4 incidents (within 24 hours)

95.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.3

PWHS-

SLO-

S.3.3.3.14

2565845 Application Hosting

Incident Management: Incident Target Resolution Time (Non-Production) for:

• Priority 3 incidents (within 3 days)

• Priority 4 incidents (within 5 days)

95.00%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 12 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.3.9

PWHS-

SLO-

S.3.3.8.8

2565846 Application

Hosting

Service Desk: Service Desk Responses for Non-Production activities are within the following timeframes:

a. New User Access – 24 hours

b. Password reset – 4 hours

c. General information – 5 business days – unless otherwise defined

d. New virtual environments provisioning – 5 business days – unless otherwise defined

e. Code promotion to different instance or environment – 2 business days – unless otherwise defined

f. Setting up a site such as GitLab clone, database import, configuration import, etc. – 2 business days – unless otherwise defined

g. Changing instance configuration - 2 business days - unless otherwise defined

h. Log files – continuous

i. Existing user log-in to assigned non-production environment 24x7

95.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.7

PWHS-

SLO-

S.3.3.6.7

2565847 Application

Hosting

Configuration Change Management:

Application Hosting Service applications changes successfully deployed as per schedule in the ticket

97.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

3% App

Hosting Service invoice and

Award Term impact

3.3.7

PWHS-

SLO-

S.3.3.6.8

2565848 Application

Hosting

Deployment of high priority patches is completed on schedule 97.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative

CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 13 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.3.7

PWHS-

SLO-

S.3.3.6.9

2565849 Application

Hosting

Applications maintain availability after patch deployment or rollback 99.99%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.7

PWHS-

SLO-

S.3.3.6.10

2565850 Application Hosting

Gap of time between a patch release and its deployment is less than 10 business days

97.00%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.3.7

PWHS-

SLO-

S.3.3.6.11

2565851 Application

Hosting

All patch planned vulnerabilities are addressed with planned deployment 99.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.2

PWHS-

SLO-

S.3.3.6.12

2565853 Application

Hosting

Capacity Utilization (Production) – Number of business days capacity utilization is in excess of 80% of the provisioned capacity

Less than 3 business days above

80%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.2

PWHS-

SLO-

S.3.3.6.13

2565855 Application Hosting

Capacity Utilization (Production) –

Number of business days required to adjust the provisioned capacity with target utilization.

TBD

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.3.2

PWHS-

SLO-

S.3.3.6.14

2565856 Application

Hosting

Capacity Utilization (Non-Production) - Number of business days capacity utilization is in excess of 80% of the provisioned capacity

Less than 3 business days above 80%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.2

PWHS-

SLO-

S.3.3.6.15

2565857 Application

Hosting

Capacity Utilization (Non-Production) – Number of business days required to adjust the provisioned capacity with target utilization.

TBD

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.3.1

PWHS-

SLO-

S.3.3.1.1

2565858 Application

Hosting

Application Hosting Support:

Application Hosting Service Availability to RUP and EUP application services as measured from end-user access environment. Inclusive of contractor maintenance activities

99.99%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

3% App Hosting

Services invoice and

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 14 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.3.1

PWHS-

SLO-

S.3.3.1.2

2565859 Application Hosting

Application Hosting Support:

Application Hosting Service Availability

(non-production) – Cumulative monthly availability of Managed Service Provider managed non-production environments based on infrastructure monitoring and reporting tools. Non-production environment includes but not limited to hardware, virtual machine, containers, software, tools to support, application development, monitoring, CI/CD and Agile

Collaboration Tools. This excludes planned down time such as patching.

Inclusive of contractor maintenance activities.

99.90%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.3.1

PWHS-

SLO-

S.3.3.1.3

2600632 Application

Hosting

System and Data backup and restoration: Monthly cumulative accuracy and validity of successful backup and restoration monitored by backup and restore application.

99.99%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting;

Backup and Restoration

Analysis

Report

Negative CPARS and

Award Term impact

3.3.4

PWHS-

SLO-

S.3.3.3.6

1370026 Application

Hosting

Problem Management: Root cause analysis for Priority 1 through Priority 4

Problem candidates completed within 30 business days of second incident

90.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.4

PSS-SLO-

S.3.4.5.10 2565861 Security

Security Vulnerability Remediation

Percentage of critical and high vulnerabilities remediated in the required timeframe as per IRS Internal

Revenue Manual

100.00%

• Security

Status and

Reports

• Monthly

Contractor Self-reporting;

Periodic

Inspection

Random Sampling

2% of Security invoice

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 15 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.4

PSS-SLO-

S.3.4.3.11

2565862 Security

Security POA&M Remediation –

Percentage of security deficiencies remediated and closed in Treasury FISMA Information Management

System (TFIMS) within the agreed to timeframe.

90%

• Security

Status and Reports

• Monthly

Contractor Self-reporting;

Trend Analysis

2% of

Security

3.4

PSS-SLO-

S.3.4.9 2565863 Security

Verify effective Security Operations are in place:

• Ensure all reporting is timely and accurate.

• Ensure all data feeds are maintained per requirements.

• Ensure activities are executed according to Security Services

Process plans and documents.

• All status reports and actions for the month are addressed and signed off by IRS IEP Authorizing Official or

Authorizing Official Designated Representative (AODR)

100.00%

• Security Status and

Reports

• Monthly

Contractor

Self-reporting;

Periodic Inspection

Random

Sampling

6% of Portal

Security invoice

3.5

PAS-SLO-

S.3.5.2.12 2565864

Application

Services

IRS.Gov & Microsite Website

Availability – Managed Service Provider as measured from end user perspective for: https://www.irs.gov and the microsites https://www.stayexempt.irs.gov, https://www.marketingexpress.irs.gov, and https://www.eitc.irs.gov for both Production and Non- Production environments

99.99%

• SLO

Compliance

Report

• Monthly

Contractor Self-reporting;

Trend

Analysis

6% of irs.gov invoice

3.5

PAS-SLO-

S.3.5.2.13 2565865

Application

Services

WCMS Availability – Monthly cumulative availability of WCMS as measured from end-user points of presence

99.99%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting;

Trend

Analysis

Negative

CPARS and

Award Term impact https://www.irs.gov/ https://www.stayexempt.irs.gov/ https://www.marketingexpress.irs.gov/ https://www.eitc.irs.gov/

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 16 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.5

PAS-SLO-

S.3.5.2.22 2600634

Application

Services

Non-Production System Response

Targets: application response time under 800ms – application response time as measured from the end users point of presence

99.00% overall.

95.00% of website responses handled under

500ms

• SLO

Compliance

Report

• Monthly

Contractor Self-reporting;

Customer Feedback, Trend Analysis

Negative CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.2.14

2565866 Application Services

Production System Response Targets:

application response time under 500ms

– website response time as measured from the end users point of presence

99.00% overall.

95.00% of website responses handled under 300ms

• SLO

Compliance Report

• Monthly

Contractor

Self-reporting;

Customer

Feedback, Trend

Analysis

Negative

CPARS and Award Term impact

3.5

PAS-SLO-

S.3.5.3.15 2565931

Application

Services

System Response Targets: search queries completed in 3 seconds or less 99.00%

• SLO

Compliance

Report

• Monthly

Contractor Self-reporting;

Trend

Analysis

Negative CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.2.24

2600635 Application Services

WCMS Search System Availability Targets: Monthly cumulative availability of Search Engine which supports WCMS as measured from end-user points of presence.

99.99%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting;

Trend

Analysis

Negative

CPARS and Award Term impact

3.5

PAS-SLO-

S.3.5.1.16 2565932

Application

Services

System Response Targets: irs.gov application transactions (user interaction with applications) completed in 3 seconds or less

95.00%

• SLO

Compliance

Report

• Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

6% of

Portal

Application Services

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 17 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.5

PAS-SLO-

S.3.5.2.17 2565933

Application

Services

WCMS: % Successful content updates.

100% of content updates (including user interfaces, templates and style sheets) to the Content Management

System must be publicly available on all access sites, with all stale instances flushed from any IEP Web hosting cache components, within 3 minutes or less of completion of the CMS "publish" task by CMS user if published via CMS workflow: CMS “publish” action if published via CMS manual publish; or

CMS publish date/time if published via scheduled publishing

99.99%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting;

Customer feedback, Trend Analysis, 100%

Inspection

Negative

CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.25 2565935

Application

Services

Application Development: All Managed

Service Provider developed applications are available for use after completion of the deployment and there is no outage attributed to the application development.

95.00%

SLO

Compliance

Report (During

Transition, As required for new products and services)

Test plans and reports are received timely and approved by the IRS

6% of Portal

Application Services

3.5

PAS-SLO-

S.3.5.26

2565938 Application Services

For applications developed and hosted by MSP, full compliance with the Section 508 and resolution of all

Section 508-related issues prior to deployment in Production in accordance with IRS policies and

Legislative mandates.

99.00%

SLO

Compliance Report (During

Transition, As required for new products and services)

Test plans and reports are received timely and approved by the IRS

Negative

CPARS and Award Term impact

3.5

PAS-SLO-

S.3.1.27 2566050

Application

Services

Implementation and Provision Services

Availability: IEP 2.0 IDIQ Filing Season application readiness

100.00% Annually

Contractor

Self-reporting;

Trend

Analysis

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 18 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.5

PAS-SLO-

S.3.1.28 2566052

Application

Services Availability of file transfer functionality 99.99%

• SLO

Compliance

Report

• Quarterly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.1.29

2566053 Application Services

Upload file round-trip response time within 1 second.

99.50%

• SLO

Compliance Report

• Quarterly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.5

PAS-SLO-

S.3.1.30 2566054

Application

Services

Bulk file upload throughput of at least

5 gigabits per second 99.50%

• SLO

Compliance

Report

• Quarterly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.1.31 2566055

Application

Services

File transfer success rate (file transfer success defined as file transferred without corruption).

99.50%

• SLO

Compliance

Report

• Quarterly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.1.32 2566057

Application

Services

File security (in transit and at rest upon completion) – All files in transit during upload and download and at rest upon completion of upload are protected from unauthorized or accidental access or exposure

100.00%

• SLO

Compliance

Report

• Quarterly

Contractor

Self-reporting

Negative

CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.27 2566059

Application

Services

For MSP developed applications, the Managed Service Provider resolved all

Section 508-related issues prior to deployment to Production in accordance with IRS policies and

Legislative mandates.

99.00%

• SLO

Compliance

Report

• Periodic, upon release

Contractor

Self-reporting

Negative

CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.28 2566061

Application

Services

IEP Application Availability - Application availability as measured from end-user points of presence.

99.99%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative

CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 19 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.5

PAS-SLO-

S.3.5.29 2566062

Application

Services

Successful and timely completion of

Filing Season updates of PUP applications developed and hosted by MSP. Filing Season critical applications include but not limited to: Earned Income Tax Credit (EITC) Assistant, Free File, Interactive Tax Assistant

(ITA), Link and Learn, Understanding Taxes, EITC Due Diligence Training, Sales Tax Deduction Calculator (STDC), and Tax Withholding Estimator (TWE).

100.00%

• SLO

Compliance

Report

• Annually

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.30 2566066

Application

Services

Web Analytics: Target fulfillment time to request for access to unsampled data - within 2 business hours

90.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative

CPARS and

Award Term impact

3.3

PAS-SLO-

S.3.5.31 2566067

Application

Services

Web Analytics: Concurrent IRS employee users having successful access to unsampled data - at least 25 users.

90.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.32

2566068 Application Services

Web Analytics: System response to request for sampled or unsampled data

- within 3 seconds

90.00%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.5

PAS-SLO-

S.3.5.33 2566069

Application

Services

Web Analytics: System response to request for generating complex reports within 5 seconds

90.00%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative

CPARS and

Award Term impact

3.5

PAS-SLO-

S.3.5.34

2566218 Application Services

Web Analytics: Rate of producing the requested data: (Total number of successfully completed requests for data in a calendar month / Total number of requests for data in a calendar month).

99.00%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 20 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.7

IEPTS-SLO-

S.3.7.1 2566070

Transition

Services

The final, IRS-approved IEP Services

Transition Plan (TP) is delivered within

2 months of IEP 2.0 contract award, and its status is provided monthly.

100.00% Contractor deliverable

IRS Review of deliverable

Negative

CPARS and

Award Term impact

3.7

IEPTS-SLO-

S.3.7.2

2566071 Transition Services

A plan to achieve Authority to Operate

(ATO) for IEP 2.0 infrastructure is delivered within 2 months of IEP 2.0 contract award, and its status is provided monthly.

100.00% Contractor deliverable

IRS Review of deliverable

Negative

CPARS and Award Term impact

3.7

IEPTS-SLO-

S.3.7.3 2566073

Transition

Services

All milestones established in the final, IRS-approved IEP Services Transition

Plan (TP) are delivered by the planned completion dates.

97.00% Contractor deliverable

IRS Review of deliverable

% off invoice that corresponds to x % of total systems due to transition that invoice period

3.7

IEPTS-SLO-

S.3.7.4 2566074

Transition

Services

All applications and platforms hosted previously in IEP 1.5 infrastructure are transitioned and operational completely and successfully to IEP 2.0 infrastructure by the planned completion date.

97.00% Contractor deliverable

IRS Review of deliverable

% off invoice that corresponds to x % of total systems due to transition that invoice period

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 21 March 2023

Area in the PWS

Proposed Performance

Measure/Standard

Acceptable Quality

Level (AQL)

Contractor Reporting

Document /

Contractor Reporting

Frequency

Primary Monitoring

Approach

Penalty for not meeting

AQL

3.7

IEPTS-SLO-

S.3.7.5

2566075 Transition Services

All incidents and problems opened during the transition period and assigned to the IEP 2.0 Contractor are resolved completely and approved for closure by IRS by the planned cutover date.

99.90% Contractor deliverable

IRS Review of deliverable

% off invoice that corresponds to x % of total systems due to transition that invoice period

3.7

IEPTS-SLO-

S.3.7.6 2566078

Transition

Services

The IRS-approved IEP Services Termination Plan is delivered by a delivery date as determined by IRS.

100.00% Contractor deliverable

IRS Review of deliverable

Negative CPARS and

Award Term impact

3.6

CWIS-SLO-

S.3.6.1 2566082

GFE Cloud

Services

Continuous availability of cloud hosting platform (production) to IRS cloud applications as measured from end-user access environment.

99.90%

• SLO

Compliance

Report

• Monthly

Contractor

Self-reporting

Negative

CPARS and

Award Term impact

3.6

CWIS-SLO-

S.3.6.2

2566093 GFE Cloud Services

Cumulative monthly availability of cloud hosting platform (non-production) to IRS cloud applications based on infrastructure monitoring and reporting tools. This excludes the planned downtime such as patching.

99.90%

• SLO

Compliance Report

• Monthly

Contractor Self-reporting

Negative

CPARS and Award Term impact

3.6

CWIS-SLO-

S.3.6.3 2566094

GFE Cloud

Services

A given IRS application workload, including all its non-production environments, is migrated and operational completely and successfully to the targeted IRS / IEP cloud environment(s) by the planned completion date.

100%

• SLO

Compliance

Report

• Monthly

• Contractor Self-reporting

• Trend Analysis

Negative CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 0.76 March 2023

5. Reporting Flow The QASP surveillance activity will be performed periodically as required by contract terms, and the results will be reported using the flow outlined in Figure 5-1 below.

Figure 5-1 Surveillance Reporting Flow

At the end of the evaluation period, WISD QM will use the data contained in the SAW to prepare a written report for the CO/COR, summarizing the overall results of the QA monitoring of the Contractor’s performance over the evaluation period. The report, documented in the format presented in Appendix B, will become part of the QA documentation for the contract.

WISD QM will submit the report to WISD Program Management, the CO, and the COR no later than two weeks after the end of the evaluation period; the data will be used as the technical input for fee determinations or achievement of other contract provisions, as appropriate. Feedback will be provided to the Contractor in accordance with the terms of the contract (I.5.1). Depending on the assessment contained in the report, the CO/COR may require the Contractor’s project manager, or a designated alternate, to meet with the CO/COR, WISD QM, and other Government personnel as deemed necessary to discuss the performance evaluation.

WISD QM, in addition to providing documentation to the CO/COR, will maintain a complete QA file. The file will contain copies of all data analyses, reports, evaluations, recommendations, and any actions related to the Government’s performance of the QA function, including the originals of all the SAWs. All such records will be retained for the life of the contract. WISD QM shall forward these records to the CO upon completion of the contract.

Appendix A: Metric Specifications Example

3.5.X Application Services Availability (Production)

Type: Availability

Category: Production

Surveillance Method:

Contractor self-reporting; Trend Analysis

Owner: WISD

SLO Description (IRS): IRS.Gov & Microsite Website Availability – Managed Service Provider as measured from end user perspective for:

https://www.irs.gov and the microsites https://www.stayexempt.irs.gov, https://www.marketingexpress.irs.gov, and https://www.eitc.irs.gov for both Production and lower environments".

Acceptable Quality Level

(IRS):

99.99%

SLO

Reporting Frequency:

- SLO

Compliance Report

- Monthly

Penalty for not meeting SLO:

Negative CPARS and Award Term impact

SLO Calculation (Proposed):

SLO met: >= 99.99%, Not met: < 99.99% Each environment measured individually and aggregated Total Number of Monthly Minutes – Total IEP Approved Maintenance Window Minutes – Total Number of Unplanned Outage Minutes / Total Number of Monthly Minutes - IEP Approved Maintenance Windows Minutes)

SLO Calculation (Negotiated):

Base Measures (Contractor):

Recovery:

Other Notes:

History:

https://www.irs.gov/ https://www.stayexempt.irs.gov/ https://www.marketingexpress.irs.gov/ https://www.eitc.irs.gov/

Appendix B: Forms

Surveillance Review Report

To: IRS CO, Vendor’s Name

IRS COR, Vendor’s Name

From: IRS WISD Quality Manager

Re: Surveillance Monitoring

Contract No:

The following report is provided as formal written feedback of the results of surveillance monitoring activities that were recently conducted by the IRS Web Infrastructure Services Division.

Surveillance Period _______________

Performance Period ________________

ID SLO Description AQL Data

Source Weight

Actual Results

Evaluation Score Comments

S.3.6.4

Percentage of the chat requests answered within 40 seconds

80.00% SLO 0 0

S.3.5.4 Search queries completed in 3 seconds or less

99.00% SLO 0 0

S.3.4.3 Security POA&M Remediation

90.00% SLO 0 0

Summary Comments: Summary Performance Results:

Total Possible Score: 0

Total Actual Score: 0

Total % Weighted SLOs Met: 0%

Signed:

IRS WISD Quality Manager

IRS DMO Manager

Appendix C: References

• A Guide to Best Practices for Performance-based Services Contracting, Office of

Federal Procurement Policy (OFPP) in the Office of Management and Budget (OMB), Final Edition, October 1998.

• United States Department of Agriculture International Technology Services Quality Assurance Surveillance Plan. 2014. ITS Hardware BPA Category 2 Laptops and Notebooks, Sample 1, Attachment A-1 (accessed on June 27, 2022 at QASP -Sample 1.pdf (usda.gov) )

• Federal Acquisition Regulation (Accessed on June 27, 2022, from the Acquistion.Gov website at https://www.acquisition.gov/far/part-46#FAR_46_102 and Regulations search | Acquisition.GOV .1106 Reporting Requirements)

• The following tables outline the Incident Impact, Prioritization and Urgency definitions as stated in the IRM in section 2.148.2.5.3 (03-30-2016) Exhibit C: Priorities (http://irm.web.irs.gov/Part2/Chapter148/Section2/IRM2.148.2.asp)

Impact: The degree of business disruption IRS Incident Impact Criteria

Impact Description

(Enterprise) - Widespread business disruption; Large number of business units are unable to operate

(Site) - Entire facility or department; Critical business units and/or an entire facility/location is unable to operate

3 (Multiple Users) - More than 1 user or department; Multiple individuals are unable to operate

4…