Section_J-_Section_C.10_Attachment_IEP1.5_QASP_v2_20160807-V1.pdf

Internal Revenue Service

Solicitation Number: TIRNO-16-R-00026

IRS Integrated Enterprise Portal 1.5 (IEP 1.5)

Enterprise Integrated Portal Performance Management

Quality Assurance Surveillance Plan (QASP)

Version 1.2

July 28, 2016

SECTION J- ATTACHMENT C.10

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 2 July 28, 2016

Portal Performance Management Program

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 i July 28, 2016

Review and Approval This Quality Assurance Surveillance Plan (QASP) documents how the Internal Revenue Service

(IRS) will monitor Service Level Objectives (SLO) performance under selected Managed

Services contracts. The IRS Enterprise Technology Implementation (ETI) Office has established a multi-tiered framework to support Integrated Enterprise Portal (IEP) 1.5 contract performance surveillance. As part of this framework, IEP 1.5 Quality Management (QM) will utilize

Contractor-supplied data to monitor performance periodically and will provide the results of this surveillance to the Contractor. The IRS ETI IEP 1.5 will augment this process with in-depth reviews (audits), as necessary.

Review of this QASP signifies formal notification of the periodic surveillance audit to monitor

IEP 1.5 Contractor technical performance. If an in-depth review (audit) is necessary, the IRS will issue separate formal notification to the Contractor of its intent to perform the audit, no later than 30 calendar days before it is scheduled to begin. All material support on part to assist the

IRS in its audit or in-depth deep dive reviews would be treated as a Project Change Request.

Review and Approval:

The following stakeholders have approved this IEP 1.5 Quality Assurance Surveillance Plan, v1.0:

ETI Director Date

ETI Quality Management Date

IEP 1.5 Contracting Officer Date

IEP 1.5 Contracting Officer Representative Date

The following stakeholders have reviewed and concur with this IEP 1.5 Quality Assurance

Surveillance Plan:

Contractor Program Manager Date

Contractor Quality Control Date

Enterprise Technology Implementation Performance Management Program

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 ii July 28, 2016

Table of Contents

Review and Approval .................................................................................................................... i

Table of Contents .......................................................................................................................... ii

Table of Tables ............................................................................................................................. iii

Table of Figures ........................................................................................................................... iii

1. Introduction

1.1 Scope

1.2 Authority

1.3 Goals and Objectives

1.4 QASP Update

2. Roles and Responsibilities

2.1 Internal Revenue Service

2.2 Contractor

3. Approach

3.1 Overall Approach

3.2 Surveillance Methods

3.3 Evaluation Methodology

4. Required Performance Metrics

5. Reporting Flow

Appendix A: Metric Specifications Example

Appendix B: Forms

Appendix C: References

Appendix D: Acronyms

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 iii July 28, 2016

Table of Tables

Table 2-1 Government Roles and Responsibilities

Table 2-2 Contractor Roles and Responsibilities

Table 4-1 Required Performance Metrics

Table of Figures

Figure 5-1 Surveillance Reporting Flow

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 1 July 28, 2016

1. Introduction

This Quality Assurance Surveillance Plan (QASP) documents the approach the Internal Revenue

Service (IRS) will take to monitor portal Contractor technical performance required by the

Performance Work Statement in effect at the time of the surveillance activity.

1.1 Scope

This QASP documents how the IRS will monitor Contractor Performance Service Level

Objective (SLO) performance under this contract. The QASP will be tailored to the characteristics and requirements of the particular contract by the addition of the germane

Surveillance Activity Table in Section 4 of this document. SLOs not contained in this document will not be considered to be within scope of the QASP implementation.

The QASP is a Government document used to monitor the Contractor’s Quality Assurance

Program (QAP), which the Contractor must have in place, or shall establish prior to contract award, to attain, sustain and advance exceptional delivery of services to the IRS. The

Contractor’s QAP will include its description of people, tools and processes that it will use to ensure compliance with all contract requirements and its proposed methods for providing continuous improvement in service delivery. The Contractor’s QAP is expected to include a detailed set of plans (for example, Quality Control (QC) Plan, Configuration Management and

Change Control Plan, Asset Management Plan, Risk Management Plan and/or other plans at the

Contractor’s discretion) that the Contractor will use to manage its execution of the Portal

Services program.

The QASP does not define or dictate how the Contractor should accomplish the work required under contract provisions; instead, the QASP measures the degree of achievement of the

Contractor’s execution of its management and QC actions necessary to meet the performance standards documented herein. Thus, the internal continuous performance monitoring performed by the Contractor is not part of the QASP implementation activity, but does yield data that are leveraged during QASP activities.

The QASP can be changed or updated. It is intended to be a “living” document that should be revised or modified as circumstances warrant. Either the Contractor or the Government may initiate changes to the QASP at any time during contract performance; changes to the QASP will require bilateral approval. Such changes shall not entitle the Contractor to any equitable adjustments or to any other compensation for performance in a prior period.

Some of the data developed during QASP activities may be used in other Enterprise Technology

Implementation (ETI) measurement activities. The descriptions of those other measurement activities are contained in documents external to this QASP.

1.2 Authority

Authority for issuance of this QASP is the Federal Acquisition Regulation (FAR) Subparts 37.6, 46.1, and 46.4, as well as contract Section E – Inspection and Acceptance, which provides for inspections and acceptance of the work and services called for in service contracts, to be executed by the Contracting Officer (CO) or a duly authorized representative.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 2 July 28, 2016

1.3 Goals and Objectives

The QASP activity is a critical component for Contractor performance measurement. This

QASP will be implemented by the ETI Quality Management (QM) function.

1.3.1 Goal

The overall goal of the QASP activity is to confirm that the IEP 1.5 Contractor’s delivery and execution of products or services meets the technical performance measures, or SLOs, included in the contract and documented in this QASP.

1.3.2 Objectives

Specific QASP objectives include:

Determine Contractor performance is meeting all contract SLOs. Contractor performance data may be gather through audits, examinations, customer feedback and measurements gathered from reports.

Provide data to the CO and COTR to assess performance and other contract provisions, as appropriate.

Identify portal program risks arising from Contractor performance in meeting contract

SLOs.

Identify Contractor corrective action(s) needed to meet SLOs or to address identified risks or issues.

1.4 QASP Update

This Plan will be updated periodically to reflect the results of a joint review, by IRS and

Contractor personnel, of SLOs, SLO definitions, and SLO monitoring approaches.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 3 July 28, 2016

2. Roles and Responsibilities

The IRS bears primary responsibility for implementing this QASP, with support from the

Contractor.

2.1 Internal Revenue Service

Table 2-1 describes Government roles and responsibilities regarding QASP implementation.

Table 2-1 Government Roles and Responsibilities

Role Responsibilities

ETI Management Establishes and approves the QASP

Provides resources to implement the QASP

Determines achievement of other contract provisions

Analyzes Quality Assurance Surveillance Review (QASR) and similar feedback and findings, as appropriate

ETI Quality Management

(QM)

Conducts QASR

Interviews business customer(s)

Reviews Contractor-provided reports

Conducts liaison with Contracting Officer

Representative(s) (COR(s))

Documents findings

Provides formal written feedback, including findings

Participates in project status reviews, as necessary

Performs in-depth quality review activities at Contractor location, as necessary and if authorized

Debriefs Contractor (with CO and/or COR) on the surveillance review results, as necessary

Contracting Officer Monitors contract compliance, contract administration, and cost control

Reviews QASR and similar feedback and findings, as appropriate

Provides contractual direction

Debriefs Contractor (with ETI QM and/or COR) on the surveillance review results, as necessary

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 4 July 28, 2016

Role Responsibilities

Contracting Officer’s

Representative Chairs Contractor deliverables review, Project Status

Reviews (PSRs), and similar performance and status reviews with IRS stakeholders and Contractor

Reviews QASR and similar feedback and findings, as appropriate

Forwards QASR and similar reports to appropriate business customer (Online Services (OLS), etc.)

Debriefs Contractor (with CO and/or ETI QM) on surveillance review results, as necessary

2.2 Contractor

Table 2-2 describes Contractor roles and responsibilities regarding QASP implementation.

Table 2-2 Contractor Roles and Responsibilities

Role Responsibilities

Contractor Program Manager Reviews and concurs with the QASP

Collects, analyzes, documents, and produces performance data

Submits data reports to CO/COR

Supports IRS QASRs and similar activities, as necessary

Identifies and implements corrective actions

Submits requests to COR for new reports or measures

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 5 July 28, 2016

3. Approach

ETI has established a multi-tiered framework to facilitate quality assurance (QA) monitoring of portal contractor performance. ETI QM will leverage Contractor-supplied data to implement this framework. The following sections describe this strategy in more detail.

3.1 Overall Approach

The ETI QA monitoring framework described in this QASP integrates and leverages ongoing monitoring activities. ETI QM performs two types of audits, QA surveillance reviews on a regular basis and in-depth reviews (“deep dives”) on an exception basis, when implementing the

QA monitoring framework. These surveillance reviews shall not unduly burden the Contractor and shall comply with FAR 52.246-4 Inspection of Services – Fixed-Price. The Contracting

Officer must be notified if the Contractor believes any action may affect contract price, terms, or conditions.

ETI QM’s basic approach is to: establish an agreed and understood quality baseline; monitor

Contractor performance against that baseline; identify any performance gaps on a timely basis;

and confirm that corrective actions are applied in a timely fashion.

ETI QM staff and other interested IRS stakeholders meet periodically with Contractor program personnel to review SLOs in the QASP and the monitoring and reporting systems to enable continuous and accurate intake, analysis, and reporting of performance data.

The Contractor implements continuous performance monitoring as agreed during the QA

Program Baseline Review (PBR), uses internal Quality Control (QC) resources to meet

SLO performance thresholds defined by the Acceptable Quality Levels (AQLs), and submits deliverables periodically to report status against the quality baseline. Although these monitoring activities themselves are not included in the scope of the IRS’ QA surveillance, they yield measurement data and deliverables that are assessed during

QASP implementation.

The IRS COR holds Project Status Reviews (PSR) with other relevant IRS stakeholders, including ETI QM and internal IRS SLO “owners”, to assess SLO status against the established baseline, identify emerging risks and issues, and determine potential “deep dive” opportunities.

The ETI QM function audits Contractor performance by, at a minimum, checking and synthesizing reported SLO data to develop quality surveillance feedback, including findings and potential corrective action requests, for submission to the CO/COR. These audit results are submitted using the form in Appendix B.

The IRS reserves the right to perform in-depth reviews of Contractor performance (“deep dives”) as necessary to assess perceived or actual performance issues. These in-depth reviews may occur at any time during the evaluation period and may address performance issues arising in the current or any prior evaluation period. If an in-depth review (audit) is necessary, the IRS will issue separate formal notification to the

Contractor of its intent to perform the review, no later than 30 calendar days before it is

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 6 July 28, 2016 scheduled to begin. To support such deep dives, the IRS may request additional data from the Contractor, including but not limited to raw data, audit results, evidence of certification, QC program data, etc. The IRS reserves the right to perform these deep dives at Contractor or subcontractor facilities.

3.2 Surveillance Methods

ETI QM has at its disposal several methods with which to perform the QA monitoring function described above:

Contractor Self-Reporting. Uses contractor-supplied performance data collected by tracking and monitoring tools and/or contractor status reports that identify problems, issues and/or risks. (See Appendix C, Reference number 3.)

Trend Analysis. Per the US Census Bureau: “This method should be used regularly and continually to monitor the contractor’s ongoing performance over time. Data for tracking trends can be gathered from all other evaluation sources and methods, [and used to populate] a database upon which the trend analysis is based. This database should be created and maintained by government personnel. The contractor’s own metrics may provide most of the information needed for the analysis.” (See Appendix C, Reference number 1.)

Quantitative Analysis. This refers to the use of statistical and other quantitative techniques to analyze raw performance data to characterize performance and/or identify trends, causal relationships, or root causes of performance issues. Examples would include analyzing results of customer survey and problem tickets against relevant SLOs.

This method is appropriate for in-depth analysis of areas of interest.

Periodic Inspection. Per the Office of Management and Budget: “This method, sometimes called ‘planned sampling,’ consists of the evaluation of tasks selected on other than a 100 percent or random basis. It may be appropriate for tasks that occur infrequently, and where 100 percent inspection is neither required nor practicable. A predetermined plan for inspecting part of the work is established using subjective judgment and analysis of agency resources to decide what work to inspect and how frequently to inspect it.” (See Appendix C, Reference number 2.)

Random Sampling. Per the Office of Management and Budget: “This is usually the most appropriate method for recurring tasks. With random sampling, services are sampled to determine if the level of performance is acceptable. Random sampling works best when the number of instances of the services being performed is very large and a statistically valid sample can be obtained.” (See Appendix C, Reference numbers 2, 3.)

100 Percent Inspection. Per the Office of Management and Budget: “This is usually only the most appropriate method for infrequent tasks or tasks with stringent performance requirements, e.g., where safety or health is a concern. With this method, performance is inspected/evaluated at each occurrence. One hundred percent inspection is too expensive to be used in most cases.” (See Appendix C, Reference numbers 2, 3.)

Customer/User/Stakeholder Feedback. This method is an independent evaluation of an activity or process to assess compliance with SLOs. This surveillance method

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 7 July 28, 2016 supplements the other surveillance methods. Customer Feedback may be necessary for certain types of tasks that do not lend themselves to random sampling of 100% inspection. Customer feedback may be obtained either from the results of formal customer satisfaction surveys or from random customer complaints or complements.

Customer complaints, to be considered valid, must set forth clearly and in writing the detailed nature of the complaint, must be signed or emailed and forwarded to the PM or

COTR who will assess the validity of the complaint and retain it in a documentation file.

3.3 Evaluation Methodology

ETI QM will use the cited observation methods to determine whether AQLs have been met over the evaluation time frame. The evaluation period will be annually. The specific evaluation methodology is as follows:

ETI QM has established a Surveillance Activity Workbook (SAW), maintained within the IEP, to capture quality surveillance data.

For each applicable SLO/performance metric, ETI QM will determine whether the associated AQL has been met. ETI QM will base this determination on SLO data contained in Contractor deliverables, using the specification in Appendix A as the baseline definition of the SLO, as well as any additional data gathered by Government personnel. The Met — Not Met determination will be captured in the SAW. In addition, any quality issues or questions noted for the reporting period will be captured as comments in the SAW.

Based on the evaluation results, the IRS may determine that additional review of

Contractor performance is warranted. In this case, ETI QM will use Standard Operating

Procedures for In-depth Review to plan and conduct a “deep dive” analysis. Results of this analysis will be provided to ETI management, the CO, and the COR before discussion with the Contractor.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 8 July 28, 2016

4. Required Performance Metrics

Table 4-1 lists Contractor performance standards and identifies the surveillance methods that may be used to monitor performance against those standards. Although only a Primary

Monitoring Approach(es) is (are) listed, the IRS reserves the right to engage, as necessary, in any of the monitoring approaches described in Section 3.2 to obtain full insight into Contractor performance.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 9 July 28, 2016

Table 4-1 Required Performance Metrics

Cross- Reference

Performance Standard Acceptable

Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

C.3.3.1 Web Hosting Service Availability (production) - Availability to application services as measured from end-user access environment

99.99% SLO Compliance Report

Monthly

Contractor Self-reporting, Customer feedback;

Trend Analysis

6% of Web Hosting invoice

C.3.3.1 Web Hosting Service Availability (non-production) – Contractor proposed measure

99.9% SLO Compliance Report

Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3

Incident Response Time (production) - P1 incident time to respond shall be 30 min or less

P2 incident time to respond shall be 2 hours or less

95% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3 Incident Notification Time – defined as the Amount of time between when the contractor becomes aware of an incident and when the contractor notifies the IRS.

Phone notification – (EOps Service Desk)

E-mail notification (ETI; business owner): P1: 30 mins

Phone notification - (EOps Service Desk)

E-mail notification (ETI; business owner): P2: 2 hours

95% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3 Incident Resolution Time (production)

P1 Incident resolution time shall be within 4 hours

P2 Incident resolution time shall be within 48 hours

95% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3 Problem Root Cause Resolution – 95% SLO Compliance Report Contractor Self-reporting; Negative CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 10 July 28, 2016

Cross- Reference

Performance Standard Acceptable

Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

Priority 1 & Priority 2 Problems and problem candidates must have a root cause discovered, replicated and resolved within 30 calendar days and communicated to

IRS

Monthly Trend Analysis Award Term impact

C.3.3.6 Portal Web Hosting Services - Changes successfully deployed

Percentage based on number of changes deployed within agreed upon time schedule in ticket/total number of changes scheduled for the month)*100

No outage or issue attributed to the web hosting service or integration point after implemented change:

Percentage based on changes validated and confirmed by implementer or requestor/total number of changes deployed

95% SLO Compliance Report

Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

6% of Portal Web Hosting

Services invoice

C.3.3.1;

C.3.3.8

Portal Web Hosting Services –

Service Request Response Time (non-production)

a) New User Access – 24 hours

b) Password reset – 4 hours

c) General information - 5 days - unless otherwise defined

d) New virtual environments provisioning - 5 days -unless otherwise defined

e) Log files - continuous

f) Existing user log-in to assigned non-production environment 24x7

95% SLO Compliance Report

Monthly

Contractor Self-reporting, Customer feedback;

Trend Analysis

Negative CPARS and Award Term impact

C.3.4 Verify effective Security Operations are in place for IEP 1.5

100% Security Status and Reports

Contractor Self-reporting;

Periodic Inspection

6% of Portal Security

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 11 July 28, 2016

Reference

Performance Standard Acceptable

Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

Ensure all reporting is timely and accurate.

Ensure all data feeds are maintained per requirements.

Ensure activities are executed according to Security Services Process plans and documents.

All status reports and actions for the month are addressed and signed off by IRS IEP Authorizing Official or Authorizing Official Designated Representative

(AODR)

Monthly Random Sampling invoice

C.3.4.5 Maintaining security configuration baseline

Demonstrate a high-level of compliance to approved configuration baseline. The list of configuration items (software and devices) is maintained in the Security Configuration and Change Management Plan.

Number of configuration items with a compliance score of at least 90% / Number of configuration items

90%

Security Status and Reports

Monthly

Contractor Self-reporting;

Periodic Inspection

Random Sampling

2% of Security invoice

C.3.4.3 Security POA&M Remediation 90%

Security Status and Reports

Monthly

Contractor Self-reporting;

Trend Analysis

2% of Security invoice

C.3.5.2 irs.gov Website Availability – Contractor proposed measure

99.99% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

6% of irs.gov invoice

C.3.5.2 CMS Availability - Monthly cumulative availability as measured based on P1 tickets

99.99% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.5.2 System Response Targets – website response time

User requests for Website content from a user page to

95% SLO Compliance Report Contractor Self-reporting;

Customer Feedback, Negative

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 12 July 28, 2016

Reference

Performance Standard Acceptable

Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

user page served completed in 3 seconds or less Monthly Trend Analysis Award Term impact

C.3.5.3 System Response Targets – search queries completed in 3 seconds or less

99% SLO Compliance Report

Monthly Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.5.1 System Response Targets – irs.gov application transactions (user interaction with applications) completed in 5 seconds or less

95% SLO Compliance Report

Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

6% of Portal Application Services invoice

C.3.5.2 WCMS - % Successful content updates.

100% of content updates (including user interfaces, templates and style sheets) to the Content Management System must be publicly available on all access sites, with all stale instances flushed from any IEP Web hosting cache components, within 30 minutes or less of completion of the CMS "publish" task by CMS user if published via CMS workflow: CMS “publish” action if published via CMS manual publish; or CMS publish date/time if published via scheduled publishing.

99.99%

SLO Compliance Report

Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

Negative CPARS and Award Term impact

C.3.6 CSR Call Back timeliness

Percentage of CSR returned calls to the user (1 attempt) within 24 business hours. CSRs cannot call a user back after 9pm (users’ local time).

95% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.6 Help Desk Response time – Phone Calls

Percentage of calls answered within 3 rings or 40 seconds (calls not answered within 3 rings will be provided the option of being forwarded to an automatic

80% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

1% of Help Desk invoice

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 13 July 28, 2016

Reference

Performance Standard Acceptable

Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

voice mail service or continuing to hold).

C.3.6 Help Desk Response time e-mails

Percentage of e-mails responded to within 24 business hours with scripted e-mail response.

Percentage of e-mails that receive an auto reply response within 1 hour

100%

SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

1% of Help Desk invoice

C.3.6 Help Desk Response time – chats

Percentage of the chat requests answered within 40 seconds.

80% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

1% of Help Desk invoice

C.3.6 Call Abandonment rate

Abandoned Calls: Not to exceed 5% <5% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

1% of Help Desk invoice

C.3.6 Portal Website Help Desk Services - Voice-mail system response

Percentage of voice-mail messages left in system during business hours retrieved and responded to within 1 hour of receipt.

Percentage of voice-mail messages left in system during non-business hours retrieved and responded to within next 8 business hours

100%

SLO Compliance Report

Monthly

Random sampling; Trend Analysis

Negative CPARS and Award Term impact

C.3.6 Portal Website Help Desk Services - Changes in required

Help Desk staff

Percentage of help desk staffing level change requests fulfilled within 3 weeks of notification

100% SLO Compliance Report

Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.1 Successful completion of the Section 508 test plans and completed test reports for additional products and 100% SLO Compliance Report Test plans and resports are received timely and Negative

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 14 July 28, 2016

Reference

Performance Standard Acceptable

Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

services or enhancements that are planned for implementation into the new Website environment.

During Transition

As required for new products and services approved by the IRS. Award Term impact

C.3.1 Information critical to the implementation and provision of services is available on a timely basis for all internal and external customers, stakeholders

100% Annually Contractor Self-reporting;

Trend Analysis

Award Term impact

C.3.1 High Levels of customer satisfaction with applicable information technology services 90% or better of customers survived rate services as satisfactory

Annually

Contractor Self-reporting;

Trend Analysis

Award Term impact

C.3.1 Contractor reporting provides sufficient, accurate data for the Government to make informed decisions regarding programmatic, strategic, and financial planning

Contractor provides prompt communication of, acknowledgements and resolution to issues and/or complaints

Financial reporting is accurate and enables the COTR and CO to monitor the contract and ensure that expenditures are kept within budget constraints.

Contractor maintains a high level of responsiveness to IRS requirements;

demonstrates reliability, completeness and overall quality of the execution of the tasks stipulated in the contract

Annually

Contractor Self-reporting;

Trend Analysis

Award Term impact

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 15 July 28, 2016

5. Reporting Flow

The QASP surveillance activity will be performed periodically as required by contract terms, and the results will be reported using the flow outlined in Figure 5-1 below.

Figure 5-1 Surveillance Reporting Flow

At the end of the evaluation period, ETI QM will use the data contained in the SAW to prepare a written report for the CO/COR, summarizing the overall results of the QA monitoring of the Contractor’s performance over the evaluation period. The report, documented in the format presented in Appendix B, will become part of the QA documentation for the contract.

ETI QM will submit the report to ETI Program Management, the CO, and the COR no later than two weeks after the end of the evaluation period; the data will be used as the technical input for fee determinations or achievement of other contract provisions, as appropriate. Feedback will be provided to the Contractor in accordance with the terms of the contract (I.5.1). Depending on the assessment contained in the report, the CO/COR may require the Contractor’s project manager, or a designated alternate, to meet with the CO/COR, ETI QM, and other Government personnel as deemed necessary to discuss the performance evaluation.

ETI QM, in addition to providing documentation to the CO/COR, will maintain a complete QA file. The file will contain copies of all data analyses, reports, evaluations, recommendations, and any actions related to the Government’s performance of the QA function, including the originals of all the SAWs. All such records will be retained for the life of the contract. ETI QM shall forward these records to the CO upon completion of the contract.

ETI QM

Conduct Regular

Surveillance Reviews

ETI QM

Conduct in-depth Reviews, as necessary (“Deep Dive”)

ETI QM Develops and Delivers the Audit Reports to ETI Management

CO/COR

ETI’s Management and CO/COR Reviews Audit

Report

CO/COR

Provides

Feedback to

ETI QM

Debriefs

Contractor, as necessary with

CO/COR

Provides

Feedback to

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 16 July 28, 2016

Appendix A: Metric Specifications Example

D.x.x.x All Task Orders: Deliverables

Type: Technical

Category: Deliverables

Surveillance Method: IRS 100% Inspection (each Deliverable)

Owner: PPM

SLO Description (IRS):

This measurement description applies to all QASP metrics for any Deliverable measured for the Start-up, Build-out or Transition Phases.

Acceptable Quality Level (IRS):

100%

SLO Reporting Frequency

(IRS):

Monthly

SLO Calculation (Contractor):

Deliverable met = 100%, Not met = 0% Base Measures (Contractor):

Deliverable Met

Description: Deliverable accepted by IRS in a timely manner per schedule Data Summarization: N/A. Each Deliverable is measured individually.

Reporting Shift: N/A Reporting & Retention: Reported monthly and kept in archive for 3 years Exceptions: If Contractor initially submits the deliverable on schedule and IRS requests changes during the initial review cycle, Contractor will be credited with meeting the due date if IRS accepts the deliverable upon the follow-on (second) review cycle.

Data Source: IRS Deliverable Tracking Matrix Data Verification: IRS Deliverable Manager and COR verify completion of Deliverables

Recovery:

N/A

Other Notes:

History:

<mm/dd/yyyy>: Baseline description approved by TBD

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 17 July 28, 2016

Appendix B: Forms

Surveillance Review Report

To: IRS CO, Vendor’s Name

IRS COR, Vendor’s Name

From: IRS ETI Quality Manager

Re: Surveillance Monitoring

Contract No:

The following report is provided as formal written feedback of the results of surveillance monitoring activities that were recently conducted by the IRS Enterprise Technology Implementation Office.

Surveillance

Period _______________

Performance

Period ________________

ID SLO Description AQL

Data

Source Weight

Actual

Results Evaluation Score Comments

S.3.6.4 Percentage of the chat requests answered within

40 seconds

80.00% SLO 0 0

S.3.5.4 Search queries completed in 3 seconds or less

99.00% SLO 0 0

S.3.4.3 Security POA&M

Remediation

90.00% SLO 0 0

Summary Comments: Summary Performance Results:

Total Possible Score: 0

Total Actual Score: 0

Total % Weighted SLOs Met: 0%

Signed:

IRS ETI Quality

Manager

IRS DMO Manager

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 18 July 28, 2016

Appendix C: References

1) US Census Bureau, Attachment 2: Performance-based Statements of Work Guidance for

Using a Quality Assurance Plan (accessed on June 24, 2010 at www.census.gov/procur/www/fssp/att02.html)

2) A Guide to Best Practices for Performance-based Services Contracting, Office of Federal

Procurement Policy (OFPP) in the Office of Management and Budget (OMB), Final Edition, October 1998

3) United States Department of Agriculture Foreign Agricultural Service, Infrastructure

Management Quality Assurance Surveillance Plan, Version V0.4, November 2008 (accessed on June 24, 2010 from the “Seven Steps to Performance-based Acquisition” website at https://www.acquisition.gov/comp/seven_steps/library3.html. See Step 5.)

http://www.census.gov/procur/www/fssp/att02.html https://www.acquisition.gov/comp/seven_steps/library3.html

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 19 July 28, 2016

Appendix D: Acronyms

Acronym Definition

AQL Acceptable Quality Level

CO Contracting Officer

COR Contracting Officer’s Representative

CPARS Contractor Performance Assessment Reporting System

OLS Online Services

FAR Federal Acquisition Regulation

FY Fiscal Year

IRS Internal Revenue Service

N/A Not Applicable

IEP Integrated Enterprise Portals

NTP Network Time Protocol

OFPP Office of Federal Procurement Policy

OMB Office of Management and Budget

PMMP (ETI) Performance Monitoring Management Plan

ETI Enterprise Technology Implementation Office

ETIPMP Enterprise Technology Implementation Performance Management Program

PSR Project Status Review

PWS Performance Work Statement

QA Quality Assurance

QA PBR QA Program Baseline Review

QASP Quality Assurance Surveillance Plan

Quality Assurance Surveillance Plan (QASP) ■ Version 1.2 20 July 28, 2016

Acronym Definition

QASR Quality Assurance Surveillance Review

QC Quality Control

QM Quality Management

RUP Registered User Portal

SAW Surveillance Audit Workbook

SLO Service Level Objective

TBD To Be Determined

VPN Virtual Private Network