SECT_J_ATTACHMENT-DRAFT_QASP_6162016.pdf

SECTION J: ATTACHMENT ??

Internal Revenue Service

Enterprise Integrated Portal Performance Management

Quality Assurance Surveillance Plan (QASP)

A

Version 1.1

May 26, 2016

Portal Performance Management Program

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA i May 26, 2016

Review and Approval This Quality Assurance Surveillance Plan (QASP) documents how the Internal Revenue Service (IRS) will monitor Service Level Objectives (SLO) performance under selected Managed Services contracts. The IRS Enterprise Technology Implementation (ETI) Office has established a multi-tiered framework to support Integrated Enterprise Portal (IEP) 1.5 contract performance surveillance. As part of this framework, IEP 1.5 Quality Management (QM) will utilize Contractor-supplied data to monitor performance periodically and will provide the results of this surveillance to the Contractor. The IRS ETI IEP 1.5 will augment this process with in-depth reviews (audits), as necessary.

Review of this QASP signifies formal notification of the periodic surveillance audit to monitor IEP 1.5 Contractor technical performance. If an in-depth review (audit) is necessary, the IRS will issue separate formal notification to the Contractor of its intent to perform the audit, no later than 30 calendar days before it is scheduled to begin. All material support on part to assist the IRS in its audit or in-depth deep dive reviews would be treated as a Project Change Request.

Review and Approval:

The following stakeholders have approved this IEP 1.5 Quality Assurance Surveillance Plan, v1.0:

ETI Director Date

ETI Quality Management Date

IEP 1.5 Contracting Officer Date

IEP 1.5 Contracting Officer Representative Date

The following stakeholders have reviewed and concur with this IEP 1.5 Quality Assurance Surveillance Plan:

Contractor Program Manager Date

Contractor Quality Control Date

Enterprise Technology Implementation Performance Management Program

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA ii May 26, 2016

Table of Contents Review and Approval .................................................................................................................... i Table of Contents .......................................................................................................................... ii Table of Tables ............................................................................................................................. iii Table of Figures ........................................................................................................................... iii Record of Changes ....................................................................................................................... iv Record of Changes ....................................................................................................................... iv Notes for Using the QASP ........................................................................................................... iv

1. Introduction

1.1 Scope

1.2 Authority

1.3 Goals and Objectives

1.4 QASP Update

2. Roles and Responsibilities

2.1 Internal Revenue Service

2.2 Contractor

3. Approach

3.1 Overall Approach

3.2 Surveillance Methods

3.3 Evaluation Methodology

4. Required Performance Metrics

5. Reporting Flow Appendix A: Metric Specifications Appendix B: Forms Appendix C: References Appendix D: Acronyms

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA iii May 26, 2016

Table of Tables Table 2-1 Government Roles and Responsibilities Table 2-2 Contractor Roles and Responsibilities Table 4-1 Required Performance Metrics

Table of Figures Figure 5-1 Surveillance Reporting Flow

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA iv May 26, 2016

Record of Changes Date

Reference (Table, Figure, or

Paragraph)

A=Add M=Modify D=Delete

Brief Description of Change

05/26/2016 1.1 Scope, pg 1 M Change scope description to clarify QASP relationship to contractor Quality Assurance Program

05/26/2016 Table 4-1, Required Performance Metrics

M Updated SLO references to PWS sections

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 1 May 26, 2016

1. Introduction This Quality Assurance Surveillance Plan (QASP) documents the approach the Internal Revenue Service (IRS) will take to monitor portal Contractor technical performance required by the Performance Work Statement in effect at the time of the surveillance activity.

1.1 Scope

This QASP documents how the IRS will monitor Contractor Performance Service Level Objective (SLO) performance under this contract. The QASP will be tailored to the characteristics and requirements of the particular contract by the addition of the germane Surveillance Activity Table in Section 4 of this document. SLOs not contained in this document will not be considered to be within scope of the QASP implementation.

The QASP is a Government document used to monitor the Contractor’s Quality Assurance Program (QAP), which the Contractor must have in place, or shall establish prior to contract award, to attain, sustain and advance exceptional delivery of services to the IRS. The Contractor’s QAP will include its description of people, tools and processes that it will use to ensure compliance with all contract requirements and its proposed methods for providing continuous improvement in service delivery. The Contractor’s QAP is expected to include a detailed set of plans (for example, Quality Control (QC) Plan, Configuration Management and Change Control Plan, Asset Management Plan, Risk Management Plan and/or other plans at the Contractor’s discretion) that the Contractor will use to manage its execution of the Portal Services program.

The QASP does not define or dictate how the Contractor should accomplish the work required under contract provisions; instead, the QASP measures the degree of achievement of the Contractor’s execution of its management and QC actions necessary to meet the performance standards documented herein. Thus, the internal continuous performance monitoring performed by the Contractor is not part of the QASP implementation activity, but does yield data that are leveraged during QASP activities.

The QASP can be changed or updated. It is intended to be a “living” document that should be revised or modified as circumstances warrant. Either the Contractor or the Government may initiate changes to the QASP at any time during contract performance; changes to the QASP will require bilateral approval. Such changes shall not entitle the Contractor to any equitable adjustments or to any other compensation for performance in a prior period.

Some of the data developed during QASP activities may be used in other Enterprise Technology Implementation (ETI) measurement activities. The descriptions of those other measurement activities are contained in documents external to this QASP.

1.2 Authority

Authority for issuance of this QASP is the Federal Acquisition Regulation (FAR) Subparts 37.6, 46.1, and 46.4, as well as contract Section E – Inspection and Acceptance, which provides for inspections and acceptance of the work and services called for in service contracts, to be executed by the Contracting Officer (CO) or a duly authorized representative.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 2 May 26, 2016

1.3 Goals and Objectives

The QASP activity is a critical component for Contractor performance measurement. This QASP will be implemented by the ETI Quality Management (QM) function.

1.3.1 Goal

The overall goal of the QASP activity is to confirm that the IEP 1.5 Contractor’s delivery and execution of products or services meets the technical performance measures, or SLOs, included in the contract and documented in this QASP.

1.3.2 Objectives

Specific QASP objectives include:

• Determine Contractor performance is meeting all contract SLOs. Contractor performance data may be gather through audits, examinations, customer feedback and measurements gathered from reports.

• Provide data to the CO and COTR to assess performance and other contract provisions, as appropriate.

• Identify portal program risks arising from Contractor performance in meeting contract SLOs.

• Identify Contractor corrective action(s) needed to meet SLOs or to address identified risks or issues.

1.4 QASP Update

This Plan will be updated periodically to reflect the results of a joint review, by IRS and Contractor personnel, of SLOs, SLO definitions, and SLO monitoring approaches.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 3 May 26, 2016

2. Roles and Responsibilities The IRS bears primary responsibility for implementing this QASP, with support from the Contractor.

2.1 Internal Revenue Service

Table 2-1 describes Government roles and responsibilities regarding QASP implementation.

Table 2-1 Government Roles and Responsibilities

Role Responsibilities

ETI Management • Establishes and approves the QASP

• Provides resources to implement the QASP

• Determines achievement of other contract provisions

• Analyzes Quality Assurance Surveillance Review (QASR) and similar feedback and findings, as appropriate

ETI Quality Management

(QM)

• Conducts QASR

- Interviews business customer(s)

- Reviews Contractor-provided reports

- Conducts liaison with Contracting Officer Representative(s) (COR(s))

- Documents findings

- Provides formal written feedback, including findings

- Participates in project status reviews, as necessary

• Performs in-depth quality review activities at Contractor location, as necessary and if authorized

• Debriefs Contractor (with CO and/or COR) on the surveillance review results, as necessary

Contracting Officer • Monitors contract compliance, contract administration, and cost control

• Reviews QASR and similar feedback and findings, as appropriate

• Provides contractual direction

• Debriefs Contractor (with ETI QM and/or COR) on the surveillance review results, as necessary

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 4 May 26, 2016

Role Responsibilities

Contracting Officer’s Representative

• Chairs Contractor deliverables review, Project Status Reviews (PSRs), and similar performance and status reviews with IRS stakeholders and Contractor

• Reviews QASR and similar feedback and findings, as appropriate

• Forwards QASR and similar reports to appropriate business customer (Online Services (OLS), etc.)

• Debriefs Contractor (with CO and/or ETI QM) on surveillance review results, as necessary

2.2 Contractor

Table 2-2 describes Contractor roles and responsibilities regarding QASP implementation.

Table 2-2 Contractor Roles and Responsibilities

Role Responsibilities

Contractor Program Manager • Reviews and concurs with the QASP

• Collects, analyzes, documents, and produces performance data

• Submits data reports to CO/COR

• Supports IRS QASRs and similar activities, as necessary

• Identifies and implements corrective actions

• Submits requests to COR for new reports or measures

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 5 May 26, 2016

3. Approach ETI has established a multi-tiered framework to facilitate quality assurance (QA) monitoring of portal contractor performance. ETI QM will leverage Contractor-supplied data to implement this framework. The following sections describe this strategy in more detail.

3.1 Overall Approach

The ETI QA monitoring framework described in this QASP integrates and leverages ongoing monitoring activities. ETI QM performs two types of audits, QA surveillance reviews on a regular basis and in-depth reviews (“deep dives”) on an exception basis, when implementing the QA monitoring framework. These surveillance reviews shall not unduly burden the Contractor and shall comply with FAR 52.246-4 Inspection of Services – Fixed-Price. The Contracting Officer must be notified if the Contractor believes any action may affect contract price, terms, or conditions.

ETI QM’s basic approach is to: establish an agreed and understood quality baseline; monitor Contractor performance against that baseline; identify any performance gaps on a timely basis;

and confirm that corrective actions are applied in a timely fashion.

• ETI QM staff and other interested IRS stakeholders meet periodically with Contractor program personnel to review SLOs in the QASP and the monitoring and reporting systems to enable continuous and accurate intake, analysis, and reporting of performance data.

• The Contractor implements continuous performance monitoring as agreed during the QA Program Baseline Review (PBR), uses internal Quality Control (QC) resources to meet SLO performance thresholds defined by the Acceptable Quality Levels (AQLs), and submits deliverables periodically to report status against the quality baseline. Although these monitoring activities themselves are not included in the scope of the IRS’ QA surveillance, they yield measurement data and deliverables that are assessed during QASP implementation.

• The IRS COR holds Project Status Reviews (PSR) with other relevant IRS stakeholders, including ETI QM and internal IRS SLO “owners”, to assess SLO status against the established baseline, identify emerging risks and issues, and determine potential “deep dive” opportunities.

• The ETI QM function audits Contractor performance by, at a minimum, checking and synthesizing reported SLO data to develop quality surveillance feedback, including findings and potential corrective action requests, for submission to the CO/COR. These audit results are submitted using the form in Appendix B.

• The IRS reserves the right to perform in-depth reviews of Contractor performance (“deep dives”) as necessary to assess perceived or actual performance issues. These in-depth reviews may occur at any time during the evaluation period and may address performance issues arising in the current or any prior evaluation period. If an in-depth review (audit) is necessary, the IRS will issue separate formal notification to the Contractor of its intent to perform the review, no later than 30 calendar days before it is

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 6 May 26, 2016 scheduled to begin. To support such deep dives, the IRS may request additional data from the Contractor, including but not limited to raw data, audit results, evidence of certification, QC program data, etc. The IRS reserves the right to perform these deep dives at Contractor or subcontractor facilities.

3.2 Surveillance Methods

ETI QM has at its disposal several methods with which to perform the QA monitoring function described above:

• Contractor Self-Reporting. Uses contractor-supplied performance data collected by tracking and monitoring tools and/or contractor status reports that identify problems, issues and/or risks. (See Appendix C, Reference number 3.)

• Trend Analysis. Per the US Census Bureau: “This method should be used regularly and continually to monitor the contractor’s ongoing performance over time. Data for tracking trends can be gathered from all other evaluation sources and methods, [and used to populate] a database upon which the trend analysis is based. This database should be created and maintained by government personnel. The contractor’s own metrics may provide most of the information needed for the analysis.” (See Appendix C, Reference number 1.)

• Quantitative Analysis. This refers to the use of statistical and other quantitative techniques to analyze raw performance data to characterize performance and/or identify trends, causal relationships, or root causes of performance issues. Examples would include analyzing results of customer survey and problem tickets against relevant SLOs.

This method is appropriate for in-depth analysis of areas of interest.

• Periodic Inspection. Per the Office of Management and Budget: “This method, sometimes called ‘planned sampling,’ consists of the evaluation of tasks selected on other than a 100 percent or random basis. It may be appropriate for tasks that occur infrequently, and where 100 percent inspection is neither required nor practicable. A predetermined plan for inspecting part of the work is established using subjective judgment and analysis of agency resources to decide what work to inspect and how frequently to inspect it.” (See Appendix C, Reference number 2.)

• Random Sampling. Per the Office of Management and Budget: “This is usually the most appropriate method for recurring tasks. With random sampling, services are sampled to determine if the level of performance is acceptable. Random sampling works best when the number of instances of the services being performed is very large and a statistically valid sample can be obtained.” (See Appendix C, Reference numbers 2, 3.)

• 100 Percent Inspection. Per the Office of Management and Budget: “This is usually only the most appropriate method for infrequent tasks or tasks with stringent performance requirements, e.g., where safety or health is a concern. With this method, performance is inspected/evaluated at each occurrence. One hundred percent inspection is too expensive to be used in most cases.” (See Appendix C, Reference numbers 2, 3.)

• Customer/User/Stakeholder Feedback. This method is an independent evaluation of an activity or process to assess compliance with SLOs. This surveillance method

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 7 May 26, 2016 supplements the other surveillance methods. Customer Feedback may be necessary for certain types of tasks that do not lend themselves to random sampling of 100% inspection. Customer feedback may be obtained either from the results of formal customer satisfaction surveys or from random customer complaints or complements.

Customer complaints, to be considered valid, must set forth clearly and in writing the detailed nature of the complaint, must be signed or emailed and forwarded to the PM or COTR who will assess the validity of the complaint and retain it in a documentation file.

3.3 Evaluation Methodology

ETI QM will use the cited observation methods to determine whether AQLs have been met over the evaluation time frame. The evaluation period will be annually. The specific evaluation methodology is as follows:

• ETI QM has established a Surveillance Activity Workbook (SAW), maintained within the IEP, to capture quality surveillance data.

• For each applicable SLO/performance metric, ETI QM will determine whether the associated AQL has been met. ETI QM will base this determination on SLO data contained in Contractor deliverables, using the specification in Appendix A as the baseline definition of the SLO, as well as any additional data gathered by Government personnel. The Met — Not Met determination will be captured in the SAW. In addition, any quality issues or questions noted for the reporting period will be captured as comments in the SAW.

• Based on the evaluation results, the IRS may determine that additional review of Contractor performance is warranted. In this case, ETI QM will use Standard Operating Procedures for In-depth Review to plan and conduct a “deep dive” analysis. Results of this analysis will be provided to ETI management, the CO, and the COR before discussion with the Contractor.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 8 May 26, 2016

4. Required Performance Metrics Table 4-1 lists Contractor performance standards and identifies the surveillance methods that may be used to monitor performance against those standards. Although only a Primary Monitoring Approach(es) is (are) listed, the IRS reserves the right to engage, as necessary, in any of the monitoring approaches described in Section 3.2 to obtain full insight into Contractor performance.

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 9 May 26, 2016

Table 4-1 Required Performance Metrics

Cross- Reference Performance Standard

Acceptable Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

C.3.3.1 Web Hosting Service Availability (production) - Availability to application services as measured from end-user access environment

99.99%

• SLO Compliance Report

• Monthly

Contractor Self-reporting, Customer feedback;

Trend Analysis

6% of Web Hosting invoice

C.3.3.1 Web Hosting Service Availability (non-production) – Contractor proposed measure 99.9%

• SLO Compliance Report

• Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3 Incident Response Time (production) - P1 incident time to respond shall be 30 min or less

P2 incident time to respond shall be 2 hours or less 95%

• SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3 Incident Notification Time – defined as the Amount of time between when the contractor becomes aware of an incident and when the contractor notifies the IRS.

Phone notification – (EOps Service Desk)

E-mail notification (ETI; business owner): P1: 30 mins

Phone notification - (EOps Service Desk)

E-mail notification (ETI; business owner): P2: 2 hours

95% • SLO Compliance Report

• Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3 Incident Resolution Time (production)

P1 Incident resolution time shall be within 4 hours

P2 Incident resolution time shall be within 48 hours

95% • SLO Compliance Report

• Monthly

Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.3.3 Problem Root Cause Resolution – 95% • SLO Compliance Report Contractor Self-reporting; Negative CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 10 May 26, 2016

Cross- Reference Performance Standard

Acceptable Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

Priority 1 & Priority 2 Problems and problem candidates must have a root cause discovered, replicated and resolved within 30 calendar days and communicated to

IRS

• Monthly Trend Analysis Award Term impact

C.3.3.6 Portal Web Hosting Services - Changes successfully deployed

Percentage based on number of changes deployed within agreed upon time schedule in ticket/total number of changes scheduled for the month)*100

No outage or issue attributed to the web hosting service or integration point after implemented change:

Percentage based on changes validated and confirmed by implementer or requestor/total number of changes deployed

95% • SLO Compliance Report

• Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

6% of Portal Web Hosting

Services invoice

C.3.3.1;

C.3.3.8

Portal Web Hosting Services –

Service Request Response Time (non-production)

a) New User Access – 24 hours

b) Password reset – 4 hours

c) General information - 5 days - unless otherwise defined

d) New virtual environments provisioning - 5 days -unless otherwise defined

e) Log files - continuous

f) Existing user log-in to assigned non-production environment 24x7

95% • SLO Compliance Report

• Monthly

Contractor Self-reporting, Customer feedback;

Trend Analysis

Negative CPARS and Award Term impact

C.3.4 Verify effective Security Operations are in place for IEP 1.5

100%

• Security Status and Reports Contractor Self-reporting;

Periodic Inspection 6% of Portal

Security

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 11 May 26, 2016

Reference Performance Standard

Acceptable Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

Ensure all reporting is timely and accurate.

Ensure all data feeds are maintained per requirements.

Ensure activities are executed according to Security Services Process plans and documents.

All status reports and actions for the month are addressed and signed off by IRS IEP Authorizing Official or Authorizing Official Designated Representative

(AODR)

• Monthly Random Sampling invoice

C.3.4.5 Maintaining security configuration baseline

Demonstrate a high-level of compliance to approved configuration baseline. The list of configuration items (software and devices) is maintained in the Security Configuration and Change Management Plan.

Number of configuration items with a compliance score of at least 90% / Number of configuration items

90%

• Security Status and Reports

• Monthly

Contractor Self-reporting;

Periodic Inspection

Random Sampling

2% of Security invoice

C.3.4.3 Security POA&M Remediation 90%

• Security Status and Reports

• Monthly Contractor Self-reporting;

Trend Analysis 2% of Security invoice

C.3.5.2 irs.gov Website Availability – Contractor proposed measure

99.99% • SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis 6% of irs.gov invoice

C.3.5.2 CMS Availability - Monthly cumulative availability as measured based on P1 tickets

99.99%

• SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.5.2 System Response Targets – website response time

User requests for Website content from a user page to

95%

• SLO Compliance Report Contractor Self-reporting;

Customer Feedback, Negative

CPARS and

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 12 May 26, 2016

Reference Performance Standard

Acceptable Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

user page served completed in 3 seconds or less • Monthly Trend Analysis Award Term impact

C.3.5.3 System Response Targets – search queries completed in 3 seconds or less

99% • SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.5.1 System Response Targets – irs.gov application transactions (user interaction with applications) completed in 5 seconds or less

95% • SLO Compliance Report

• Monthly Contractor Self-reporting;

Customer feedback, Trend Analysis

6% of Portal Application Services invoice

C.3.5.2 WCMS - % Successful content updates.

100% of content updates (including user interfaces, templates and style sheets) to the Content Management System must be publicly available on all access sites, with all stale instances flushed from any IEP Web hosting cache components, within 30 minutes or less of completion of the CMS "publish" task by CMS user if published via CMS workflow: CMS “publish” action if published via CMS manual publish; or CMS publish date/time if published via scheduled publishing.

99.99%

• SLO Compliance Report

• Monthly

Contractor Self-reporting;

Customer feedback, Trend Analysis

Negative CPARS and Award Term impact

C.3.6 CSR Call Back timeliness

Percentage of CSR returned calls to the user (1 attempt) within 24 business hours. CSRs cannot call a user back after 9pm (users’ local time).

95%

• SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

C.3.6 Help Desk Response time – Phone Calls

Percentage of calls answered within 3 rings or 40 seconds (calls not answered within 3 rings will be provided the option of being forwarded to an automatic

80%

• SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis

1% of Help Desk invoice

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 13 May 26, 2016

Reference Performance Standard

Acceptable Quality Level

(AQL)

Contractor Reporting Document

Contractor Reporting Frequency

Primary Monitoring Approach

Penalty for not meeting

AQL

voice mail service or continuing to hold).

C.3.6 Help Desk Response time e-mails

Percentage of e-mails responded to within 24 business hours with scripted e-mail response.

Percentage of e-mails that receive an auto reply response within 1 hour

100%

• SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis

1% of Help Desk invoice

C.3.6 Help Desk Response time – chats Percentage of the chat requests answered within 40 seconds.

80% • SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis 1% of Help

Desk invoice

C.3.6 Call Abandonment rate Abandoned Calls: Not to exceed 5%

<5% • SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis 1% of Help

Desk invoice

C.3.6 Portal Website Help Desk Services - Voice-mail system response

Percentage of voice-mail messages left in system during business hours retrieved and responded to within 1 hour of receipt.

Percentage of voice-mail messages left in system during non-business hours retrieved and responded to within next 8 business hours

100%

• SLO Compliance Report

• Monthly Random sampling; Trend

Analysis

Negative CPARS and Award Term impact

C.3.6 Portal Website Help Desk Services - Changes in required Help Desk staff Percentage of help desk staffing level change requests fulfilled within 3 weeks of notification

100%

• SLO Compliance Report

• Monthly Contractor Self-reporting;

Trend Analysis

Negative CPARS and Award Term impact

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 14 May 26, 2016

5. Reporting Flow The QASP surveillance activity will be performed periodically as required by contract terms, and the results will be reported using the flow outlined in Figure 5-1 below.

Figure 5-1 Surveillance Reporting Flow

At the end of the evaluation period, ETI QM will use the data contained in the SAW to prepare a written report for the CO/COR, summarizing the overall results of the QA monitoring of the Contractor’s performance over the evaluation period. The report, documented in the format presented in Appendix B, will become part of the QA documentation for the contract.

ETI QM will submit the report to ETI Program Management, the CO, and the COR no later than two weeks after the end of the evaluation period; the data will be used as the technical input for fee determinations or achievement of other contract provisions, as appropriate. Feedback will be provided to the Contractor in accordance with the terms of the contract (I.5.1). Depending on the assessment contained in the report, the CO/COR may require the Contractor’s project manager, or a designated alternate, to meet with the CO/COR, ETI QM, and other Government personnel as deemed necessary to discuss the performance evaluation.

ETI QM, in addition to providing documentation to the CO/COR, will maintain a complete QA file. The file will contain copies of all data analyses, reports, evaluations, recommendations, and any actions related to the Government’s performance of the QA function, including the originals of all the SAWs. All such records will be retained for the life of the contract. ETI QM shall forward these records to the CO upon completion of the contract.

ETI QM

Conduct Regular

Surveillance Reviews

ETI QM

Conduct in-depth Reviews, as necessary (“Deep Dive”)

ETI QM Develops and Delivers the Audit Reports to ETI Management

CO/COR

ETI’s Management and CO/COR Reviews Audit

Report

CO/COR

Provides

Feedback to Contractor

ETI QM

Debriefs

Contractor, as necessary with

CO/COR

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 15 May 26, 2016

Appendix A: Metric Specifications Example

D.x.x.x All Task Orders: Deliverables Type: Technical Category: Deliverables Surveillance Method: IRS 100% Inspection (each Deliverable) Owner: PPM

SLO Description (IRS):

This measurement description applies to all QASP metrics for any Deliverable measured for the Start-up, Build-out or Transition Phases.

Acceptable Quality Level (IRS):

100%

SLO Reporting Frequency

(IRS):

Monthly

SLO Calculation (Contractor):

Deliverable met = 100%, Not met = 0% Base Measures (Contractor):

Deliverable Met

Description: Deliverable accepted by IRS in a timely manner per schedule Data Summarization: N/A. Each Deliverable is measured individually.

Reporting Shift: N/A Reporting & Retention: Reported monthly and kept in archive for 3 years Exceptions: If Contractor initially submits the deliverable on schedule and IRS requests changes during the initial review cycle, Contractor will be credited with meeting the due date if IRS accepts the deliverable upon the follow-on (second) review cycle.

Data Source: IRS Deliverable Tracking Matrix Data Verification: IRS Deliverable Manager and COR verify completion of Deliverables

Recovery:

• N/A

Other Notes:

History:

• <mm/dd/yyyy>: Baseline description approved by TBD

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 16

Appendix B: Forms Surveillance Review Report

To: IRS CO, Vendor’s Name

IRS COR, Vendor’s Name

From: IRS ETI Quality Manager

Re: Surveillance Monitoring

Contract No:

The following report is provided as formal written feedback of the results of surveillance monitoring activities that were recently conducted by the IRS Enterprise Technology Implementation Office.

Surveillance Period _______________

Performance Period ________________

ID SLO Description AQL Data

Source Weight Actual

Results Evaluation Score Comments S.3.6.4 Percentage of the chat requests answered within 40 seconds

80.00% SLO 0 0

S.3.5.4 Search queries completed in 3 seconds or less

99.00% SLO 0 0

S.3.4.3 Security POA&M Remediation

90.00% SLO 0 0

Summary Comments: Summary Performance Results:

Total Possible Score: 0

Total Actual Score: 0

Total % Weighted SLOs Met: 0%

Signed:

IRS ETI Quality Manager

IRS DMO Manager

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 17

Appendix C: References

1) US Census Bureau, Attachment 2: Performance-based Statements of Work Guidance for

Using a Quality Assurance Plan (accessed on June 24, 2010 at www.census.gov/procur/www/fssp/att02.html)

2) A Guide to Best Practices for Performance-based Services Contracting, Office of Federal Procurement Policy (OFPP) in the Office of Management and Budget (OMB), Final Edition, October 1998

3) United States Department of Agriculture Foreign Agricultural Service, Infrastructure Management Quality Assurance Surveillance Plan, Version V0.4, November 2008 (accessed on June 24, 2010 from the “Seven Steps to Performance-based Acquisition” website at https://www.acquisition.gov/comp/seven_steps/library3.html. See Step 5.)

http://www.census.gov/procur/www/fssp/att02.html https://www.acquisition.gov/comp/seven_steps/library3.html

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 18

Appendix D: Acronyms

Acronym Definition

AQL Acceptable Quality Level

CO Contracting Officer

COR Contracting Officer’s Representative

CPARS Contractor Performance Assessment Reporting System

OLS Online Services

FAR Federal Acquisition Regulation

FY Fiscal Year

IRS Internal Revenue Service

N/A Not Applicable

IEP Integrated Enterprise Portals

NTP Network Time Protocol

OFPP Office of Federal Procurement Policy

OMB Office of Management and Budget

PMMP (ETI) Performance Monitoring Management Plan

ETI Enterprise Technology Implementation Office

ETIPMP Enterprise Technology Implementation Performance Management Program

PSR Project Status Review

PWS Performance Work Statement

QA Quality Assurance

QA PBR QA Program Baseline Review

QASP Quality Assurance Surveillance Plan

Quality Assurance Surveillance Plan (QASP) ■ Version 1.1AAA 19

Acronym Definition

QASR Quality Assurance Surveillance Review

QC Quality Control

QM Quality Management

RUP Registered User Portal

SAW Surveillance Audit Workbook

SLO Service Level Objective

TBD To Be Determined

VPN Virtual Private Network

1.4 QASP Update