D - ITIOSS _TO1_QASP_SLAs.docx

ATTACHMENT D

IT Infrastructure Operations Support Services (ITIOSS) Task Order 1 IT Infrastructure Operations Support Services (ITIOSS) Operations and Maintenance (O&M) Development Modernization and Enhancements (DM&E) Quality Assurance Surveillance Plan (QASP)

7-1-19

Table of Contents

1. INTRODUCTION

This quality assurance surveillance plan (QASP) establishes procedures and guidelines ITIOD will use to ensure the Contractor achieves the required performance standards or service levels on the task order. This QASP identifies performance objectives, defines the methodologies used to monitor and evaluate the Contractor’s performance, describes quality assurance documentation requirements, and describes the analysis of quality assurance monitoring results. The cornerstone of performance management is the ability to make decisions based on the analysis of performance data; this analysis yields information that indicates whether the contractor is achieving expected outcomes for the project.

a. Methodologies to Monitor Performance PBGC will monitor performance through stakeholder surveys, 100% review of monthly and quarterly deliverables and through the receipt of unscheduled feedback. PBGC will collect all performance data for review by the Performance Review Board (PRB) on a monthly basis. Additionally, the PRB reviews all incidents in addition to the monthly Program Status Report and PBGC and the Contractor will jointly address issues or downward trends in customer satisfaction. PBGC will work with the Contractor to implement improvement initiatives that impact internal PBGC processes as well as to define improved processes, roles and forums for governing all service areas. In addition to surveys, each month, the COR will conduct a review of all contractor-generated deliverables and collect a log of negative feedback from PBGC end-users during performance. Note: the term “User” is inclusive of all end-users, customers, or events that cause an incident to be reported.

2. ROLES AND RESPONSIBILITIES

The Contracting Officer

The Contracting Officer (CO) is responsible for monitoring contract compliance, contract administration, and cost control and for resolving any differences between the observations documented by the Contracting Officer's Representative (COR) and the Contractor. The CO will designate one full-time COR as the government authority for performance management. The number of additional representatives serving as technical inspectors depends on the complexity of the services measured, as well as the contractor’s performance, and must be identified and designated by the COR.

The Contracting Officer’s Representative

The Contracting Officer’s Representative (COR) is designated in writing by the CO to act as his or her authorized representative to assist in administering a contract. COR limitations are contained in the written appointment letter. The COR is responsible for technical administration of the project and ensures proper government surveillance of the contractor’s performance. The COR is not empowered to make any contractual commitments or to authorize any contractual changes on the government’s behalf. Any changes that the contractor deems may affect contract price, terms, or conditions shall be referred to the CO for action. The COR will have the responsibility for completing QA monitoring forms used to document the inspection and valuation of the contractor’s work performance. Government surveillance may occur under the inspection of services clause for any service relating to the contract.

Contract Monitor

Contract Monitors are the representatives closest to technical work performed. Monitors are personnel who have responsibility for reviewing technical performance and/or interacting with the contractor on a regular basis. The responsibilities of the Contract Monitors are to:

· Provide detailed information monthly to the COR and the Award Fee Board Members in writing and or orally before the board on the contractor's performance

· May attend board meeting at the discretion of the AFEB chair

Performance Review Board (PRB)

· Refer to section 4.2 of the Award Fee Determination Plan for the roles and responsibilities of the PRB.

3. PERFORMANCE REQUIREMENTS SUMMARY TABLE

3.1 CPAF Performance Metrics

3.2 FFP Performance Metrics

FFP CLINs Services scope are covered by the PWS’ End-User Services Section (IT Service Desk & Site Support), IT Service Catalog Support Section and Disaster Recovery/Continuity of Operations Planning (COOP) and Testing Section

· Following KPI metrics shall be included in the contractor self-assessment but they excluded from Cost Plus Award Fee calculation table and the process.

· Some of the metrics are shared between FFP and CPAF CLINs since they are performed by resources under both CLINs. For example some of the escalated priority incidents are resolved both Site Support(FFP) or other platform teams.

Service Level Agreements (SLAs) The SLAs represent the Government’s proposed performance measurement structure. Offerors are strongly encouraged to recommend changes to the SLA that improve PBGC’s ability to achieve its program and service objectives. During transition (see § 5.2 of O&M and DM&E Task Order), the Contractor will finalize and formally adopt contract SLAs. As part of that process, PBGC and the Contractor will analyze SLAs to ensure that they are fair and reasonable for the ensuing transfer of control. Both parties must review changes, and upon acceptance, the changes will be reflected by modification of the contract by the end of the transition period and prior to the Contractor taking full control of services. During this period, SLAs will be tracked, monitored, adjusted, and reported against targets as part of the monthly review by the Performance Review Board and may be adjusted through bilateral agreement.

The Contractor shall be evaluated on the proposed and accepted SLAs that apply specifically to transition activities during the first two months of award.[footnoteRef:1] The Contractor will be evaluated in accordance with contract SLAs beginning the month 3 of the contract. The first performance period under which the Contractor will be evaluated under the contract SLAs with therefore be four months. After that the contractor will be evaluated on a quarterly basis. [1: See § 5.2 of Task Order 02 regarding requirements during transition and related provisions of the Award Fee Determination Plan. ]

Remedy Plan for Underperformance When failure to meet an AQL occurs two (2) or more consecutive months with the same metric, the Contractor shall propose a remedy plan. This plan shall be approved by the Government and executed by the Contractor to bring the latter’s performance up to AQLs as defined by the SLAs.

4. INCIDENT ESCALATION SYSTEM

PBGC uses a tiered incident escalation system (Tiers 1, 2, 3 and 4) to classify the level of knowledge, skill, and access privileges typically required of IT support to resolve an incident. All incidents are first addressed at the Tier 1 level (first-call resolution). In some circumstances, Tier 1 is insufficient to resolve an incident and will require escalation to a higher-level tier such as the platform teams for resolution.

Figure A – Tier Level Descriptions

5. PRIORITIZING SUPPORT SERVICES BY IMPACT AND URGENCY

Incidents are prioritized in accordance with impact and urgency ratings assigned to the end-user’s incident upon contacting the Service Desk. Each incident is assigned a predefined triage level based on the Priority Levels, and priority level incidents are reported monthly to the Performance Review Board. Priority levels are determined based the determination of the impact of the incident on PBGC as well as the urgency as determined by the table below.

Using the number that results from the above matrix, the incidents are prioritized in accordance with the following Service Priority Levels. The incidents are also assigned a time-to-resolve as well as an acceptable quality level (AQL). PBGC measures the time-to-resolve from initial contact, response, and final resolution of an incident. Time-to-resolve may end when the user has been returned to a state of normal operation but has not responded to three documented contact attempts to ascertain the user’s full satisfaction. All instances of user non-response will be escalated to federal staff for resolution.

Figure B – Service Priority Levels

· An unplanned outage of a mission-essential production application or service (as documented by “COOP Essential” or “COOP Critical” within mAppIT, excluding desktop applications)

· A primary GFE desktop, laptop or mobile application or infrastructure outage impacting a VIP’s ability to perform work. Identification of VIP tickets shall rely upon the ‘requested for’ field in SM9.

· A desktop application outage impacting 50 or more users.

3 business hours.

AQL=85%

TQL=90%.

· A desktop application outage impacting 2-49 users or impacting a single user such that this user is completely unable to work in any capacity using IT systems because of the outage.

· A degradation in infrastructure or application performance or fault tolerance of a critical/essential production business application/service.

· A non-Production (CDE-T, CDE-I, ITC) service outage impacting many users and/or seriously imperiling a scheduled software release.

1 business day (12 business hours).

AQL=85%

TQL=93%

· A degradation in infrastructure or application performance or fault tolerance of a non-essential production business application/service. C) A non-Production (CDE-T, CDE-I, ITC) service/application outage impacting several users 2 business days (24 business hours).

AQL=85%

TQL=93%

· A non-Production (CDE-T, CDE-I, ITC) service/application outage impacting a single user

· A non-Production (CDE-T, CDE-I, ITC) service/application degradation.

4 business days (48 business hours).

AQL=85%

TQL=94%

ATTACHMENT A

SLA Detailed Descriptions:

CS-1 Tier 1 Customer Survey This metric captures the overall customer experience performance of Tier-1 Service Desk staff in resolving issues for PBGC users. When the incident is closed, an email is delivered which includes a link to a customer service survey. Users are invited but not required to participate in the survey. Surveys submitted by ITIOSS contractor team members are accepted for service improvement reasons but are excluded from the SLA performance metric. The current Acceptable Quality Level (AQL) is 70% and Target Quality Level (TQL) is 80% of the overall responses being “Good” (4) or “Excellent” (5).

CS-2 Tier 2+ Customer Survey This metric captures the overall customer satisfaction in addressing incidents escalated to ITIOSS Tier 2. When the incident is closed, an email is delivered which includes a link to a customer service survey. Customers are invited but not required to participate in the survey. Surveys submitted by ITIOSS contract team members are accepted for service improvement reasons but are excluded from the SLA performance metric. The current Acceptable Quality Level (AQL) is 70% and Target Quality Level (TQL) is 80% of the overall responses being “Good” (4) or “Excellent” (5) Below is an example of the CS-1 and CS-2 survey questionnaire:

CS-5 COR Opinion/ITIOD Stakeholders Survey This metric provides the COR an opportunity to evaluate the contractor’s performance during and directly influence award fee determination. The COR determines the final score based on survey inputs and other factors. The result is calculated by the average of all responses being “Unacceptable”, “Poor”, “Acceptable”, “Good” or “Excellent” (scores 1,2, 3, 4 or 5). The Accepted Quality Level(AQL) is the average of all responses being “Acceptable” (average score of 3) and the target quality level (TQL) is the average of all responses being “Good” or “Excellent” (scores 4 or 5).

Below is the survey that is sent to ITIOD stakeholders via a SharePoint page link on a monthly basis.

TR-10: Infrastructure Request Response ITIOD maintains an Infrastructure service catalog request and tracking system on SharePoint. This catalog, distinct from GetIT (PBGC enterprise service catalog), provides access to common services such as new server builds, and software packaging requests. Currently "Server: New Server" and "Oracle: Data Fix" catalog are measured. New server builds include only virtual servers (Windows and RedHat) and are not expected to be more than 20 per month. The SLA will be scored as ½ point for servers and ½ point for data fixes. The AQL and TQL thresholds are based on the specific delivery deadlines defined in the infrastructure service catalog (2 business weeks for server builds, 2 business days for data fixes). Builds assigned to non ITIOSS contractors were excluded.

TR-13 Time to Address Request for Information(RFI) RFI are the IM tickets (In ServiceNow) that categorized as request for information in ServiceNow. TR-13 measures the timeliness in which RFI's are resolved. RFI's are primarily categorized as Priority 3 (VIP) or Priority 4(non-VIP) Tickets. P3 tickets must be resolved in 4 business days and P4 tickets must be resolved in 8 business days. The SLA percentage is calculated by taking the numbers of RFI's completed on time over the total number of RFIs TR-14 First Call Resolution/Tier 1 Resolution TR-14 measures how many phone tickets are being resolved by the Service Desk. It calculated by taking the number of phone tickets resolved by the service desk over the total number of phone tickets resolved.

TR-15 Time to Resolve Service Desk Ticket TR-15 measures the resolution timeliness of phone tickets resolved by the service desk. The TQL for resolution time for phone tickets resolved by the service desk two hours. The SLA is calculated by taking the number of on time resolved phone tickets over the total number of phone tickets by the service desk.

TR-16 Tier 1 Assignment Quality TR-16 measures the amount reassignments a phone ticket has once it has been escalated to another team by the service desk. The TQL for this SLA is 0 reassignments. The SLA is calculated by taking the number of tickets with 0 reassignments over the total number of tickets.

SA-4 Backup Success Rate SA-4 measures the daily backup of PROD activities. It is calculated by taking the number of successful backup jobs over the total number of backup jobs run.

AR-2 Security Incident Response This metric is calculated as the time from when an event is declared an incident—after the identification phase of the Incident Handling process—until the Enterprise Cybersecurity Division (ECD) is notified that a security incident has occurred. The incident identification is documented in the Service Desk ticket recording the suspicious security event, and ECD is normally notified through email as an official communication. This SLA is calculated by taking the number of incidents responded within one hour divided by the total security incidents during the period.

AR-7b Major Incident Process Adherence This SLA provides an incentive for the accurate and efficient application of the Major Incident Management Process. The SLA will take the form of an after-action interview or survey with the ITIOD Federal Incident Manager to characterize performance based on several categories explained in the SLA report/checklist below. AR-7b measures the quality of response to a Major Incident. Major Incidents are scored using the Major Incident Scorecard that totals 51 points. The SLA score is calculated by taking the number of points received on the scorecard by the federal incident manager over the total number of points available.

Major Incident Checklist/Scorecard (example score) AR-7 Major Incident Scorecard

Incident:

Incident date:

Quarterly=1

Description:

Time determine whether incident is deemed “major”

Timely notification of Federal Incident Manager

Time to create IM ticket

Timely notification of PBGC stakeholders

Effectiveness of troubleshooting, and determination of work around if applicable

Effectiveness and timeliness of ongoing communications to FIM and Stakeholders throughout incident

Time to restore service, or implement workaround

Documentation of incident diagnosis and resolution changes in approved emergency RFC

Effectiveness of end-user communication

Timeliness of AAR

Quality of AAR

Summary Score

Secure Configuration Baseline (SCB) SLAs Secure Configuration Baseline (SCB) SLAs

1. AR-8 SCB Non-Compliant Individual CI Remediation Timeliness:

This metric measures the timely remediation when an individual configuration item (CI), e.g. workstation server, database, device, etc. with an approved security configuration baseline checklist falls below the minimal CI baseline compliance threshold defined in approved baseline documents and noted in the table below. PBGC’s custom developed Continuous Automated Compliance Monitoring (CACM) solution, will record a non-compliance exception and a notification to the Service Desk on the first Monday that follows when an individual CI has fallen below the minimal CI baseline compliance threshold. Contractor is responsible for remediation within 30 days of the first detection by correcting the configuration issue, removing the CI from network, or documenting a risk acceptance. The current Acceptable Quality Level (AQL) is remediation of 99% and Targeted Quality Level (TQL) is remediation of 100% of all detected non-compliance exceptions initially detected by CACM during the POP end - 30 days. Specifics about remediation are as follows:

1. Correcting the configuration issue and bringing the individual CI abovethe minimalCI baseline compliance threshold

2. Submit Risk Acceptance to ISSO after obtaining concurrence from federal POC that the specific CI and setting is a candidate for Risk Acceptance

3. Remove the device from network (O&M staff is empowered to remove it from the network when they have provided 3 notices within 30 days to the owner of the CI and they have not responded to the requests) When an issue is resolved, CACM will automatically record the exception has been resolved including the date resolution was detected. Individual CIs that have not been on the network for more than 30 days are not scored. Reports will be generated from CACM on a monthly basis for quarter-to-date and the final quarterly report will be used to calculate the SLA score. Below are the fields that will be included on the report:

1. ComputerName

2. BaselineName

3. FirstReportDate

4. TargetResolutionDate

5. ResolutionStatus

6. ResolutionDetectionDate

7. InAdvanceofTargetResolutionDate

The SLA score will be calculated as follows:

Count of InAdvanceofTargetResolutionDate (yes) / (Count of InAdvanceofTargetResolutionDate (yes) + Count of InAdvanceofTargetResolutionDate (no))

Overall SLA weight is 2 points (~4% of all SLAs). Overall earned SLA score = SLA weight (2pts) * Total earned score

Sample Report and calculation:

Count of InAdvanceofTargetResolutionDate (yes) / (Count of InAdvanceofTargetResolutionDate (yes) + Count of InAdvanceofTargetResolutionDate (no)) => 16 / (18 +0) (88.9%).

100.0% would then be multiplied by the AR-8 SCB Non-Compliant Individual CI Remediation Timeliness SLA weight: 88.9% x 2 = 1.78 points.

2. AR-9 SCB Aggregate Compliance SLA:

This metric measures the overall Secure Configuration Baseline (SCB) compliance for all configuration items (CIs), e.g. workstation server, database, device, etc. with an approved security configuration baseline checklist in alignment with PBGC’s acceptable and target aggregate CI baseline compliance thresholds as outlined in the table that follows. This table outlines the relative weights for each baseline, which generally gives baselines with more Cis more weight, as well as the AQL and TQL levels for each baseline. PBGC’s custom developed Continuous Automated Compliance Monitoring (CACM) solution, records daily aggregate compliance levels for each baseline and these are averaged over the period of performance to calculate the SLA score. Additionally, a non-compliance exception is recorded in CACM when any aggregate baseline score has fallen below PBGC’s acceptable aggregate CI baseline compliance threshold to provide additional visibility. Contractor is responsible for ensuring aggregate baseline scores are maintained above acceptable and ideally at target levels by correcting configuration issues, removing CIs from the network when they fall below the minimal CI baseline compliance threshold and cannot be corrected, and/or documenting a risk acceptance.

Reports will be generated from CACM on a monthly basis for quarter-to-date and the final quarterly report will be used to calculate the SLA score. Below are the fields that will be included on the report:

1. BaselineName

2. SLAWeight

3. SLA_AQL_AggregateThreshold

4. SLA_TQL_AggregateThreshold

5. AverageAggregateBaselineScore

6. WeightedBaselineSLAScore This is calculated by multiplying the Baseline’s SLAWeight times:

1 if the AverageAggregateBaselineScore >= SLA_TQL_AggregateThreshold 0 if the AverageAggregateBaselineScore >= SLA_AQL_AggregateThreshold < SLA_TQL_AggregateThreshold -1 if the AverageAggregateBaselineScore < SLA_AQL_AggregateThreshold

The SLA score will be calculated as follows:

Sum of WeightedBaselineSLAScore / Sum of SLAWeight

Overall SLA weight is 2 points (~4% of all SLAs). Overall earned SLA score = SLA weight (2pts) * Total earned score

Sample Report and calculation:

Sum of WeightedBaselineSLAScore / Sum of SLAWeight => 8 / 17 (47.1%).

47.1% would then be multiplied by the AR-9 SCB Aggregate Compliance SLA weight: 47.1% x 2 = .942 points.

Patch and Vulnerability Management Process (PVMG)

image3.png image4.png image1.jpg image2.png