J.2 Performance Measurement Management Plan - Final.pdf

Performance Measurement Management Plan

1 INTRODUCTION

This Performance Measurement Management Plan (PMMP) describes how the Cloud Service

Providers (CSP) performance will be measured and monitored against defined service levels for the criteria provided, the approach for payment deductions, and how the detailed SLA document will be constructed. The components of this methodology described in this document include:

• Performance Management Framework

• Performance Metrics and Measures

• SLA Framework

Note that the PMMP sets forth the parameters for those services/deliverables that are subject to

SLA monitoring and does not apply to inspection of other deliverables/services not identified in the

PMMP that will undergo the inspection standards/requirements set forth in the contract.

2 PERFORMANCE MANAGEMENT FRAMEWORK

The Performance Management Framework elements are described in Table 1 below.

Table 1 - Performance Management Framework Elements

Performance Measurement Title: Name/label for the performance measure.

Performance Measure Description:

Description of measurement and measurement parameters to be used to observe and evaluate performance.

Minimum Service Level: Minimum level of performance required.

Full Performance Service Level: Level of performance required to warrant full payment.

Monitoring Method: Source of observations used to evaluate actual service levels delivered.

Measurement Period: Periodicity of measurement collection, aggregation and evaluation.

Below Full Performance Deduction:

Percentage Deduction that is applied when Full Performance Service Level is not met (Minimum Service Level has been met).

Below Minimum Deduction:

Percentage Deduction that is applied when Full Performance Service Level and

Minimum Service Level are both not met.

3 PERFORMANCE MEASURES

For each element under surveillance, relevant performance measures, metrics and required service levels are described in Table 2 below that lists applicable Performance Measures and Service

Level targets for each Task used to monitor and measure contractor performance. The measurement of the SLAs will start at the beginning of Operations. These performance monitoring elements are incorporated into individual SLAs. The Below Full Performance and

Below Minimum Deduction columns are to be taken from the CSPs standard commercial deductions for the appropriate Performance Measure.

Attachment J.2

Table 2 - Performance Metrics and Measures

PERFORMANCE

MEASURE TITLE

PERFORMANCE

MEASURE

DESCRIPTION

MINIMUM SERVICE

LEVEL

FULL

PERFORMANCE

SERVICE LEVEL

MONITORING

METHOD/ DATA

CAPTURE/ SOURCES

MEASUREMENT

PERIOD

BELOW FULL

PERFORMANCE

DEDUCTION

BELOW

MINIMUM

DEDUCTION

Storage Availability

Percentage of time the storage system is available for use.

99.90% for the High availability tier – production storage and

99.00% for the

Standard availability tier – development and testing storage and

98.00% for the Low availability tier – archival storage

99.99% for the

High availability tier – production storage and

99.90% for the Standard availability tier – development and testing storage and

99.00% for the Low availability tier – archival storage

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Storage Provisioning

Time required in provisioning the requested storage

95.00% of the High availability tier – production storage requests within 24 hours and

95.00% of all other tier requests within three (3) days.

99.00% of the High availability tier – production storage requests within 24 hours and

99.00% of all other tier requests within three (3) days.

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

IaaS/PaaS Availability

The percentage of time that IaaS/PaaS solutions are available for normal business operations

100.00% of the IaaS/PaaS solutions are available 99.95% of the time.

100.00% of the IaaS/PaaS solutions are available 99.99% of the time.

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

IaaS/PaaS Provisioning

Time to make new fully configured IaaS/PaaS solutions in the environment available

99.00% of servers provisioned within

24 hours.

99.90% of servers provisioned within

24 hours.

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

SaaS Application Availability

The percentage of time that the application is available.

99.50% application availability

99.80% application availability

Periodic Inspection of 100.00% of data set/ Service Desk

Performance Reports/Logs

Monthly 10% of applicable billing period cost

25% of applicable billing period cost

SaaS Application and

Environment Back-Ups

Timeliness of application & environment daily, weekly, and monthly as requested.

Back-ups should be tested periodically for quality assurance.

97.00% of backups performed in accordance with individual SaaS performance parameters

99.00% of backups performed in accordance with individual SaaS performance parameters

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Data Replication

Timeliness of Data Replication

97.00% of hourly replications performed on time

99.00% of hourly replications performed on time

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Back-Up Retrieval

Timeliness of Back-Up Retrieval

97.00% of retrievals within 24 hours of the request or incident

99.00% of retrievals within 24 hours of the request or incident

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Back-up Quality Quality of

Back-Up File 99.90% of back-up files without errors

99.99% of back-up files without errors

Sample Audit Monthly and/or during file restoration

10% of applicable billing period cost

25% of applicable billing period cost

PERFORMANCE

MEASURE TITLE

PERFORMANCE

MEASURE

DESCRIPTION

MINIMUM

SERVICE LEVEL

FULL

PERFORMANCE

SERVICE LEVEL

MONITORING

METHOD/ DATA

CAPTURE/ SOURCES

MEASUREMENT

PERIOD

BELOW FULL

PERFORMANCE

DEDUCTION

BELOW

MINIMUM

DEDUCTION

Performance Data

Accurate and timely provision of performance data/ statistics into the

Cloud Management Platform (CMP)

Performance Data provided is

98.00% accurate and

Server Availability alerts provided within 30 minutes upon change in availability status 100.00% of the time.

Performance Data provided is 99.00% accurate and

Server Availability alerts provided within 30 minutes upon change in availability status 100.00% of the time.

Review of performance reports/logs

Monthly and Ad- hoc at

Government Request

10% of applicable billing period cost

25% of applicable billing period cost

Service Catalog Accuracy

Accuracy of Service Catalog item data

98.00% accurate at the item record level

100.00% accurate at the item record level

Periodic Inspection of

100% of data set

Monthly & Ad- Hoc at

Government Request

10% of applicable billing period cost

25% of applicable billing period cost

Service Catalog Updates

Time to make Service Catalog update following authorized change

95.00% of updates made within three days following notification of change

98.00% of updates made within three days following notification of change

Periodic Inspection of

100% of data set

Monthly & Ad- Hoc at

Government Request

10% of applicable billing period cost

25% of applicable billing period cost

Customer Satisfaction –

Training

Measure of end-user/customer training experience

80.00% "very satisfied" or

"satisfied" for training surveys and

80.00% “very satisfied” or

“satisfied” on formal surveys

85.00% or higher "very satisfied" or

"satisfied" for training surveys and

85.00% or higher “very satisfied” or

“satisfied” on formal surveys

100% Inspection of end user training surveys

100% Inspection of formal end-user training surveys

Monthly Per Survey

10% of applicable billing period cost

25% of applicable billing period cost

Service Desk Availability

The percentage of time the Service Desk applications are available

99.90% availability over measurement period

99.99% availability over measurement period

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Incident Notification - Unauthorized

Access

Time to resolve incidents involving:

Individuals gaining unauthorized logical or

Physical access to a judicial network, system or

Application or illicitly accesses data or other resources.

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within two (2) hours of detection and

Customers are notified of Low Impact incidents within 4 hours of detection

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within two (2) hours of detection and

Customers are notified of Low Impact incidents within four (4) hours of detection.

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable

PERFORMANCE

MEASURE TITLE

PERFORMANCE

MEASURE

DESCRIPTION

MINIMUM

SERVICE LEVEL

FULL

PERFORMANCE

SERVICE LEVEL

MONITORING

METHOD/ DATA

CAPTURE/ SOURCES

MEASUREMENT

PERIOD

BELOW FULL

PERFORMANCE

DEDUCTION

BELOW

MINIMUM

DEDUCTION

Incident Notification –

Denial of Service (DoS)

Time to resolve Incidents involving an attack that successfully prevents or impairs the normal authorized functionality of networks, systems, or applications by exhausting resources.

This activity includes either being the victim or participating in the

DoS.

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within 15 minutes of detection and

Customers are notified of Low Impact incidents within one (1) hour of detection.

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within 15 minutes of detection and

Customers are notified of Low Impact incidents within one (1) hour of detection.

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Incident Notification –

Malicious Code

Time to resolve incidents involving successful or attempted installation of malicious software (e.g., virus, worm, trojan, or other code-based malicious entity) that infects an operating system, firmware, or application.

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within two (2) hours of detection and

Customers are notified of Low Impact incidents within four (4) hours or detection.

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within two (2) hours of detection and

Customers are notified of Low Impact incidents within four (4) hours of detection.

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Incident Notification -

Improper Usage

Time to resolve Incidents involving a person performing an action that violates a

Guide policy or otherwise exposed the judiciary to harm, such as: Utilizing peer-to-peer programs within the judiciary network or downloading hacker tools outside of their job requirements.

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within two (2) hours of detection and

Customers are notified of Low Impact incidents within four (4) hours of detection.

Customers are notified of High Impact incidents within 15 minutes of detection and

Customers are notified of

Moderate Impact incidents within two (2) hours of detection and

Customers are notified of Low Impact incidents within four (4) hours of detection

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable

PERFORMANCE

MEASURE TITLE

PERFORMANCE

MEASURE

DESCRIPTION

MINIMUM

SERVICE LEVEL

FULL

PERFORMANCE

SERVICE LEVEL

MONITORING

METHOD/ DATA

CAPTURE/ SOURCES

MEASUREMENT

PERIOD

BELOW FULL

PERFORMANCE

DEDUCTION

BELOW MINIMUM

DEDUCTION

Incident Notification -

Phishing & Spam

Time to resolve Incidents involving investigation of phishing and spam emails.

A successful phish resulting in an account or system compromise will be recategorized accordingly with root cause determination or sub-categorization as phishing.

Spam consists of reported email events that have been deemed non-malicious and closed accordingly.

Customers are notified of High Impact incidents within one (1) hour of detection.

and

Customers are notified of

Moderate Impact incidents within four (4) hours of detection.

and

Customers are notified of Low Impact incidents within eight (8) hours of detection.

Customers are notified of High Impact incidents within one (1) hour of detection.

and

Customers are notified of

Moderate Impact incidents within four (4) hours of detection.

and

Customers are notified of Low Impact incidents within eight (8) hours of detection.

Review of performance reports/logs

Monthly 10% of applicable billing period cost

25% of applicable billing period cost

Security Incident Notification

Time

Time from initial incident detection until

Judiciary Security Operations Center

(JSOC) notification.

If the incident has already been detected by the JSOC, the incident detection by the contractor is not applicable.

100% of Security Incidents reported within 30 minutes of security event detection.

100% of Security Incidents reported within 15 minutes of security event detection.

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Flaw Remediation

Compliance with SI-02 Flaw Remediation of FedRAMP Security Baseline Controls.

0-Day remediations within 48 hours of update approval and

All other remediations within five (5) business days of remediation approval

0-Day remediations within 36 hours of update approval and

All other remediations within three (3) business days of remediation approval

Review of performance reports/logs/change records

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Maintaining a Baseline for the

Asset Management and

Security Software/ Hardware Currency

Measure of security software, appliance, and hardware versions in relation to the manufacturing vendor’s current version

97.00% at N-1 or higher

99.00% at N-1 or higher

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

Timeliness of Security Signature

Measures of updates of intrusion detection signatures.

97.00% at N-1 or higher

99.00% at N-1 or higher

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable billing period cost

IT Refresh and

End-Of-Life Product

Management

Update and Upgrade cycle of the existing HW, SW, and middleware versions in consideration with the manufacturing vendor’s provisioned lifecycle

97.00% at N-1 or higher within 6 months of availability

99.00% at N-1 or higher within 6 months of availability

Review of performance reports/logs

Monthly

10% of applicable billing period cost

25% of applicable

PERFORMANCE

MEASURE TITLE

PERFORMANCE

MEASURE

DESCRIPTION

MINIMUM

SERVICE LEVEL

FULL

PERFORMANCE

SERVICE LEVEL

MONITORING

METHOD/ DATA

CAPTURE/ SOURCES

MEASUREMENT

PERIOD

BELOW FULL

PERFORMANCE

DEDUCTION

BELOW MINIMUM

DEDUCTION

Data/Usage System

Availability

Measure the availability of the reporting system and any associated dashboards/alerting tools

98.60% availability

99.00% availability

Review of performance reports/logs

Monthly 10% of applicable billing period cost

25% of applicable billing period cost

Data/Usage System Accuracy

Measure the accuracy of data provided by dashboards, alerts and other tools used by the AOUSC to manage the

CSP

98.00% accurate at the item record level

100.00% accurate at the item record level

Periodic Inspection of 100% of data set

Monthly and Ad- Hoc at

Government Request

10% of applicable billing period cost

25% of applicable billing period cost

4 SERVICE LEVEL AGREEMENT FRAMEWORK

4.1 Performance Deduction Approach

Each performance period, the Judiciary will evaluate SLA metrics to validate that SLAs either met full performance measures, did not met full performance measures but did meet minimum performance measures, or did not meet minimum performance measures. Some

SLAs may have the same measure for both full performance and minimum performance.

The Judiciary will then authorize payment for services rendered based on the percentage deductions listed in Table 2 - Performance Metrics and Measures.

4.2 Service Level Agreement (SLA) Development and Maintenance

SLAs will be periodically reviewed, annually at a minimum, and may be modified by mutual agreement to better reflect lessons learned during execution. Agreed to changes become effective after 90 days to allow the contractor to realign resources and revise affected processes appropriately for execution against the revised SLAs.