Attachment_J.1_Appendix_C_ITIL_Key_Performance_Indicators_and_Metrics.pdf

Appendix C

U.S. Army Regional Cyber Center – CONUS (USARCC-C)

ITIL Key Performance Indicators and Metrics

Process Plan

27 December 2016

Document Control Number: XXXXXXX Version Number: 1.4

Table of Contents

1 INTRODUCTION

1.1 PROPONENT

1.2 ACRONYMS

1.3 REFERENCES

2 METRIC DEFINITIONS

2.1 SERVICE METRICS

2.2 TECHNOLOGY METRICS

2.3 PROCESS METRICS

3 EVENT MANAGEMENT

3.1 RELIABLE AUTO-GENERATED INCIDENTS

3.1.1 Criteria

3.2 CIRS CAPTURED BY EVENT MANAGEMENT

3.2.1 Criteria

3.3 AUTO-GENERATED VS. USER-INITIATED INCIDENTS

3.3.1 Criteria

3.4 AUTO-GENERATED SECURITY EVENTS

3.4.1 Criteria

4 INCIDENT MANAGEMENT

4.1 INCIDENT RESPONSE

4.1.1 Criteria (Critical Response)

4.1.2 Criteria (High and Medium Response)

4.1.3 Criteria (Low Response)

4.2 INCIDENT RESOLUTION

4.2.1 Criteria (Critical Resolution)

4.2.2 Criteria (High and Medium Resolution)

4.2.3 Criteria (Low Resolution)

5 PROBLEM MANAGEMENT

5.1 PROBLEM INVESTIGATIONS GENERATE KNOWN ERRORS

5.1.1 Criteria

6 ACCESS MANAGEMENT

6.1 ACCOUNT REQUEST RESOLUTION TIME

6.1.1 Criteria

6.2 INFORMATION SECURITY TRAINING

6.2.1 Criteria

7 CHANGE MANAGEMENT

7.1 SUCCESS RATE (PRE-IMPLEMENTATION)

7.1.1 Criteria

7.2 SUCCESS RATE (EXECUTION)

7.2.1 Criteria

7.3 SUCCESS RATE (POST-IMPLEMENTATION)

7.3.1 Criteria

8 SERVICE ASSET & CONFIGURATION MANAGEMENT

8.1 CMDB ACCURACY

8.1.1 Criteria

8.2 VENDOR MAINTENANCE

8.2.1 Criteria

9 INFORMATION SECURITY MANAGEMENT

9.1 SYSTEM AND INFORMATION INTEGRITY

9.1.1 Criteria

10 IT SERVICE CONTINUITY MANAGEMENT

10.1 CONTINGENCY PLAN TESTING

10.1.1 Criteria

List of Tables

TABLE 1: EVM – RELIABLE AUTO-GENERATED INCIDENTS

TABLE 2: EVM – CIRS CAPTURED BY EVENT MANAGEMENT

TABLE 3: EVM – AUTO-GENERATED VS. USER-INITIATED INCIDENTS

TABLE 4: EVM – RELIABLE AUTO-GENERATED SECURITY EVENTS

TABLE 5: IM – CRITICAL INCIDENT TICKET RESPONSE

TABLE 6: IM – HIGH AND MEDIUM INCIDENT TICKET RESPONSE

TABLE 7: IM – LOW INCIDENT TICKET RESPONSE

TABLE 8: IM – CRITICAL INCIDENT TICKET RESOLUTION

TABLE 9: IM – HIGH AND MEDIUM INCIDENT RESOLUTION

TABLE 10: IM – LOW INCIDENT TICKET RESOLUTION

TABLE 11: PM – PROBLEM INVESTIGATIONS GENERATE KNOWN ERRORS

TABLE 12: ACM – ACCOUNT REQUEST RESOLUTION TIME

TABLE 13: ACM – INFORMATION SECURITY TRAINING

TABLE 14: CHM – SUCCESS RATE (PRE-IMPLEMENTATION)

TABLE 15: CHM – SUCCESS RATE (EXECUTION)

TABLE 16: CHM – SUCCESS RATE (POST-IMPLEMENTATION)

TABLE 17: SACM – CMDB ACCURACY

TABLE 18: SACM – VENDOR MAINTENANCE

TABLE 19: ISM – SYSTEM AND INFORMATION INTEGRITY

TABLE 20: ISM – CONTINGENCY PLAN TESTING

DCN: xxxxxxxxxx v1.4 27 December 2016

1 INTRODUCTION

The purpose of this document is capture all defined and agreed key performance indicators (KPIs) and metrics associated with each ITIL process used to support the US Army Regional Cyber Center – CONUS (RCC-C) mission.

1.1 Proponent

The RCC-C Program Manager and/or Deputy Program Manager is the proponent for this document.

1.2 Acronyms

Reference (a) provides a list of standard acronyms incorporated in this document by reference.

1.3 References

The following documents serve as references for this ITIL Key Performance Indicators and Metrics Process Plan:

(a) List of Standard Acronyms (DCN 1001785)

(b) Event Management Process Plan (DCN 1002172)

(c) Incident Management Process Plan (DCN 1001308)

(d) Problem Management Process Plan (DCN 1000342)

(e) Access Management Process Plan (DCN 1002136)

(f) Change Management Process Plan (DCN 1000527)

(g) Service Asset & Configuration Management Process Plan (DCN 1002268)

(h) Information Security Management Process Plan (DCN 1002057)

(i) ITIL® Continual Service Improvement

2 METRIC DEFINITIONS

Within the ITIL framework, there are three metric types that an organization typically collects to support assessment of performance and CSI activities. CSI would use these metrics as input in identifying improvement opportunities for each process.

2.1 Service Metrics

These metrics are a measure of the end-to-end service performance. Individual technology and process metrics are used when calculating the end-to-end service metrics.

2.2 Technology Metrics

These metrics are often associated with component and application-based metrics such as performance, availability, etc.

2.3 Process Metrics

These metrics are captured in the form of critical success factors (CSFs), KPIs, and activity metrics for the service management processes and can help determine the overall health of a process.

3 EVENT MANAGEMENT

3.1 Reliable Auto-Generated Incidents

Table 1: EvM – Reliable Auto-Generated Incidents

Type: Process Frequency: Monthly

Measurement

Measure 1 𝑀1 = Number of Incidents resolved/closed/cancelled, generated from an automated source, during the reporting period

Measure 2 𝑀2 = Number of Incidents resolved/closed/cancelled, generated from an automated source, during the reporting period that were not a result of missed polls, improper modeling, duplicates, or missed maintenance mode

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 85% ≥ 85% ≥ 90% ≥ 95%

3.1.1 Criteria

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record status = ‘Resolved’ or ‘Closed’ or ‘Cancelled’ o Incident record submitter = ‘netcool.automation’ o Incident record ‘Last Resolved Date’ >= the first day of the reporting period and <= the last day of the reporting period

Measure 2 o The volume for this measure is determined through manual review of each Incident record from Measure 1

3.2 CIRs Captured by Event Management

Table 2: EvM – CIRs Captured by Event Management

Type: Technology Frequency: Monthly

Measure 1 𝑀1 = Number of CIRs related to RCC-C managed systems during the reporting period

Measure 2 𝑀2 = Number of CIRs related to RCC-C managed systems during the reporting period that were initially detected by an auto-generated Incident

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 80% ≥ 80% ≥ 90% ≥ 95%

3.2.1 Criteria

Measure 1 o The volume for this measure is determined through manual review of each CIR opened during the reporting period

Measure 2 each CIR and associated Incident record.

NOTE: in some cases, the Incident record documented in the CIR is not, in fact, the original. Some analysis of auto-generated Incident records must be done to determine if the RCC-C was aware of a service impact prior to the first user-initiated Incident being opened.

3.3 Auto-generated vs. User-initiated Incidents

Table 3: EvM – Auto-generated vs. User-initiated Incidents

Measure 1 𝑀1 = Number of user-generated Incidents resolved or closed during the reporting period

Measure 2 𝑀2 = Number of user-generated Incidents resolved or closed during the reporting period where an auto-generated Incident was not possible

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 85% ≥ 85% ≥ 92% ≥ 97%

3.3.1 Criteria

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record ‘Service Type’ ≠ ‘User Service Request’ o Incident record status = ‘Resolved’ or ‘Closed’ o Incident record submitter ≠ ‘netcool.automation’ o Incident record ‘Last Resolved Date’ >= the first day of the reporting period and <= the last day of the reporting period

Measure 2 o The volume for this measure is determined through manual review of each

Incident record from Measure 1

3.4 Auto-Generated Security Events

Table 4: EvM – Reliable Auto-Generated Security Events

Measure 1 𝑀1 = Number of Incidents resolved/closed/cancelled, generated from an automated source, during the reporting period

Measure 2 𝑀2 = Number of Incidents resolved/closed/cancelled, generated from an automated source, during the reporting period that were not a result of missed polls, improper modeling, duplicates, or missed maintenance mode

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 90% ≥ 93% ≥ 96% ≥ 99%

3.4.1 Criteria

Measure 1 o Security Event records is assigned to appropriate USARCC-C support groups within Army Cyber Defense Console o Security Event status = ‘Closed/QA Passed’ o Security Even record submitter = ‘ArcSight Rule Fire Export to

Remedy’ o Security Event record ‘Create Date/Time’ = ArcSight Rule Fire End time.

Measure 2 each Security Event record from Measure 1

4 INCIDENT MANAGEMENT

All KPIs for Incident Management are measured utilizing the BMC Remedy ITSM Service Level Management (SLM) module.

4.1 Incident Response

Table 5: IM – Critical Incident Ticket Response

Type: Service Frequency: Monthly

Measure 1 𝑀1 = Number of Critical-priority Incidents opened during the reporting period

Measure 2 𝑀2 = Number of Critical-priority Incidents opened during the reporting period that met established response targets (≤ 1 hour)

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 94% ≥ 94% ≥ 97% ≥ 99%

4.1.1 Criteria (Critical Response)

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record ‘Service Type’ ≠ ‘User Service Request’ o Incident record priority = ‘Critical’ o Service Target title = ‘Critical INC Response 1 Hour’ o Incident record ‘Submit Date’ >= the first day of the reporting period and

<= the last day of the reporting period

Measure 2 o Incident records from Measure 1 where the ‘Measurement Status’ = ‘Met’

Table 6: IM – High and Medium Incident Ticket Response

Measure 1 𝑀1 = Number of High- and Medium-priority Incidents opened during the reporting period

Measure 2 𝑀2 = Number of High- and Medium-priority Incidents opened during the reporting period that met established response targets (High ≤ 2 core hours; Medium ≤ 5 core hours), respectively

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 92% ≥ 92% ≥ 94% ≥ 96%

4.1.2 Criteria (High and Medium Response)

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record ‘Service Type’ ≠ ‘User Service Request’ o Incident record priority = ‘High’ or ‘Medium’ o Service Target title = ‘High INC Response 2 Hrs’ or ‘Medium INC

Response 5 Bus Hr’ o Incident record ‘Submit Date’ >= the first day of the reporting period and

<= the last day of the reporting period

NOTE: while the KPIs for ‘High’ and ‘Medium’ priority Incident response are combined in this document, they will be reported as separate measures.

Table 7: IM – Low Incident Ticket Response

Measure 1 𝑀1 = Number of low priority Incidents opened during the reporting period

Measure 2 𝑀2 = Number of Low priority Incidents opened during the reporting period that met established response targets ≤ 24 core hours

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 92% ≥ 92% ≥ 94% ≥ 96%

4.1.3 Criteria (Low Response)

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record ‘Service Type’ ≠ ‘User Service Request’ o Incident record priority = ‘Low’ o Service Target title = ‘24 Business Hours’ o Incident record ‘Submit Date’ >= the first day of the reporting period and

<= the last day of the reporting period

NOTE: the KPIs for ‘Low’ priority Incident response for these measures will be reported

4.2 Incident Resolution

Table 8: IM – Critical Incident Ticket Resolution

Measure 1 𝑀1 = Number of Critical-priority Incidents resolved or closed during the reporting period

Measure 2 𝑀2 = Number of Critical-priority Incidents resolved or closed during the reporting period that met established resolution targets (≤ 6 hours)

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 94% ≥ 94% ≥ 97% ≥ 99%

4.2.1 Criteria (Critical Resolution)

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record ‘Service Type’ ≠ ‘User Service Request’ o Incident record priority = ‘Critical’ o Service Target title = ‘Critical INC Resolution 6 Hours’ o Incident record ‘Last Resolved Date’ >= the first day of the reporting period and <= the last day of the reporting period

Measure 2 o ‘Vendor Name’ field is blank/null

Table 9: IM – High and Medium Incident Resolution

Measure 1 𝑀1 = Number of High- and Medium-priority Incidents resolved or closed during the reporting period

Measure 2 𝑀2 = Number of High- and Medium-priority Incidents resolved or closed during the reporting period that met established resolution targets (High ≤ 8 core hours; Medium ≤ 24 core hours)

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 92% ≥ 92% ≥ 94% ≥ 96%

4.2.2 Criteria (High and Medium Resolution)

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record ‘Service Type’ ≠ ‘User Service Request’ o Incident record priority = ‘High’ or ‘Medium’ o Service Target title = ‘High INC Resolution 8 Hrs’ or ‘Medium INC

Resolution 24 Bus Hrs’ o Incident record ‘Last Resolved Date’ >= the first day of the reporting period and <= the last day of the reporting period

Measure 2

NOTE: while the KPIs for ‘High’ and ‘Medium’ priority Incident resolutions are combined in this document, they will be reported as separate measures.

Table 10: IM – Low Incident Ticket Resolution

Measure 1 𝑀1 = Number of Low priority Incidents resolved or closed during the reporting period

Measure 2 𝑀2 = Number of Low priority Incidents resolved or closed during the reporting period that met established resolution targets ≤ 80 core hours

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 92% ≥ 92% ≥ 94% ≥ 96%

4.2.3 Criteria (Low Resolution)

Measure 1 o Incident records are assigned to any one of the RCC-C support groups within ITSM o Incident record ‘Service Type’ ≠ ‘User Service Request’ o Incident record priority = ‘Low’ o Service Target title = ‘Low Incident Resolution <80 business Hours’ o Incident record ‘Last Resolved Date’ >= the first day of the reporting period and <= the last day of the reporting period

Measure 2

NOTE: the KPIs for ‘Low’ priority Incident resolution measures will be reported

5 PROBLEM MANAGEMENT

5.1 Problem Investigations Generate Known Errors

Table 11: PM – Problem Investigations Generate Known Errors

Measure 1 𝑀1 = Number of Problem investigations completed during the reporting period

Measure 2 𝑀2 = Number of Problem investigations completed during the reporting period that have a documented root cause and initiated a Known Error

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 90% ≥ 90% ≥ 95% ≥ 99%

5.1.1 Criteria

Measure 1 o Problem records are assigned to any one of the RCC-C support groups within ITSM

Measure 2 o Problem records from Measure 1 where:

‘Resolution’ is not blank/null and is validated to contain a root cause A Known Error record was initiated by and related to the Problem record

NOTE: Analysis of Problem records must be done manually to determine if the appropriate actions were taken to progress through the process.

6 ACCESS MANAGEMENT

6.1 Account Request Resolution Time

Table 12: AcM – Account Request Resolution Time

Measure 1 𝑀1 = Number of account request tickets resolved/closed during the reporting period

Measure 2 𝑀2 = Number of account request tickets resolved/closed during the reporting period that met established resolution targets (≤ 24 core hours)

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 85% ≥ 85% ≥ 90% ≥ 95%

6.1.1 Criteria

Measure 1 o Service Request records are assigned to any one of the RCC-C support groups within ITSM o Service Request record ‘Service Type’ = ‘User Service Request’ o Service Target title = ‘Account Incidents Resolved in Less Than 24 Hours’ o Service Request record ‘Last Resolved Date’ >= the first day of the reporting period and <= the last day of the reporting period

Measure 2 o Service Request records from Measure 1 where the ‘Measurement Status’

= ‘Met’

NOTE: this KPI is measured utilizing the BMC Remedy ITSM Service Level Management module.

6.2 Information Security Training

Table 13: AcM – Information Security Training

Measure 1 𝑀1 = Number of contract personnel assigned to the RCC-C for greater than six months

Measure 2 𝑀2 = Number of contract personnel assigned to the RCC-C for greater than six months that have completed all required training and/or certifications

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Non-Compliant Compliant

< 100% 100%

6.2.1 Criteria

Measure 1 o The number of contract personnel that have been assigned to the RCC-C for greater than six months

Measure 2 o The number of contract personnel from Measure 1 that have completed the following certifications and/or training:

NOTE: DCN 1002838, ‘RCC-C Training Courses and Annual Requirements Checklist’, contains the authoritative list of required training.

7 CHANGE MANAGEMENT

7.1 Success Rate (Pre-Implementation)

Table 14: ChM – Success Rate (Pre-Implementation)

Measure 1 𝑀1 = Number of changes that completed the pre-implementation lifecycle phase during the reporting period

Measure 2 𝑀2 = Number of changes that completed the pre-implementation lifecycle phase during the reporting period where all pre-implementation tasks were completed successfully

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 90% ≥ 90% ≥ 95% ≥ 99%

7.1.1 Criteria

Measure 1 o Change records are assigned to any one of the RCC-C support groups within ITSM

Measure 2 o Change records from Measure 1 where pre-implementation tasks required to prepare the change for introduction into the live environment were completed successfully and the Change Request was approved to advance in the lifecycle

NOTE: Analysis of all pre-implementation tasks must be done manually to determine the percentage within target for this performance measure. All Change Requests that completed this phase of the process during the reporting period will be included, regardless of current status.

7.2 Success Rate (Execution)

Table 15: ChM – Success Rate (Execution)

Measure 1 𝑀1 = Number of changes that completed the execution lifecycle phase during the reporting period

Measure 2 𝑀2 = Number of changes that completed the execution lifecycle phase during the reporting period that were successful

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 92% ≥ 92% ≥ 95% ≥ 99%

7.2.1 Criteria

Measure 1 o Change records are assigned to any one of the RCC-C support groups within ITSM

Measure 2 o Change records from Measure 1 where the ‘Status Reason’ = ‘Successful’ or ‘Successful with Issues’ and the Change Request was approved to advance to the next lifecycle stage

NOTE: All Change Requests that completed this phase of the process during the reporting period will be included, regardless of current status.

7.3 Success Rate (Post-Implementation)

Table 16: ChM – Success Rate (Post-Implementation)

Measure 1 𝑀1 = Number of changes closed during the reporting period

Measure 2 𝑀2 = Number of changes closed during the reporting period where all post-implementation tasks were completed successfully and within 10 business days after the Execution phase

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 85% ≥ 85% ≥ 90% ≥ 95%

7.3.1 Criteria

Measure 1 o Change records are assigned to any one of the RCC-C support groups within ITSM o Change record status = ‘Closed’ o Change record ‘Status History Closed Time’ >= the first day of the reporting period and <= the last day of the reporting period

Measure 2 o Change records from Measure 1 where post-implementation tasks required to close out the change were completed successfully and within 10 business days after the Execution phase

NOTE: Analysis of all post-implementation tasks must be done manually to determine the percentage within target for this performance measure.

8 SERVICE ASSET & CONFIGURATION MANAGEMENT

8.1 CMDB Accuracy

Effective date: 11/26/2012

Table 17: SACM – CMDB Accuracy

Measure 1 𝑀1 = Number of CIs audited during the reporting period

Measure 2 𝑀2 = Number of CIs audited during the reporting period validated as being accurate (no missing or inaccurate information)

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 98% ≥ 98% ≥ 99% 100%

8.1.1 Criteria

Measure 1 o A 10% sampling, by mission area, of RCC-C CIs that are in ‘Deployed’ status

Measure 2 o CIs from Measure 1 that were complete and accurate, i.e. all required attributes were checked and validated during the monthly audit

8.2 Vendor Maintenance

Table 18: SACM – Vendor Maintenance

Measure 1 𝑀1 = Number of CIs audited during the reporting period

Measure 2 𝑀2 = Number of CIs audited during the reporting period validated as having current vendor maintenance

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Marginal Satisfactory Very Good Exceptional

< 98% ≥ 98% ≥ 99% 100%

8.2.1 Criteria

Measure 1 o A 10% sampling of RCC-C CIs that are in ‘Deployed’ status

Measure 2 o CIs from Measure 1 that were checked and validated as having a vendor maintenance contract that is executable and the period of performance has not expired

NOTE: Measure 2 has the following exclusions:

CIs where financial responsibility for lifecycle management resides with an organization other than the RCC-C

CIs where lapses in budget approval caused an otherwise timely procurement action to result in a break in support

9 INFORMATION SECURITY MANAGEMENT

9.1 System and Information Integrity

Table 19: ISM – System and Information Integrity

Measure 1 𝑀1 = Number of vulnerabilities identified through IAVM messages during the reporting period

Measure 2 𝑀2 = Number of vulnerabilities identified through IAVM messages during the reporting period that were appropriately mitigated, POA&M, MAP, or DRA

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Non-Compliant Compliant

< 100% 100%

9.1.1 Criteria

Measure 1 o Number of IAVM messages that apply to operating systems installed on

RCC-C assets where the suspense occurred during the reporting period

Measure 2 o Number of IAVM messages from Measure 1 that were appropriately mitigated, POA&M, MAP, or DRA on or prior to the suspense date

10 IT SERVICE CONTINUITY MANAGEMENT

10.1 Contingency Plan Testing

Table 20: ISM – Contingency Plan Testing

Measure 1 𝑀1 = Number of accreditation packages due contingency plan testing during the reporting period

Measure 2 𝑀2 = Number of accreditation packages due contingency plan testing during the reporting period where the testing was completed

Calculation 𝑀2

𝑀1 × 100% = Percentage within target

Standards

Non-Compliant Compliant

< 100% 100%

10.1.1 Criteria

Measure 1 o Self-explanatory o Self-explanatory