Attachment 31 - EOC-SOP-SBO-22-102-Problem Management.pdf

Problem Management EOC Standard Operating Procedure

April 2022

Produced by:

Enterprise Operations Center Department of Technology Services Administrative Office of the U.S. Courts One Columbus Circle, NE Washington, DC 20544

Notice – Limited Distribution This document is designated Sensitive. Distribution is strictly controlled and further distribution is prohibited. Duplication of this document is prohibited.

FOR OFFICIAL USE ONLY

Problem Management: EOC Standard Operating Procedure

Final Authorization and Approval Signature

Office Chief, Enterprise Operations Center

Deputy Office Chief, Enterprise Operations Center

Division Chief, Enterprise Operations Center Infrastructure Operations Division

Division Chief, Enterprise Operations Center Support Services Division

Table of Contents

1 Overview

2 Guiding Principles

3 Process Overview

4 Roles and Responsibilities

4.1 RACI Definitions

4.2 RACI Matrix

5 Problem Identification

6 Prioritization and Categorization

6.1 Urgency and Impact

6.2 Priority/Impact Matrix:

6.3 Problem Record Categorization

7 Problem Investigation

7.1 Lean Six Sigma (LSS)

7.2 8-Step Problem Solving Process

8 Quality Assessment

8.1 Specific Measurements to be used

9 Revision History

1 Overview This document details the Enterprise Operations Center (EOC) procedures for managing problems to eliminate or mitigate incidents and their impacts to Information Technology (IT) services.

This document applies to all EOC personnel and participating partners who are responsible for identifying, investigating, and solving problems or operating/overseeing the Problem Management (PRM) process. The EOC Infrastructure Operations Division (IOD), EOC Support Services Division (SSD), and EOC Continual Service Improvement (CSI) team will review this document once per fiscal year.

NOTE: A problem is something that caused, or potentially caused, one or more incidents.

Problems can be identified proactively or reactively (in response to a single significant incident or multiple similar incidents).

2 Guiding Principles The guiding principles for the PRM process include:

• One standard PRM process is used to define the EOC response to problems for all nationally supported services.

• All problem reporting occurs in the EOC Information Technology Service Management (ITSM) system, providing users with a single source of information.

• All problems recorded in the ITSM system are prioritized, and their solutions or outcomes are documented.

• All information added to the problem record uses correct grammar, punctuation, and spelling, and is detailed and relevant.

• All problems subscribe to a standard set of criteria for identifying problem urgency and impact levels that are consistent across the AO.

• Problem processing and handling aligns with overall Service Level Commitments (SLCs) and objectives to ensure user expectations are met. Problem management aligns with Operational Level Agreements (OLAs) that may be in place with the Program Office (PO).

• Before officially declaring problems resolved/closed, problem owners document one of the following:

− Confirmation that the solution was successful.

− Rationale behind the decision not to pursue a solution.

• A Known Error Database (KEDB) is maintained by CSI to provide up-to-date information on issues that have previously been identified.

• CSI audits problem records and the KEDB daily to ensure they were entered and categorized correctly, and the relevant information was clearly documented for future knowledge articles.

• CSI facilitates problem meetings, investigations, and analysis to ensure effective resolutions are found.

3 Process Overview The Problem Management process phases are as follows:

• New – Problem pending review to confirm suitability

• Assess – Problem assessment by a problem manager to confirm suitability, determine assignments, and confirm priority

• Root Cause Analysis – Problem under active investigation and can no longer be canceled.

Investigation should result in a fix applied or a risk accepted

• Fix in Progress – Proposed fix is being implemented

• Resolved – Fix was implemented successfully OR risk was accepted

• Closed – All documentation was updated and the Problem was permanently fixed

4 Roles and Responsibilities The various roles for the PRM process are:

Role Definition Tier 1 The primary, initial support contact for customers of a service. Typically, staff at the National

Service Desk (NSD).

Problem Owner/Analyst

The group/person responsible for:

• Developing and maintaining the system, service, or process with which a problem record is associated.

• Resolving an identified problem.

Leadership The person or group of persons with appropriate authority to make decisions about how to proceed with a given course of action regarding proposed solutions.

Problem Manager

A member of the CSI team assigned to facilitate problem management activities and verify compliance with the problem management process.

4.1 RACI Definitions

The EOC uses a RACI matrix to lay out the responsibilities for each role and activity. The following table defines the elements of this RACI matrix:

Responsible (R) Those who do the work to achieve the task. There is at least one role with a participation type of responsible, although others can be delegated to assist in the work required.

Accountable (A) The one ultimately answerable for the correct and thorough completion of the deliverable or task, and the one who delegates the work to those responsible. In other words, an accountable must sign off (approve) work that responsible provides. There must be only one accountable specified for each task or deliverable.

Consulted (C) Those whose opinions are sought, typically subject matter experts; and with whom there is two-way communication.

Informed (I) Those who are kept up-to-date on progress, often only on completion of the task or deliverable; and with whom there is just one-way communication.

4.2 RACI Matrix

The Problem Management RACI matrix is the same for all priority levels:

Activity Tier 1 Problem Owner

Leadership Problem Manager (CSI)

1. Detect Problem R R R, I R, A

2. Log Problem R R I R, A

3. Categorize and Prioritize R R R, A

4. Investigate and Update Problem Record R R C, A

5. Identify/Document Workaround I, C R C, A

6. Post/Update RCA I R I C, A

7. Create/Update Known Error Record I R R, A

8. Identify Potential Fix/Recommendations R, A C C, I

9. Verify Fix Success I R I I, A

10. Update Problem Record R C, A

11. Review/Audit Problem C, I R, A

Activity Tier 1 Problem Owner

Leadership Problem Manager (CSI)

12. Close Problem Record C, I R, A

5 Problem Identification Problem identification may occur through several different channels. The primary ways to identify problems include:

• Proactively reviewing data captured by CSI to detect potential problems before they impact service delivery

• Performing trend analysis of incident records

• Detecting duplicate and recurring issues Identifying during major incident management that an incident could recur

• Analyzing information received from suppliers and partners

• Analyzing information received from internal software developers, test teams, project teams, and the NSD

• Analyzing available feedback from customers through formal and informal channels

6 Prioritization and Categorization Problem prioritization and categorization aligns with the Incident Management process prioritization and categorization. During the course of problem investigation, Problem Owners, in collaboration with Problem Managers, can change the priority and category when new information indicates such changes are merited.

6.1 Urgency and Impact

The EOC uses urgency and impact to determine priority:

Metric Measured By

Impact How large the interruption is to the business areas:

• High/Extensive/Widespread: There is a Judiciary-wide issue with a service (all users are impacted)

• Medium/Significant/Large: There is an issue that is impacting an entire court location.

• Low/ Moderate/Limited: There is a single or multi-user issue for a service that is not significant

Urgency The speed at which the incident needs to be resolved:

• High: The business area cannot provide a critical service to judiciary users or the public. For a list of critical services, see the EOC Incident Management SOP, Appendix A.

• Medium: The business area cannot provide a non-critical service (see list below) to judiciary users or the public.

• Low: A single user cannot complete a job function.

6.2 Priority/Impact Matrix:

Priority Impact

High Medium Low

Urgency

High Priority 1 Priority 2 Priority 3

Medium Priority 2 Priority 3 Priority 4

Low Priority 3 Priority 4 Priority 5

6.3 Problem Record Categorization

All problems are categorized by type (such as, Hardware, Software, or Network) Problem categories align with incident record categories to facilitate problem tracking, measurement, and their relationships to Incidents.

7 Problem Investigation The EOC investigates problems to determine their root causes and identify workarounds/solutions. CSI facilitates problem investigations by leveraging Lean Six Sigma (LSS) methodologies and the 8-step problem solving process to enable identification of root causes and potential solutions.

7.1 Lean Six Sigma (LSS)

LSS is a methodology used for solving problems and improving processes by following five phases:

Define •Define the problem

Measure •Map out the current process

Analyze •Identify the cause of the problem

Improve •Implement and verify the solution

Control •Maintain the solution

7.2 8-Step Problem Solving Process The 8-step problem solving process is as follows:

8 Quality Assessment A critical component of assessing quality to ensure the organization successfully meets service level targets is for the IT organization to hold itself accountable for deviations from acceptable performance. This is accomplished by producing meaningful reports, studying Key Performance Indicators (KPIs), looking for trends, and taking measures to maintain Acceptable Quality Levels (AQLs). As a result, CSI creates reports as dashboards and reviews them regularly in accordance with the measurement plan and KPI Library that CSI maintains. CSI also provides regular process quality and performance assessments as required by EOC leadership.

1. Clarify The Problem

2. Break Down The Problem/ Identify Performance Gaps

3. Set Improvement Target

4. Determine Root Causes

5. Develop Countermeasures

6. See Countermeasures Through

7. Confirm Results & Process

8.Standardize Successful Processes

Observe

Orient

Decide

Act

Plan

Act

Check

Do

8.1 Specific Measurements Used

CSI is responsible and accountable for tracking/reporting basic demographic data (such as number of problems opened/closed, open and closed dates, and durations) and the following to measure Problem Management activities:

Measure Description Granularity Calculation Frequency

Problem Velocity

Problem Velocity measures how many incidents per month an open Problem is costing us. It’s calculated by dividing the number of linked incidents by the number of months the Problem has been open.

• Calculated at the individual problem record level

• Also calculated for all open problems collectively (for all groups, or by service/system)

Linked Incidents/months open = Problem Velocity

Monthly

Number of actions taken

Count of the number of Corrective Actions or Improvements resulting from Problem Management activities

• All Problems (COUNT(Corrective Actions))+(COUNT(Improvements)) = Number of Actions Taken

Monthly

Percentage of Problems with Root Cause Identified

The percentage of open Problems which have a Root Cause identified

• All Problems ((COUNT(Problems with Root Cause))/(COUNT(Problems)))*100 = % with Root Cause

Monthly

Percentage of Problems with Workaround Identified

The percentage of open Problems which have a Workaround identified

• All Problems ((COUNT(Problems with Workaround))/(COUNT(Problems)))*100 = % with Workaround

Monthly

Revision History Describe each change so the reader only needs to focus on revisions.

Version Date Revision

0.5 April 20, 2022 Final Draft