Attachment 28 - EOC-NSD-SOP-SBO-24-102-Incident-Management.pdf

Incident Management EOC Standard Operating Procedure

February 2024

Produced by:

Enterprise Operations Center Department of Technology Services Administrative Office of the U.S. Courts One Columbus Circle, NE Washington, DC 20544

Notice – Limited Distribution This document is designated Sensitive. Distribution is strictly controlled and further distribution is prohibited. Duplication of this document is prohibited.

FOR OFFICIAL USE ONLY

Incident Management: EOC Standard Operating Procedure

Document Control

Version Author Date Revision

2.0 Julie Bruce February 12, Updated Final Authorization and Approval Signatures.

Replaced procedure in the first paragraph with Standard Operating Procedure

(SOP).

Removed EOC and NSD from the second paragraph of the Overview. Added Support Services Division and a reference to the appendix.

Moved the note about the Incident being an event under Guiding Principles and made it a paragraph. Added a sentence about what the ICM process is.

Updated the bullet points under Guiding Principles as follows:

• Added a description of major incidents.

• Revised the bullet point on incident reporting for nationally-supported services.

• Changed all in the bullet point on EOC managing, tracking, escalating, closing, and communicating status to the initial.

• Revised the bullet point on incidents to spell out ITSM, remove reference to the

Incident Work Instruction, and combine it with the next bullet.

• Rearranged the content in the bullet point on information added to the incident record for clarity.

• Updated the bullet point on the criterion to change severity to priority, add a reference to section 5.1, and spell out AO.

• Combined three bullet points on incident processing and handling and added a reference to section 5.2.

• Added bullet points about business hours and what happens when customers stop responding.

• Revised the bullet point about before officially declaring an incident resolved.

• Added a sub bullet about the Tier 2/3 Incident Owner ensuring incident tasks are closed.

• Made the bullet point about resolving incidents a sub bullet point, and then revised the sentence.

• Added a bullet point about NSD/NMF reviewing resolved incidents.

• Added a sentence about the complex nature of Major Incidents.

• Updated the bullet point about reviewing resolved incidents and added information to address action on non-responsive customer.

• Updated the bullet point on incident record auditing, changing the wording from passive voice to active voice Deleted sections on Logging and Prioritization and ICM Integration.

Updated the Process Overview section as follows:

• Updated step labels for consistency.

• Step 5: Changed Workaround in KEDB to Solution in Knowledge.

• Steps 6 and 10: Changed Apply workaround to Apply Solution.

• Step 8: Added a call out about Tier 2/1 On-Call personnel being alerted for P1/2 incidents.

• Step 10: Added a call out about the knowledge management process.

• Added a paragraph about Incident Owners needing to perform additional steps to verify resolution before marking Major Incidents as resolved.

Changed Incident Detection, Recording, and Classification RACI under Roles and Responsibilities to Incident Processing RACI.

Updated the Incident Processing RACI as follows:

• Made table line items parallel.

• Spelled out NMF.

• Added (System, Service, Impact, and Urgency) next to Classification.

• Added an R to the EOC NSD and Tier 2/3 columns for Row 1.

• Added an A to the EOC NSD column for Row 8.

• Changed the Tier 2/3 column for Row 8 from A to R.

• Removed Tier 2 from the Activity column for rows 9-11.

Deleted the Procedures, Key Performance Indicators, and Related Documentation sections and moved the Urgency and Impact, Acceptable Quality Level, Quality Assessment, and Critical Services/Applications sections.

Added an introductory sentence under the Service Level Commitment section.

• Updated the Service Level Commitment lead-in statement.

• Updated the Urgency and Impact table.

Updated the Acceptable Quality Level section as follows:

• Changed acknowledging to accepting.

• Changed 4 workdays to 96 business hours and 5 workdays to 120 business hours.

• Removed the footnote about critical incidents not acknowledged within 15 minutes.

• Removed the Escalation Notification for Non-Responses section.

Updated the Incident Update Frequency Requirements section as follows:

• Added the phrase for adding customer-facing notes to an incident.

• Changed 24 hours to 8 business hours.

• Changed 48 hours to 24 business hours.

• Changed 72 hours to 48 business hours.

Replaced all instances of Acceptable Quality Levels (AQL) with AQL.

Moved the Revision History section to the front of the document.

Updated the Critical Services/Applications as follows:

• Moved ATLAS, DSS, ERS, ESB Web Services, iPACTS, PICTS Mobile, Risk

Assessment, and SIRS to the PACTS row.

• Removed Client Mapping, CoSign, GovDelivery, ITSM, Splunk, and USCOURTS.gov.

• Added NPV and NVTCS.

• Changed JPort to JPort/Ivanti.

Final Authorization and Approval Signature

Office Chief (Acting), Enterprise Operations Center

Division Chief, Enterprise Operations Center, Infrastructure Operations Division

Division Chief, Enterprise Operations Center, Support Services Division

Staff Chief, Continuous Improvement Staff

Table of Contents

1 Overview

2 Guiding Principles

3 Process Overview

4 Roles and Responsibilities

4.1 RACI Definitions

5 Service Level Commitment

5.1 Urgency and Impact

5.2 Acceptable Quality Level

5.2.1 Incident Update Frequency Requirements

6 Quality Assessment

Appendix A: Critical Services/Applications

1 Overview This document details the Enterprise Operations Center (EOC) Standard Operating Procedure (SOP) for managing Incidents to return Information Technology (IT) services to normal as quickly as possible.

This document applies to all personnel who are responsible for restoring services and operating/managing the Incident Management (ICM) process. The EOC Infrastructure Operations Division (IOD), EOC Support Services Division (SSD), and EOC Continual Service Improvement (CSI) team will review this document, including the appendix, once per fiscal year.

2 Guiding Principles An incident is an event that is not part of standard operations and causes or has the potential to cause a service interruption or reduce the quality of a service. A major incident is an incident that results in significant disruption to the business and demands a response beyond the routine incident management process. An incident gets promoted to a major incident when the scope of impact is broad enough to warrant status updates be broadcast to the Judiciary (such as when multiple courts report a similar incident in a short time or the AO validates that an incident is impacting many different users). The ICM process is a defined set of procedures for managing incidents.

The guiding principles for the ICM process include:

• One standard ICM process is defined and used to support all IT service users.

• All incident reporting for nationally-supported services goes through the EOC [through the self-service portal, the National Service Desk (NSD) call queue, or monitoring tool integration], providing users with a single point of contact.

• The EOC manages, tracks, escalates, closes, and communicates status for all incident records, and they are responsible for the initial incident assignments.

• All incidents logged in the IT Service Management (ITSM) system are prioritized, and their solutions are recorded.

• All information added to the incident record contains sufficient details, is relevant, and uses correct grammar, punctuation, and spelling.

• All incidents subscribe to a standard criterion that is used for identifying incident priority levels and escalations (see section 5.1, Urgency and Impact) consistently across the Administrative Office (AO).

• Incident processing and handling aligns with overall Service Level Commitments (SLCs) and objectives (see section 5.2, Acceptable Quality Level) to ensure user expectations are met. Incidents are also managed in accordance with Operational Level Agreements (OLAs) that may be in place with the Program Office (PO).

https://fedcourts.servicenowservices.com/sp https://fedcourts.sharepoint.com/sites/zNSD/Pages/ContactUs.aspx

• If customers stop responding about an active incident, the Incident Owner marks the Incident state as On Hold and applies a reason of Awaiting Caller. If, after three business days of daily attempted contact, there is no customer response, the Incident Owner changes the incident state to Resolved.

• Before officially declaring an incident resolved, the Incident Owner must attempt to make contact with the user to confirm the provided solution addressed the incident. The contact must be documented (as a customer facing note) in the incident (see steps 7 and 11 in section 3 Process Overview).

− The Tier 2/3 Incident Owner must ensure any created incident tasks are closed before resolving the incident; the system will not prevent you from closing the incident if a task is still active. NSD does not use incident tasks.

− The Incident Owner resolves incidents when they are reasonably sure the issue is no longer impacting the customer. Incident closure occurs manually, after the customer confirms there is no longer an impact, or automatically, after seven days (see step 12 in section 3 Process Overview).

− Due to the complex nature of Major Incidents, Incident Owners must verify resolution with at least 20% of any related children records linked to those incidents before marking them as resolved.

• The NSD/NMF will review all resolved incidents and attempt to contact the caller when the notes are unclear if the caller has verified the resolution. After customer acknowledgment, or 7 days after resolution without customer response, the incident will be closed.

• Incident record auditing occurs regularly to ensure incidents are entered and categorized correctly, and resolutions are clearly documented for future knowledge articles.

3 Process Overview

4 Roles and Responsibilities The EOC uses a Responsible, Accountable, Consulted, Informed (RACI) matrix to lay out the responsibilities for each role and activity. The Incident Processing RACI matrix is the same for all priority levels.

4.1 RACI Definitions

The following table defines the elements of the RACI matrix:

The Incident Processing RACI matrix for Incident Management is:

Activity Court EOC NSD NMF Tier 2/3

1) Detect and Record Incident R R, A R R

2) Perform Standard Network Management Facility (NMF) Processes

R, A

3) Notify On-Call R, A

4) Classify Incident (System, Service, Impact, and Urgency) R, A R

5) Investigate and Analyze – Tier 1 R, A

6) Apply Solution– Tier 1 R, A

7) Confirm Incident Resolution – Tier 1

R, A

8) Escalate for further Investigation and Analysis R, A R

9) Investigate and Analyze R, A

10) Apply Solution R, A

11) Confirm Incident Resolution R, A

12) Close Incident R, A

Responsible (R) Those who do the work to achieve the task. There is at least one role with a participation type of responsible, although others can be delegated to assist in the work required.

Accountable (A) The one ultimately answerable for the correct and thorough completion of the deliverable or task, and the one who delegates the work to those responsible. In other words, an accountable must sign off (approve) work that responsible provides. There must be only one accountable specified for each task or deliverable.

Consulted (C) Those whose opinions are sought, typically subject matter experts; and with whom there is two-way communication.

Informed (I) Those who are kept up-to-date on progress, often only on completion of the task or deliverable; and with whom there is just one-way communication.

5 Service Level Commitment EOC uses Urgency and Impact to determine incident priority based on whether the service is considered critical or non-critical (see Appendix A: Critical Services/Applications). The incident priority determines the Acceptable Quality Level (AQL) for incidents.

5.1 Urgency and Impact

The EOC defines impact and urgency as follows:

The following matrix shows how impact and urgency are combined to determine priority:

Priority Impact

High Medium Low

Urgency

High Priority 1* Priority 2* Priority 3

Medium Priority 2* Priority 3* Priority 4

Low Priority 3* Priority 4 Priority 5 *Denotes Major Incident – Requires Communications Process as defined in the Communications SOP

5.2 Acceptable Quality Level

The following table describes the five possible priority levels that the EOC uses for incidents, and it provides the time frames for accepting incidents and restoring services:

Priority Description Target Time to Acknowledge1 Target Time to Restore 1 Critical 15 minutes < 2 hours 2 High 30 minutes < 8 hours 3 Medium 4 business hours < 16 business hours 4 Low 4 business hours < 96 business hours 5 Information 4 business hours < 120 business hours

1. Within this time frame from the initial request

Metric Measured By

Impact How large the interruption is to the business areas:

• High (Extensive/Widespread): There is an issue with a service impacting a majority of the Judiciary.

• Medium (Significant/Large): There is an issue with a service impacting users at different court locations.

• Low (Moderate/Limited): There is an issue with a service impacting a single user or small group of users.

Urgency The speed at which the incident needs to be resolved:

• High: A critical service cannot be provided.

• Medium: A non-critical service cannot be provided.

• Low: Informational.

5.2.1 Incident Update Frequency Requirements

The frequency by which incidents must be updated is critical for ensuring:

• Efficient coordination among key IT support personnel, stakeholders, and program offices.

• Status awareness for Tier 0/Self-Service users.

The following matrix outlines the frequency requirements for adding customer-facing notes to an incident:

Priority Description Incident/Task Status Update Frequency 1 Critical < 1 hours 2 High < 4 hours 3 Medium < 8 business hours 4 Low < 24 business hours 5 Informational < 48 business hours

6 Quality Assessment A critical component of successfully meeting service level targets is for the IT organization to hold itself accountable for deviations from acceptable performance. This is accomplished by producing meaningful reports, studying Key Performance Indicators (KPIs), looking for trends, and taking measures to ensure we consistently meet AQLs. The KPIs related to this SOP are listed in the EOC KPI Library. CSI creates reports as dashboards and reviews them regularly in accordance with the measurement plan. CSI will provide regular process quality and performance assessments as required by EOC leadership.

Appendix A: Critical Services/Applications Application/Function Critical Aspect (What it could possibly mean to users)

EOC Supported Application

Case Management/Electronic Case Files (CM/ECF)

• CM/ECF webpage will not render, resolve, or load or does so at a speed that makes it unusable.

• Notices of Electronic Filing (NEFs) or Notices of Docket Activity (NDAs) not generated or delivered to Internet mail servers.

• Courts unable to process payments using Pay.gov.

Internal or external users unable to log on to CM/ECF.

Cloud/View Virtual desktop environment not available for use.

DCN/Internet Access • Internal websites (like JNet, AOWeb, and internal applications) inaccessible.

• External websites (like USCourts.gov) inaccessible.

• All aspects of the Internet Data Center (IDC) (like Wide Area Network

(WAN) connections, internet access, century link, and AT&T circuits) degraded.

• Express route having latency or utilization issues.

Jury Management System (JMS) Public access to eJuror web applications inaccessible.

National Active Directory (NAD)/ Judiciary Enterprise Network Information Exchange (JENIE)

Problems with authentication to applications.

National Fax (NFAX) • Cannot fax paperwork outgoing/incoming.

National Internet Protocol Telephony (NIPT)

• Unable to place/receive hard phone, desktop, IP Communicator, or Jabber calls.

• IPT - court auto attendance system inaccessible.

• Cisco Unified Contact Center Express (UCCX) - Cisco Agent Desktop

(CAD) system inaccessible (For the 13 courts that use UCCX)

• End users unable to access Call Manager, which impacts the ability to place requests.

• IP Multicast routing protocol and Informacast paging servers not functioning, which impacts emergency and daily paging.

• Cisco Unity Connection (CUC) not functioning, which impacts user ability to utilize voice mail capabilities.

• Cisco Prime Collaboration (CPC) not functioning, which impacts help desk and user ability to manage NIPT devices.

National Password Vault (NPV) • The CyberArk Universal Resource Locator (URL)/web portal is down or inaccessible.

• CyberArk Privileged Session Manager (PSM) is down or inaccessible.

• Users are unable to retrieve passwords from CyberArk

National Video Teleconference Service (NVTCS)

• Unable to place or receive internal or external calls.

• NVTCS - Auto Attendants inaccessible.

• Cisco Meeting Server (CMS) - Users unable to log on to join.uc.uscourts.gov.

• Unable to access Telepresence Management Suite (TMS), which impacts the service for scheduling and monitoring video conference calls. Users unable to log on to schedule or monitor calls.

• Unable to register with TMS, which impacts customer ability to use Alias Dialing.

Office 365 and Outlook • Unable to access Microsoft collaboration tools (Office 365 Teams and e-mail).

• E-mail messages not routing to the proper court personal or external agents.

Probation and Pretrial Services Automated Case Tracking System (PACTS) and PACTS applications, including:

• Access to LAw enforcement

Systems (ATLAS)

• Decision Support System (DSS)

• Electronic Reporting System

(ERS)

• Enterprise Service Bus (ESB)

Web Services

• iPACTS

• Psychological Inventory of

Criminal Thinking Styles (PICTS) Mobile

• Risk Assessment

• Safety and Information

Reporting System (SIRS)

• Inability to access the application.

• Data returned from the application is incorrect

Public Access to Court Electronic Records (PACER)

• PACER inaccessible.

• Unable to log on to PACER.

• Unable to use PACER Case Locator (PCL).

• PACER support function and call center for PACER inaccessible.

Remote Access Authentication

(RAA)

• RAA applications (like JPort/Ivanti and Cisco Any Connect) will not allow access to the Virtual Private Network/Data Communications Network (VPN/DCN), which causes issues connecting to other applications/services.

• DUO/Two Factor Authentication is down, which prevents users from accessing the DCN/VPN.