PL 004 - updated.pdf

MMRP

Operations & Maintenance

Incident Management Process

Version 4.0

Last Revised: 6/23/2020

Prepared by:

South Carolina Department of Health and Human Services (SCDHHS)

Member Management Replacement Program (MMRP)

O&M Incident Management Process V3.1

6/23/2020 Page 2 of 37

Table of Contents

1.0 Introduction

1.1 Document Review Plans

1.2 Document Location

1.3 Scope

1.4 Objectives

1.5 Referenced Documents

1.6 Acronyms/Glossary

1.7 Roles and Responsibilities

2.0 Incident Severity Levels

2.1 Severity Level Definitions

2.2 Severity Level Update Criteria

2.3 Severity Level Upgrade or Downgrade

3.0 Incident Resolution

3.1 SCDHHS Response to Resolved Tickets

3.2 Cúram Incident Log

3.3 Root Cause Analysis Management Process

3.3.1 RCA Process

3.3.2 RCA Process Narrative

3.3.3 RCA Timelines

3.4 Escalation of Unresolved Tickets

3.5 Re-opening Incident Tickets

4.0 Incident Management Process High Level Flow

4.1 Incident Management High Level Process Flow

4.2 Incident Management High Level Process Flow Narrative

4.3 Issue Resolution Sub-Process Flow

4.4 Issue Resolution Sub-Process Flow Narrative

5.0 Stakeholder Incident Notification Process

5.1 Stakeholder Notification for Severity 1 Incidents Process Flow

6/23/2020 Page 3 of 37

5.2 Stakeholder Notification for Severity 1 Incidents Process Flow Narrative

5.3 Stakeholder Notification for Severity 2 or 3 Incidents Process Flow

5.4 Stakeholder Notification for Severity 2 or 3 Incidents Process Flow Narrative

6.0 Tools, Templates and Forms

6.1 Templates and Forms

6.2 Tools

7.0 Appendix

Appendix 7.1 Format of Email to Listserv

Appendix 7.2 Pre-approved Unplanned Outage Message (Citizen Portal) for scdhhs.gov

Appendix 7.3 Pre-approved System Available Message after Unplanned Outage (Citizen

Portal) for scdhhs.gov

Appendix 7.4 Contact Methods for 3rd Parties

Appendix 7.5 MMRP CGIS Problem Escalation Process

8.0 Document Control

8.1 Document Control/Revision History

8.2 Review History

8.3 Approvals

6/23/2020 Page 4 of 37

List of Tables

Table 1: Related Documents/References

Table 2: Severity Level Definitions

Table 3: RCA Process Narrative

Table 4: Incident Management Process Flow Narrative

Table 5: Issue Resolution Sub-Process Flow Narrative

Table 6: Stakeholder Notification for Severity 1 Incidents Process Narrative

Table 7: Stakeholder Notification for Severity 2 or 3 Incidents Process Flow Narrative

List of Figures

Figure 1: RCA Process Flow

Figure 2: Incident Management High Level Process Flow

Figure 3: Issue Resolution Sub-Process Flow

Figure 4: Stakeholder Notification Process for Severity 1 Incidents Process Flow

Figure 5: Stakeholder Notification for Severity 2 or 3 Incidents Process Flow

O&M Incident Management Process V3.3

3/30/2020 Page 5 of 37

1.0 Introduction

This document provides an overview of the services and interfaces related to the Incident

Management Process developed in support of South Carolina Department of Health and Human

Services (SCDHHS) Member Management Replacement Program (MMRP) Operations and

Maintenance (O&M) component.

An incident is any unplanned interruption to an Information Technology (IT) service, or a reduction in the quality of an IT service. The purpose of Incident Management is to restore production systems to normal service operation as quickly as possible to minimize the effect of outages on business operations. This is accomplished by ensuring the incidents are recorded, routed and tracked to closure.

Interruptions to a non-production system are reported via email

(MMRP_PROD_Support@scdhhs.gov) or Jira, as needed (e.g. Prod Support outage).

IBM is the vendor contracted to provide technical O&M services for in scope applications/systems and will be referred to as the O&M Contractor throughout the document.

Clemson University (CU) is the vendor responsible for hosting the MMRP environments and will be referred to as the MMRP Hosting and Infrastructure Contractor throughout the document.

In addition to CU, there are other 3rd party vendors involved in the O&M processes, such as print vendors, call centers, etc.

All roles and titles mentioned in the SCDHHS MMRP O&M Incident Management document will be referring to the SCDHHS (MMRP) roles and titles unless otherwise indicated.

1.1 Document Review Plans

This document will be reviewed and updated as defined below:

• Following a regularly scheduled review, at a minimum every 12 months, and

• As changes or updates occur to correct or enhance information content.

1.2 Document Location

The approved MMRP O&M Incident Management Process will be located in the Operations and

Maintenance library of the MMRP SharePoint site.

1.3 Scope

This process applies to any issue identified by or presented to the O&M Staff, including Severity

1, 2 and 3 MMRP incidents (refer to Severity Level Definitions) for in-scope systems, https://schhs.sharepoint.com/sites/1211207/Operations%20and%20Maintenance/Forms/AllItems.aspx

O&M Incident Management Process V4.0

6/23/2020 Page 6 of 37 applications and tasks supported by the O&M Contractor (Vendor) Team as defined by the contract between the Contractor and SCDHHS. The list of in scope applications/systems supported by the O&M Contractor Team is located on the Application List on the MMRP

SharePoint site. Incidents impacting the MMRP operations may be the responsibility of the

O&M Contractor Team, the MMRP Hosting and Infrastructure Contractor, or a 3rd Party vendor, therefore an Incident Owner will be determined for each incident.

This process also addresses the appropriate action for other incidents reported to the O&M

Team (SCDHHS and/or Contractor), such as incidents related to servers or connectivity. Such incidents will be referred to the Information Technology Services Desk (ITSD).

This process starts when:

A potential incident is identified by or communicated to any O&M Staff Member. Potential incidents may be reported by a SCDHHS System User, an O&M Team Member (SCDHHS and/or

Contractor), or other stakeholders.

This process ends when either:

1) The Severity 1, 2, or 3 incident has been resolved, closed and a Root Cause Analysis

(RCA) or After Action Report (AAR) is initiated,

2) The non-MMRP related incident has been referred to ITSD or other appropriate entity, or

3) The Business Case (BC) Process (as defined in the Enhancement Management section of the O&M Manual) has been initiated for non-Severity 1, 2 or 3 incident (issue) and/or steps have been provided and followed to resolve or mitigate the issue.

1.4 Objectives

The objectives of Incident Management are to:

• Recover standard service operations as quickly as possible to minimize disruption to the business caused by an incident.

• Minimize the duration and impact of any service outages associated with an incident by:

o Optimizing time and effort spent resolving incidents, and o Maximizing productivity of resources.

• Reduce the number of incidents by initiating RCA or AAR and the resulting system corrections and/or process improvements to minimize recurrence of similar incidents.

• Outline the steps to be followed from identification of an issue (non-Severity 1, 2 or 3 incident) to communication of a solution and/or initiation of the Enhancement https://schhs.sharepoint.com/sites/1211207/Lists/Application%20List/AllItems.aspx https://schhs.sharepoint.com/sites/1211207/Operations%20and%20Maintenance/Forms/AllItems.aspx?id=/sites/1211207/Operations%20and%20Maintenance/SCDHHS%20MMRP%20Operations%20and%20Maintenance%20Manual%20V3.0.pdf&parent=/sites/1211207/Operations%20and%20Maintenance

6/23/2020 Page 7 of 37

Management Process (as outlined in the O&M Manual located in the Operations and

Maintenance library of the MMRP SharePoint site).

• Incident Management responses may override/supersede other work activities (planned schedules, meetings, daily tasks, etc.) based on the severity of the incident and resources required.

1.5 Referenced Documents

The following documents and sites are important to this process and provide additional information for review. If a link is provided in the table below, a version and date will be provided only when the specific version impacts content within the MMRP O&M Incident

Management Process.

Site/Document Name Version Date last updated

Link

MMRP O&M Manual Click here to access the O&M Manual

Table 1: Related Documents/References

1.6 Acronyms/Glossary

Click here to view the MMRP Acronyms and Glossary. .

1.7 Roles and Responsibilities

The MMRP is responsible for the operation and provision of Cúram and all its supporting systems. The MMRP provides services including IT infrastructure, business analysis, application development, and technical support. The MMRP Roles and Responsibilities spreadsheet is maintained in the MMRP SharePoint site in the Project Management Planning Documents library.

https://schhs.sharepoint.com/sites/1211207/Operations%20and%20Maintenance/Forms/AllItems.aspx?id=/sites/1211207/Operations%20and%20Maintenance/SCDHHS%20MMRP%20Operations%20and%20Maintenance%20Manual%20V3.0.pdf&parent=/sites/1211207/Operations%20and%20Maintenance https://schhs.sharepoint.com/sites/1211207/Operations%20and%20Maintenance/Forms/AllItems.aspx?id=/sites/1211207/Operations%20and%20Maintenance/SCDHHS%20MMRP%20Operations%20and%20Maintenance%20Manual%20V3.0.pdf&parent=/sites/1211207/Operations%20and%20Maintenance https://schhs.sharepoint.com/sites/1211207/Lists/MMRP%20Acronyms%20%20Glossary/AllItems.aspx https://schhs.sharepoint.com/:x:/r/sites/1211207/Project%20Management%20Planning%20Documents/MMRP%20Roles%20and%20Responsibilities.xlsx?d=wda5120a59f654e11a6235ac6a72f6acb&csf=1&e=csT7k9

3/30/2020 Page 8 of 37

2.0 Incident Severity Levels

2.1 Severity Level Definitions

The Severity Levels described in the table below will be used to categorize all incidents submitted from the SCDHHS MMRP User Community or the O&M Team (SCDHHS and/or

Contractor). The Severity Level will be assigned initially by the O&M Team and validated by the

Incident Owner.

Incidents will be treated in severity level order (with 1 being the highest severity level) and within the same severity level, in the order received, unless directed otherwise by the O&M

Team.

Note that the examples in the chart are not an exhaustive list and that the examples are provided to assist with determining the incident severity level.

Sev Severity Description Examples Action Required 1 Production application down or major malfunction resulting in a product inoperative condition. All Users unable to perform their normal functions.

• Unplanned outage(s) impacting users’ productivity (e.g., users unable to perform their job duties).

• Citizen and/or Worker Portal down.

• Slow system performance that impacts all users’ productivity (e.g. users can update evidence but unable to apply changes).

• Production Jobs deemed Critical Level 1 in the OM Prod Job Descriptions spreadsheet.

• Email

MMRP_Prod_Alert@scdhhs.g ov listserv within fifteen (15) minutes (follow email format as shown in Appendix 7.1).

• If outside normal business hours (8:00 am – 5:30 pm), call the On- Call number 864-656-3723.

• Initiate triage call within thirty

(30) minutes of the reported incident.

• Continued updates required every hour thereafter, or more or less frequently as specified during the initial conference call.

2 Critical loss of application functionality or performance resulting in high number of users unable to perform their normal functions. Major feature/product failure; inconvenient workaround or no workaround exists.

• Slow system performance that is not impacting entire state personnel. Minimal impact to user productivity (e.g. Work can still be done, but at a slower pace than normal).

• FFM Production.

• Person Composite Service.

• Interfaces (e.g., Federal HUB, OnBase, MEDS, MMIS).

• Production Jobs deemed Critical Level 2 in the O&M Prod Job Descriptions spreadsheet.

• Email MMRP_Prod_Alert@scdhhs.gov listserv within thirty (30) minutes (follow email format as shown in Appendix 7.1).

• If outside normal business hours call the On-Call number 864- 656-3723.

• Initiate conference call if requested by O&M Leadership.

• Continued updates required every hour thereafter.

https://schhs.sharepoint.com/:x:/r/sites/1211207/Production%20Support/O-M%20Prod%20Job%20Descriptions.xlsx?d=wd2260264df0e4c449740e205c35bf96a&csf=1&web=1&e=PBjAYd https://schhs.sharepoint.com/:x:/r/sites/1211207/Production%20Support/O-M%20Prod%20Job%20Descriptions.xlsx?d=wd2260264df0e4c449740e205c35bf96a&csf=1&web=1&e=PBjAYd https://schhs.sharepoint.com/:x:/r/sites/1211207/Production%20Support/O-M%20Prod%20Job%20Descriptions.xlsx?d=wd2260264df0e4c449740e205c35bf96a&csf=1&web=1&e=PBjAYd https://schhs.sharepoint.com/:x:/r/sites/1211207/Production%20Support/O-M%20Prod%20Job%20Descriptions.xlsx?d=wd2260264df0e4c449740e205c35bf96a&csf=1&web=1&e=PBjAYd

6/23/2020 Page 9 of 37

Table 2: Severity Level Definitions

The program is usable but severely limited.

3 Moderate loss of application functionality or performance resulting in multiple users unable to perform their normal functions.

Minor feature/product failure, convenient workaround exists/minor performance degradation/not impacting entire production system.

• Any other conditions that do not fall under Levels 1 and 2 above.

• Production Jobs that are not deemed Critical Level 1 and 2 in the Prod Job Descriptions spreadsheet.

• Any incidents related to Non- Production (Dev Environment or higher) will be treated as a Severity 3.

• Email MMRP_PROD_Support@scdhhs.

gov listserv within one (1) hour of reported incident (follow email format as shown in Appendix 7.1).

• If outside normal business hours call the On-Call number 864- 656-3723.

• Initiate conference call if requested by O&M Leadership.

• Continued updates every two (2) hours thereafter.

https://team.scdhhs.gov/pmo/ProjectRepository/1211207/_layouts/xlviewer.aspx?id=/pmo/ProjectRepository/1211207/Production%20Support/O-M%20Prod%20Job%20Descriptions.xlsx&Source=https%3A%2F%2Fteam%2Escdhhs%2Egov%2Fpmo%2FProjectRepository%2F1211207%2FProduction%2520Support%2FForms%2FAllItems%2Easpx&DefaultItemOpen=1

3/30/2020 Page 10 of 37

2.2 Severity Level Update Criteria

Updates related to Severity 1, 2 or 3 incidents must be provided according to the “Action

Required” column in the Severity Level Definition chart. The updates must include what caused the issue (if available), what has been and is being done to resolve the issue and the timeline for fully resolving the issue. Each update must vary from the previous update provided to

SCDHHS and should include the current activities being performed, the estimated timeline to complete those activities and the next steps.

2.3 Severity Level Upgrade or Downgrade

The severity of an incident is based on multiple factors, including how widespread the problem is, the impact on users, etc. In some cases, further information may lead to the severity level being upgraded or downgraded. After receiving and investigating the incident, the Incident

Owner may determine that the incident was flagged with the incorrect severity level and may request the upgrade or downgrade of the incident’s severity level, as appropriate. The

Production Support Manager must approve an upgrade or downgrade of the severity level.

Whenever there is a disagreement regarding issue severity or issue resolution status between

SCDHHS and the Incident Owner, the team member(s) will strive to work out the problem internally. As needed, the severity level decision will be escalated to the Program Manager for resolution.

If the severity level is upgraded, the “Action Required” column in the Severity Level Definition chart, including timelines, for the new severity level applies immediately upon the change of the severity level in Jira. Effort expended while at the previous (lower) severity level is not considered toward the required action.

If the severity level is downgraded, the “Action Required” column in the Severity Level

Definition chart, including timelines, for the new severity level applies immediately upon the change of the severity level in Jira and effort expended while at the previous (higher) severity level is considered toward the required action.

3/30/2020 Page 11 of 37

3.0 Incident Resolution

3.1 SCDHHS Response to Resolved Tickets

The O&M Team will be notified via email by the appropriate distribution group including the distribution group originally used for notification or incident and Jira when incidents have been resolved. Upon notification, the Eligibility and Enrollment (E&E) Systems Helpdesk Manager or designee will validate that the incident has been successfully resolved. There may be instances when the O&M Team may not be available to immediately confirm the solution for all open

Helpdesk tickets related to the incident.

In the event the Incident Owner does not have access to Jira, the O&M Contractor Team will update Jira as needed.

3.2 Cúram Incident Log

The Production Support Manager enters the incident information into the Cúram Incident Log by the end of the business day following successful validation of the system resolution.

3.3 Root Cause Analysis Management Process

Root Cause is defined as the underlying or initiating cause of one or more incidents. The purpose of the Root Cause Analysis (RCA) Management Process is to determine and resolve the root causes of incidents and to prevent recurrence of incidents related to these root causes.

The RCA Management Process will be used to capture, evaluate and manage root causes of incidents in production.

Note: The MMRP Hosting and Infrastructure Contractor and 3rd Party Vendor may use an AAR in lieu of the RCA Template. However, the RCA Management Process must be followed.

The objectives of RCA Management Process are to:

• Determine the root cause of incidents,

• Take actions to prevent recurrence of incidents,

• Manage the lifecycle of all root causes,

• Minimize the adverse impact of incidents on the IT solutions and business processes, and

• Keep a record of root causes for reporting purposes and to integrate with other processes to drive continuous improvement.

https://schhs.sharepoint.com/sites/1211207/Lists/ACCESS%20Outage%20Log/AllItems.aspx

6/23/2020 Page 12 of 37

3.3.1 RCA Process

Figure 1: RCA Process Flow

3.3.2 RCA Process Narrative

Steps Task/Process Task

ID(s)

Narration

1 Incident identified 1.0 The process begins after the incident has been identified following the Incident

Management Process. Follow Step 2.

2 Perform RCA 1.1 The Incident Owner performs the RCA activities, following the RCA Timelines, and enters the results in the preliminary/draft

RCA Template (or AAR). Follow Step 3.

3 Approve RCA? 2.1 The O&M Team reviews the RCA results and determines whether they are complete and comes to a reasonable conclusion about the root cause of the incident.

https://schhs.sharepoint.com/:w:/r/sites/1211207/Templates/O-M%20Templates/RCA%20Template%20V2.docx?d=wa3807d9499d0467889895329f22e7118&csf=1&web=1&e=ptDWpI

6/23/2020 Page 13 of 37

If the O&M Team approves the RCA, follow

Step 4.

If the O&M Team does not approve the

RCA, follow Step 2.

4 Remediation needed? 2.2 Remediation actions may not be needed if the RCA was resolved during the Incident

Management Process or if the incident occurred during an event that will not or is extremely unlikely to happen again.

If the O&M Team determines that remediation is needed, follow Step 5.

If remediation is not needed, follow Step 9.

5 Develop Remediation Plan 1.2 The Incident Owner (in coordination with appropriate teams/team members) develops a remediation plan to permanently prevent the RCA from reoccurring and posts the remediation plan in the Jira Incident ticket . Follow Step 6.

6 Remediation Plan Approved? 2.3 The O&M Team reviews the remediation plan to determine if it is the best method to prevent the root cause from reoccurring.

If the plan is approved, follow Step 7.

If the plan is not approved, follow Step 5.

7 Determine if want to pursue remediation

2.4 The O&M Team will determine whether to

implement the remediation plan or not.

If the O&M Team decides to pursue the remediation, follow Step 8.

If the O&M Team decides not to implement, follow Step 9.

8 Pursue remediation 2.4

1.4

1.5

2.5

The Incident Owner (in coordination with appropriate teams/team members) and

O&M Team will follow the Enhancement

Management Process (as outlined in the

O&M Manual).

Note: The intake process would occur via

“Analysis Intake Request” in Service

Manager.

6/23/2020 Page 14 of 37

The O&M Team will update the Jira ticket with the final results of the RCA. Follow

Step 9.

9 Close Jira RCA Task 2.6

2.7

The O&M Team will close the Jira ticket

(which includes the RCA activities) . The

O&M Team will communicate the outcome to the O&M Contractor Team and any other affected parties.

Note: “Closed” means that the RCA was provided and accepted by SCDHHS.

Table 3: RCA Process Narrative

3.3.3 RCA Timelines

Upon resolution of any Severity 1, 2 or 3 issue, an RCA report (or AAR) must be provided to the

O&M Team as outlined below:

• A preliminary/draft RCA must be provided via Jira at an agreed upon timeframe (as set during the triage call) explaining what caused the issue, what is/was planned to resolve the issue and what steps are being taken to ensure the issue does not occur again.

• A detailed RCA report must be provided in accordance with the timelines listed below and must include an explanation of what was done to resolve the issue and what steps are being taken to ensure the issue does not occur again. (Refer to the RCA Template) o A draft Detailed RCA for Severity 1 incidents must be completed within two (2) business days of incident resolution. The final Detailed RCA must be provided within thirty (30) calendar days.

o A draft Detailed RCA for Severity 2 and 3 incidents must be completed within five

(5) business days of incident resolution. The final Detailed RCA must be provided within thirty (30) calendar days.

o The Incident Owner may request additional time to complete the RCA by emailing the Production Support Manager at least twenty-four (24) hours prior to the due date. The reason for the delay and the date the RCA will be delivered must be included in the request. The Production Support Manager will respond to the request via email and will notify others of the delay as appropriate.

3.4 Escalation of Unresolved Tickets

Whenever there is a disagreement regarding an issue resolution status between SCDHHS and the Incident Owner, the team member(s) will strive to work out the problem internally. If escalation is required, the following steps are to be followed:

https://schhs.sharepoint.com/:w:/r/sites/1211207/Templates/O-M%20Templates/RCA%20Template%20V2.docx?d=wa3807d9499d0467889895329f22e7118&csf=1&web=1&e=ptDWpI

6/23/2020 Page 15 of 37

• If an incident is not resolved within twelve (12) hours of being designated a Severity 1, the Production Support Manager and O&M Contractor Delivery Manager and/or Third

Party POC will meet to resolve the issue. The issue will be escalated to the Program

Director and/or Executive Oversight Committee (EOC), as needed.

o Additional time may be allotted by O&M Leadership in response to a Resolution

Plan that outlines the reason additional time is needed and a timeline for implementation of the approved solution.

• If a Resolution Plan has not been presented within twenty-four (24) hours of an incident being designated a Severity 2, the Production Support Manager and O&M Contractor

Delivery Manager and/or Third Party POC will meet to determine an acceptable plan and timeline. As needed, the issue will be escalated to the Program Director and/or

EOC.

• If a resolution plan has not been presented within forty-eight (48) hours of an incident being designated a Severity 3, the Production Support Manager and O&M Contractor

Delivery Manager and/or Third Party POC will meet to determine an acceptable plan and timeline. As needed, the issue will be escalated to the Program Director and/or

EOC. The resolution plan will be added to the Jira ticket.

• During any incident resolution, the O&M Contractor Team and/or Third Party POC agrees to provide support relating to items not in dispute to the extent practical pending resolution of the incident.

3.5 Re-opening Incident Tickets

Incident tickets will not be reopened. In the event an incident is deemed unresolved after closure, a new incident ticket will be created, and the first ticket number will be included in the details of the new ticket.

6/23/2020 Page 16 of 37

4.0 Incident Management Process High Level Flow

4.1 Incident Management High Level Process Flow

Figure 2: Incident Management High Level Process Flow

4.2 Incident Management High Level Process Flow Narrative

ID(s)

Narration

1 The E&E Systems

Helpdesk/O&M Staff receives a potential incident report

1.1 A potential incident is identified by or

communicated to the E&E Systems Helpdesk or other O&M Staff Member. The incident may be reported by a system user, call center representatives, other MMRP (SCDHHS or

Contractor) Staff, or other stakeholders. Follow

Step 2.

2 Is the potential incident

Cúram related?

1.2

1.3

If the potential incident is Cúram related it will be escalated to the E&E Systems Helpdesk

Manager and/or Production Support Manager.

Follow Step 3.

6/23/2020 Page 17 of 37

1.4

1.5

Any issues related to MEDS, MMIS, connectivity (internet), etc. will be referred to the Information Technology Service Desk (ITSD) for investigation and resolution. (If the

Helpdesk is unsure, they will consult with management before taking the next step). The process ends.

3 Incident Severity Level is determined

2.1 The E&E Systems Helpdesk Manager and/or

the Production Support Manager will determine the incident severity level. The

O&M Business Analyst (BA) Manager will be included in the decision making as needed.

Follow Step 4.

4 Severity 1, 2 or 3? 2.2 Is the incident a Severity 1, 2 or 3?

If yes, follow Step 5.

If no, follow Step 6.

5 Confirm Incident Action

Steps

2.10 When an incident is identified as a Severity 1, 2

or 3, a parallel process occurs.

2.11

2.12

1. The E&E Systems Helpdesk Manager or Production Support Manager will follow/initiate the Stakeholder Incident Notification Process.

2.13 2. The E&E Systems Helpdesk Manager or Production Support Manager will create an incident ticket in Jira. Details gathered during the triage will be included in the ticket. Follow Step 7.

6 Follow the MMRP O&M

Issue Resolution Sub-Process

2.3

For escalated issues that do not meet the definition of an incident, the E&E Systems

Helpdesk Manager, Production Support

Manager or O&M BA Manager will initiate the

O&M Issue Resolution Sub-Process, called the

Enhancement Management Process (included in the O&M Manual). Follow Step 19.

7 Incident ticket is acknowledged

3.1 The O&M Contractor Team acknowledges the

incident ticket by updating the “Response

(Acknowledge) Date” field and comments in the Jira ticket. Follow Step 8.

8 Determine Incident Owner 3.2 The O&M Contractor Team will coordinate efforts to determine the Incident Owner. The

Incident Owner may be the O&M Contractor, the MMRP Hosting and Infrastructure

6/23/2020 Page 18 of 37

Contractor, or a Third Party vendor. Follow

Step 9.

Note: The Incident Owner will be captured in

Jira as the assignee owner.

9 Action Required Instructions are followed

4.1 The Incident Owner initiates the Triage calls

and periodically sends email updates via the

MMRP_Prod_Alert@scdhhs.gov listserv referencing the incident per the Action

Required instructions (in the Severity Level

Definitions chart). Follow Step 10.

10 Incident investigations starts 4.2 The Incident Owner investigates the incident and provides periodic email updates via the MMRP_Prod_Alert@scdhhs.gov or MMRP_PROD_Support@scdhhs.gov listserv per the Action Required Instructions (in the Severity Level Definitions chart). The incident should be investigated using information from the initial triage results, Confluence, Wikis, Zabbix, Splunk, System Log files, etc. The Jira ticket should be updated with the results of the investigation. Follow Step 11.

Note: In the event the Incident Owner does not have access to Jira, the O&M Contractor Team will update Jira as needed.

11 Develop Resolution Plan 4.3 The Incident Owner develops the resolution plan to include the preliminary RCA results, planned resolution activities, roles, time frames and user impacts. The Incident Owner updates the status in Jira (if applicable), includes the resolution plan in the comments in Jira, and notifies the E&E Systems Helpdesk Manager and Production Support Manager via Jira and the appropriate listserv (

MMRP_Prod_Alert@scdhhs.gov or

MMRP_PROD_Support@scdhhs.gov) when the plan is ready for review and approval. Follow

Step 12.

Note: In the event the Incident Owner does not have access to Jira, the O&M Contractor Team will update Jira as needed.

mailto:MMRP_Prod_Alert@scdhhs.gov mailto:MMRP_PROD_Support@scdhhs.gov mailto:MMRP_Prod_Alert@scdhhs.gov mailto:MMRP_PROD_Support@scdhhs.gov

6/23/2020 Page 19 of 37

12 Resolution plan is reviewed 2.14 The E&E Systems Helpdesk Manager and

Production Support Manager review the resolution plan documented in the comments in Jira. They will include the O&M Contractor

Team and the O&M BA Team in the review as needed. Follow Step 13.

13 Is approval of the resolution plan needed?

2.15 If the resolution plan includes a negative

impact to users, configuration, or code changes the resolution plan will require approval from the O&M Leadership. Follow Step 14.

If no, follow Step 14.

14 Determine if the Resolution

Plan captures what the business needs

2.16

2.17

Is the Resolution Plan acceptable to the E&E

Systems Helpdesk Manager and the Production

Support Manager?

If yes, the E&E Systems Helpdesk Manager and

Production Support Manager approves the plan. The approval notification should be sent via the MMRP_Prod_Alert@scdhhs.gov or

MMRP_PROD_Support@scdhhs.gov listserv.

Follow Step 15.

Note: If approvals are required, they will be provided and validated in this step at (Task ID

2.17).

If no, the E&E Systems Helpdesk Manager and/or Production Support Manager will add comments to Jira re: concerns with the plan and will send notification via the

MMRP_Prod_Alert@scdhhs.gov or

MMRP_PROD_Support@scdhhs.gov listserv.

Follow Step 11.

15 Update Jira 4.4 The Incident Owner will update the Jira incident ticket status and implement the resolution as per the approved resolution plan.

The Incident Owner and the Production

Support Manager will coordinate O&M

Contractor/Third party involvement to implement resolution as needed. Follow Step

16.

6/23/2020 Page 20 of 37

Note: In the event the Incident Owner does not have access to Jira, the O&M Contractor Team will update Jira as needed.

16 Determine if Release

Management is needed

4.5

3.3

3.4

Does the resolution require Release

Management?

Note: Release Management is required when changes to resolve the incident must be formally committed to the Cúram code base.

If yes, follow/initiate the O&M Release

Management Process (as outlined in the O&M

Manual).

Note: The Incident Owner will follow /initiate the MMRP O&M RCA Management Process upon successful implementation of the incident resolution.

4.6 If no, the Incident Owner will update the Jira

incident ticket status and notify the E&E

Systems Helpdesk Manager and Production

Support Manager via the

MMRP_Prod_Alert@scdhhs.gov or

MMRP_PROD_Support@scdhhs.gov listserv that the solution has been implemented. (In the event the Incident Owner does not have access to Jira, the O&M Contractor Team will update Jira as needed.) Follow Step 17.

17 Validate the Resolution 2.18 The E&E Systems Helpdesk Manager or designee will validate the resolution. Follow

Step 18.

18 Determine if the resolution was successful

2.19

2.20

The E&E Systems Helpdesk Manager or designee will determine if the resolution was successful.

If yes, the E&E Systems Helpdesk Manager acknowledges the success of the resolution and updates the Jira ticket status. Follow Step 19.

2.14 If no, the Incident Owner will have to develop a

new resolution plan. The Jira status will be updated and cycle back through the process until the resolution is determined successful by

6/23/2020 Page 21 of 37 the E&E Systems Helpdesk Manager. Follow

Step 11.

18 Follow the RCA Process 4.7

4.8

The Incident Owner will follow/initiate the RCA

Management Process upon successful implementation of the incident resolution.

19 Determine if Release

Management needed

2.4

2.5

2.7

If yes, follow/initiate the Business Case Process

(Section 7.1.3 in the O&M Manual).

Note: Entry from the RCA Process occurs at

Task ID 2.4.

2.5

2.6

If no, follow the Non-Business Case Work

Requests procedures (Section 7.1.12 in the

O&M Manual). The process ends.

Table 4: Incident Management Process Flow Narrative

4.3 Issue Resolution Sub-Process Flow

The Issue Resolution Sub-Process Flow Narrative (Section 4.4) outlines the steps for addressing issues identified by or communicated to the O&M Team which do not meet the criteria of a

Severity 1, 2 or 3.

6/23/2020 Page 22 of 37

Figure 3: Issue Resolution Sub-Process Flow

6/23/2020 Page 23 of 37

4.4 Issue Resolution Sub-Process Flow Narrative

Steps Task/Process Task ID(s)

Narration

1 Entry from Incident Management Process

2.0 The Sub-Process is called by the Incident

Management Process when an escalated issue is determined to not meet the definition of an incident. Follow Step 2.

2 Review and determine next steps

2.1 The E&E Systems Helpdesk Manager, Production

Support Manager, and/or the O&M BA Manager will review issues and determine the next steps.

Follow Step 3.

3 Is a resolution available? 2.2

The issue may have been presented previously and a resolution, such as a mitigation/ workaround, may already be known and a BC may already be in progress. In this case, follow Step 4.

If no, follow Step 6.

4 Communicate resolution to appropriate O&M Staff

2.3 The E&E Systems Helpdesk Manager, Production

Support Manager, and/or the O&M BA Manager will share information available regarding the issue and provide steps for resolving the problem. Follow Step 5.

5 Follow steps to resolve issue 1.1 1.2

The O&M Staff Member will follow the steps (or workaround) provided to resolve/mitigate the issue. No further action is needed by the O&M Staff. The process ends.

Note: If the issue cannot be fully resolved until after implementation of a BC, the E&E Systems Helpdesk Manager will coordinate the needed post-release activities.

6 Is a Business Case needed? 2.4 If yes, follow Step 9.

If unknown, follow Step 7.

7 Meeting to discuss issue 2.5 If further discussion is needed to determine if a

BC is needed, O&M Leadership will meet to discuss the issue and determine the next steps.

This may be via an existing meeting such as the

Cúram Prod Support Leadership Priorities

Meeting, Production Support – Scrum Meeting, or other internal O&M meetings.

Follow Step 9.

8 Was a decision made to create a Business Case?

2.6 If no, because a BC is not needed (possibly

already in progress or acceptable resolution already in place), follow Step 4.

6/23/2020 Page 24 of 37

2.7 If yes, follow Step 9.

9 Determine if a workaround is applicable

2.8 2.9

If a BC is needed, is there a workaround needed to address the issue in the interim? (Follow internal process for developing and validating workaround.)

If yes, the O&M Leadership will coordinate the development of the workaround. Follow Step

10.

2.13 2.14

If no, the O&M BA Manager will return to the Incident Management Process and initiate the Enhancement Management Process.

Note: If the ticket is not considered an incident, it will be cancelled or closed, and a Service Manager Analysis Request ticket will be created.

10 Share workaround 2.10 2.11 2.12

A parallel process takes place.

The E&E Systems Helpdesk Manager, Production

Support Manager, and/or the O&M BA Manager will share the workaround to resolve/mitigate the issue.

2.13 2.14

The O&M BA Manager will return to the Incident

Management Process and initiate the

Enhancement Management Process.

Table 5: Issue Resolution Sub-Process Flow Narrative

6/23/2020 Page 25 of 37

5.0 Stakeholder Incident Notification Process

5.1 Stakeholder Notification for Severity 1 Incidents Process Flow

The following model and narrative outlines the process to be followed to ensure all stakeholders are notified in the event of a Severity 1 incident.

Figure 4: Stakeholder Notification Process for Severity 1 Incidents Process Flow

5.2 Stakeholder Notification for Severity 1 Incidents Process Flow Narrative

ID(s)

Narration

1 Severity 1 incident ticket is created

1.1 The E&E Systems Helpdesk Manager, SI, or

Production Support Manager creates a Severity 1 incident ticket in Jira. Follow Step 2.

2 Email is sent to the listserv 1.2 1.3

The E&E Systems Helpdesk Manager or Production Support Manager sends an email to the MMRP_Prod_Alert@scdhhs.gov listserv notifying EEMS Leadership, Information Technology Service Desk (ITSD), the O&M Contractor Team, and the

6/23/2020 Page 26 of 37

MMRP Hosting and Infrastructure Contractor of the incident. The Jira ticket # should be included in the message. Follow Step 3.

3 The Production Support Manager or E&E Systems Helpdesk Manager will alert the Stakeholders

1.4 The E&E Systems Helpdesk Manager or Production

Support Manager sends language to the Communications Team for the Agency Broadcast.

If after 4:00 pm, also text the Information Technology Services (ITS) Program Director (see External-Internal Stakeholders). Follow Step 4.

Note: The E&E Systems Helpdesk Manager or Production Support Manager will also send notification, as appropriate, to external stakeholders and/or Third party contacts.

2.1 The O&M Contractor Team and/or Incident Owner

receives notification that an issue has occurred.

Follow Step 5.

Note: The O&M Contractor Team will send notifications, as needed, to federal partners (see

Third party contacts).

1.5

Does the incident impact the Citizen Portal?

If yes, follow Step 6.

1.11 If no, the process ends.

4 Communications Team sends out an Agency Broadcast

3.1 3.2

Communications Team sends out an Agency Broadcast email referencing the Cúram incident.

Messages should include subject, brief description of issue, apology and commitment to update when the issue has been resolved. The process ends.

5 Follows the Incident Management Process

2.2 The O&M Contractor Team and/or Incident Owner

will follow the MMRP O&M Incident Management Process for communications, investigation, and resolution. (If the incident impacts the Citizen Portal, a technical difficulties/outage message will automatically display in the Citizen Portal.) Follow Step 7.

6 Production Support Manager notifies the Webmaster

1.6 The Production Support Manager will notify the

Webmaster to post the preapproved message to the scdhhs.gov website. Follow Step 8.

7 Per Incident Management a notification is sent

2.3 The O&M Contractor Team and/or Incident Owner

notifies O&M Leadership that the incident is resolved via the MMRP_Prod_Alert@scdhhs.gov listserv (the technical difficulties/outage message displayed in the Citizen Portal will be removed automatically when resolved). Follow Step 9.

https://schhs.sharepoint.com/:w:/r/sites/1211207/Production%20Support/External-Internal%20Stakeholders.docx?d=w6ed2d195a57645a19d521763e6c3bbf2&csf=1&web=1&e=mNeevY

6/23/2020 Page 27 of 37

8 Webmaster posts preapproved message

4.1 4.2

The Webmaster posts the preapproved message to the scdhhs.gov website. The process ends.

9 Alert Stakeholders 1.7 1.8

The Production Support Manager or E&E Systems Helpdesk Manager will send language to the Communications Team for the Agency Broadcast.

Follow Step 10.

1.9

Did the incident impact the Citizen Portal?

If yes, follow Step 11.

1.12 If the incident did not impact the Citizen Portal, the process ends.

10 Communications Team sends resolution notification

3.3 3.4

The Communications Team sends out an Agency Broadcast referencing the resolution to the Cúram incident. The process ends.

11 Webmaster is notified 1.10 The Production Support Manager will notify the Webmaster to post the System Available message to the scdhhs.gov website. Follow Step 12.

12 Post System Available message

4.3 4.4

The Webmaster posts the System Available message to scdhhs.gov website. The process ends.

Table 6: Stakeholder Notification for Severity 1 Incidents Process Narrative

5.3 Stakeholder Notification for Severity 2 or 3 Incidents Process Flow

The following model and narrative outlines the process to be followed to ensure all stakeholders are notified in the event of a Severity 2 or 3 incident.

6/23/2020 Page 28 of 37

Figure 5: Stakeholder Notification for Severity 2 or 3 Incidents Process Flow

5.4 Stakeholder Notification for Severity 2 or 3 Incidents Process Flow Narrative

ID(s)

Narration

1 Severity 2 or 3 incident ticket is created

1.1 The E&E Systems Helpdesk Manager or

Production Support Manager creates a Severity 2 or 3 incident ticket in Jira. Follow Step 2.

2 Email is sent to the listserv 1.2 1.3

The E&E Systems Helpdesk Manager or Production Support Manager sends an email to the MMRP_Prod_Alert@scdhhs.gov or MMRP_PROD_Support@scdhhs.gov (for Sev3) listserv notifying EEMS Leadership, Office of Communications, the O&M Contractor Team, and the MMRP Hosting and Infrastructure Contractor of the incident. The Jira ticket #

6/23/2020 Page 29 of 37 should be included in the message. Follow Step 3.

3 The Production Support

Manager or E&E Systems Helpdesk Manager will alert the Stakeholders

1.4 1.6

Is an agency broadcast needed?

If yes, the E&E Systems Helpdesk Manager or the Production Support Manager sends language to the Office of Communications for the Agency Broadcast. Follow Step 4.

Note: The E&E Systems Helpdesk Manager or Production Support Manager will also send notification, as appropriate, to external stakeholders and/or Third party contacts.

1.5 If no, the process ends.

2.1 The O&M Contractor Team and/or Incident

Owner receives notification that an issue has occurred. Follow Step 5.

Note: The O&M Contractor Team will send notifications, as needed, to federal partners (see Third party contacts).

4 Office of Communications sends out an Agency Broadcast

3.1 3.2

Office of Communications sends out an Agency Broadcast email referencing the Cúram incident. The message should include subject, brief description of issue, apology and commitment to update when issue has been resolved. Process ends.

5 Follows the Incident Management Process

2.2 The O&M Contractor Team and/or Incident

Owner will follow the MMRP O&M Incident Management Process for communications, investigation, and resolution. Follow Step 6.

6 Per Incident Management a notification is sent

2.3 The O&M Contractor Team and/or Incident

Owner notifies O&M Leadership that the incident is resolved via the MMRP_Prod_Alert@scdhhs.gov listserv.

Follow Step 7.

7 Send language to Office of Communications for the Agency Broadcast, if needed

1.7

The Production Support Manager or E&E Systems Helpdesk Manager will send language to the Office of Communications for the Agency Broadcast if follow-up is needed. Follow Step 8.

8 Office of Communications sends resolution notification

3.3 3.4

Office of Communications sends out an Agency Broadcast referencing the resolution to the Cúram incident. The process ends.

Table 7: Stakeholder Notification for Severity 2 or 3 Incidents Process Flow Narrative https://team.scdhhs.gov/pmo/ProjectRepository/1211207/_layouts/WordViewer.aspx?id=/pmo/ProjectRepository/1211207/Operations%20and%20Maintenance/SCDHHS%20MMRP%20O_M%20Incident%20Management.docx&Source=https%3A%2F%2Fteam%2Escdhhs%2Egov%2Fpmo%2FProjectRepository%2F1211207%2FOperations%2520and%2520Maintenance%2FForms%2FAllItems%2Easpx%3FGroupString%3D%253B%2523MMRP%2520Operations%2520Document%253B%2523In%2520Review%253B%2523%26IsGroupRender%3DTRUE&DefaultItemOpen=1 https://team.scdhhs.gov/pmo/ProjectRepository/1211207/_layouts/WordViewer.aspx?id=/pmo/ProjectRepository/1211207/Operations%20and%20Maintenance/SCDHHS%20MMRP%20O_M%20Incident%20Management.docx&Source=https%3A%2F%2Fteam%2Escdhhs%2Egov%2Fpmo%2FProjectRepository%2F1211207%2FOperations%2520and%2520Maintenance%2FForms%2FAllItems%2Easpx%3FGroupString%3D%253B%2523MMRP%2520Operations%2520Document%253B%2523In%2520Review%253B%2523%26IsGroupRender%3DTRUE&DefaultItemOpen=1

6/23/2020 Page 30 of 37

6/23/2020 Page 31 of 37

6.0 Tools, Templates and Forms

6.1 Templates and Forms

Root Cause Analysis Template

6.2 Tools

Tools used in support of the Incident Management process are listed below.

• Service Manager

• Jira

• Splunk - Deep Space Nine

• MMRP SharePoint https://team.scdhhs.gov/pmo/ProjectRepository/1211207/Templates/Forms/AllItems.aspx?RootFolder=%2Fpmo%2FProjectRepository%2F1211207%2FTemplates%2FO%2DM%20Templates&FolderCTID=0x01200018F7888EFBFC1D4E91F2F66B3D69147E&View=%7b7E477AAB-7F2E-43F2-81AF-0A8140AF8DDC%7d https://hhs-splunk.clemson.edu/en-US/app/search/deep_space_9 https://schhs.sharepoint.com/sites/1211207/SitePages/Home.aspx

6/23/2020 Page 32 of 37

7.0 Appendix

Appendix 7.1 Format of Email to Listserv

This provides a sample email to appropriate listserv to report an incident. The sample email provides the appropriate format for reporting an incident to the listserv. At minimum, the email must include a brief description of the problem, who is impacted (i.e. # of users), and when the problem began.

Any information sent to the MMRP_Prod_Alert@scdhhs.gov/ MMRP_PROD_Support@scdhhs.gov listserv should also be added to the Jira ticket.

Email sent to listserv should include:

• To: MMRP_Prod_Alert@scdhhs.gov/MMRP_PROD_Support@scdhhs.gov

• Subject: Cúram Production – Severity (#)

• Message (example): Around 8:30 a.m. the E&E Systems Helpdesk began receiving calls from workers throughout the state complaining of slow system response. They are also getting the error message, “Attempt to lock a record that is already locked by another user” when applying changes. It’s taking 15 – 30 minutes to authorize an application case. Also, there are complaints that the Deemed Baby Wizard is not working—after you enter the mother’s Medicaid number and click Save, the system just spins. We apologize for any inconveniences and will provide an update when the issue has been resolved.

*Note: Screenshots should not be included in the email. PHI/PII must not be included in the message. The Jira ticket number should be sent in a follow up email once the ticket is created.

mailto:MMRP_Prod_Alert@scdhhs.gov/

6/23/2020 Page 33 of 37

Appendix 7.2 Pre-approved Unplanned Outage Message (Citizen Portal) for scdhhs.gov

The following language has been approved to post to the scdhhs.gov site in the event of an unplanned outage to the Citizen Portal.

Online Medicaid Eligibility Application unavailable. Read More

The Read More would be:

Our online application system is currently unavailable. We are working to restore service as quickly as possible and apologize for any inconvenience. If you want to continue the application process, please visit our Where to Go for Help page.

Appendix 7.3 Pre-approved System Available Message after Unplanned Outage (Citizen

Portal) for scdhhs.gov

The following language has been approved to post to the scdhhs.gov site once the system is available following an unplanned outage to the Citizen Portal.

Online Medicaid Eligibility Application is available. Read More

The Read More would be:

Our online application system is available. We apologize for any inconvenience.

Appendix 7.4 Contact Methods for 3rd Parties

• MMRP_Prod_Alert@scdhhs.gov

• MMRP_PROD_Support@scdhhs.gov

• SCDHHS ITSD Helpdesk@scdhhs.gov (Address to Rod Davis, ITS Program Director)

• Clemson MITS (MEDS and MMIS) – (mshelp@clemson.edu)

• Clemson Network Operations Center (NOC) – (noc@clemson.edu)

• Clemson (Mainframe) Production Control and Administration (PCA) –

(pca@clemson.edu)

• Hosting and Infrastructure Contractor (Clemson) – (ccit_iso_hcis@lists.clemson.edu)

• Maximus - Contact SCDHHS MMRP

• Notice Distribution - (MMRP_PROD_Notices@scdhhs.gov)

• Oracle – (support.oracle.com)

• Jira (Tickets for Non-Production Incidents) https://www.scdhhs.gov/site-page/where-go-help mailto:MMRP_Prod_Alert@scdhhs.gov mailto:Helpdesk@scdhhs.gov mailto:mshelp@clemson.edu mailto:noc@clemson.edu mailto:pca@clemson.edu mailto:ccit_iso_hcis@lists.clemson.edu mailto:MMRP_PROD_Notices@scdhhs.gov

6/23/2020 Page 34 of 37

Additional contact information can be found in the External/Internal Stakeholder List on the

MMRP SharePoint site.

Appendix 7.5 MMRP CGIS Problem Escalation Process

The SCDHHS MMRP CGIS Problem Escalation Process describes the MMRP CGIS Problem

Definition and CGIS Escalation Process. CGIS problems will be tracked via Jira, and communications will occur between the authorized SCDHHS MMRP Staff and the IBM MMRP

Team.

The purpose of the MMRP CGIS Problem Escalation Process is to facilitate efficient escalation of

CGIS Problems in the MMRS Production Environment to the IBM MMRP Team in order to recover standard production service operations as quickly as possible to minimize disruption to the business.

https://schhs.sharepoint.com/:w:/r/sites/1211207/Production%20Support/External-Internal%20Stakeholders.docx?d=w6ed2d195a57645a19d521763e6c3bbf2&csf=1&e=LYQIgO https://schhs.sharepoint.com/:w:/r/sites/1211207/Operations%20and%20Maintenance/SCDHHS%20MMRP%20CGIS%20Problem%20Escalation%20Process.docx?d=w8c73394ef43a4c0488fac01e0d2d084a&csf=1&e=0waP8w

6/23/2020 Page 35 of 37

8.0 Document Control

8.1 Document Control/Revision History

Version Date Author (s) Description of Change

0.1 11/20/2017 Lisa Carlyle/ Erin

Coleman Initial draft – based on feedback from review of SCDHHS IBM MMRP: Operations & Maintenance:

Incident Management Process V1.2, last revised 04/17/2017 and Incident Management Notification Process V 0.1, last revised 07/17/2017 and meetings with MMRP O&M team

0.2 12/7/2017 Lisa Carlyle/Erin Coleman

Updated based on feedback from O&M Team review of document

0.3 12/19/17 Erin Coleman Updated based on feedback from O&M Team review of document

0.4 1/17/18 Lisa Carlyle/Erin Coleman

Updated based on feedback from review and 1/11 meeting; updated model and added pre-approved message language related to unplanned outage to appendix

0.5 1/26/2017 Lisa Carlyle/Erin Coleman

Updated based on team feedback and prepared to send for approvals

0.6 2/7/2018 Lisa Carlyle/Erin Coleman

Updates made to identify responsibilities of the O&M Contractor Team and the Incident Owner.

1.0 2/28/2018 Erin Coleman Updated the Title Page, footer, header, and watermark to…