PL 003.pdf

Disaster Recovery Plan June 2021 V6.0 Page 1 of 70

DISASTER RECOVERY (DR)

PLAN

MMS AND ONBASE

Prepared by Clemson University Medicaid Information Technology Services (CU MITS) This document is provided to the State of South Carolina Department of Human Services (SCDHHS) (“Client”) pursuant to A20181719A contract dated July 1, 2015, and is subject in all respects to the terms and conditions of such contract.

June 25, 2021

PL 003

Disaster Recovery Plan June 2021 V6.0 Page 2 of 70

APPROVALS

The undersigned acknowledge that they have reviewed the Disaster Recovery Plan and agree with the information presented within the document. Changes to this plan will be coordinated with, and approved by, the undersigned, or their designated representatives. Key SCDHHS

Leadership will be notified when approvals occur.

Name Title Approval Date

Rhonda Morrison MMRP Program Director/Deputy CIO 6/16/2021

Chris Cranford MMRP O&M Manager 6/15/2021

Thomas Taylor Executive Director – MITS 6/23/2021

Devananda Muthakana Chief Architect and Delivery Executive

SCMMRP (IBM)

6/24/2021

Disaster Recovery Plan June 2021 V6.0 Page 3 of 70

REVISION HISTORY

Version Date Author Description of Changes

0.1 September 30, 2015 Stacey Thornton Initial draft.

0.2 February 19, 2016 Lee Currington Response to feedback from SCDHHS.

0.3 March 4, 2016 Lee Currington Response to March 3, 2016 meeting with SCDHHS:

Updates to reflect MMS (EEMMS), change DHHS to SCDHHS, and add appendices.

0.4 March 7, 2016 Lee Currington Response to March 7, 2016 meeting with SCDHHS:

0.5 March 16, 2016 Lee Currington Response to March 11, 2016 feedback from

SCDHHS.

0.6 March 17, 2016 Janina Johnson Tracked changes accepted and reviewed CU provided comments

0.7 March 21, 2016 Tywanda Pringle Work with Clemson to get updates. Still in process.

0.8 March 23, 2016 Phyllis Coleman Responded to SCDHHS comments.

0.9 April 12, 2016 Phyllis Coleman Revisions to SCDHHS comments per feedback from CU team.

1.0 April 13, 2016 James Parrish Proof and edit.

1.0 April 29, 2016 Stasey Tackett Edits.

1.1 March 8, 2017 Phyllis Coleman Updates to date of the document, references added related to network connection redundancy, miscellaneous edits.

2.0 July 31, 2017 Phyllis Coleman Final version – DR 2017

3.0 September 24, 2018 Phyllis Coleman Final Version – DR 2018

3.1 June 12, 2019 Erin Coleman Updated version number and date prior to sending out for updates to CU, SCDHHS, and IBM.

3.2 July 15, 2019 Erin Coleman Updated based on feedback received from CU, SCDHHS, and IBM.

3.3 July 17, 2019 Erin Coleman Updated based on feedback received from CU, SCDHHS, and IBM.

4.0 August 6, 2019 Erin Coleman Updated date, version, and approval table based on approvals. Accepted all track changes and removed all comments.

4.1 May 15, 2020 Erin Coleman Created new revised draft in preparation for the annual review. Updated date, version, and watermark.

4.2 June 4, 2020 Erin Coleman Updated based on feedback.

4.3 June 11, 2020 Erin Coleman Updated RACI details from ISA.

4.4 July 7, 2020 Erin Coleman Updated based on Clemson feedback and confirmation in final review. Remove 2.1, updated

4.1 to include lessons learned, update 5.1 to refer to chart of responsibilities only, and Table A update to include Lee Currington and Austin Holmes.

5.0 July 14, 2020 Erin Coleman Updated date, version, and approval table based on approvals. Accepted all track changes and removed all comments.

Disaster Recovery Plan June 2021 V6.0 Page 4 of 70

Version Date Author Description of Changes

5.1 May 5, 2021 Erin Coleman Created new revised draft in preparation for the annual review. Updated date, version, and watermark.

5.1 May 26, 2021 Renjan Bhasi Added a couple of review comments, mainly about adding MFT.

5.2 June 15, 2021 Erin Coleman Minor updates and notes for open comments were added. Sent out for approval.

6.0 June 25, 2021 Erin Coleman Updated date, version, and approval table based on approvals. Accepted all track changes and removed all comments.

Disaster Recovery Plan June 2021 V6.0 Page 5 of 70

Table of Contents

1. Introduction

1.1 Overview and Purpose

1.2 Definition of Terms and Acronyms

1.3 Scope /Design

1.4 Roles and Responsibilities

1.5 Disaster Recovery Team

1.6 Assumptions

2.0 Concept of Operations

2.1 Emergency Response

2.2 Mitigation

2.3 Recovery Phases

2.4 System Description and Architecture

2.5 Document Application Interdependencies

3.0 Notification and Activation Phase

3.0.1 Notification and Activation Checklist

3.1 Line of Succession

3.2 Key Personnel

4.0 Damage Assessment Phase

4.1 Root Cause Analysis (RCA)

5.0 Recovery and Restoration

5.1 Recovery Team Responsibility

5.2 Alternate Storage/Processing Site and Telecommunication Services

5.3 Recovery and Restoration Checklist

6.0 Return-to-Normal Operations Phase

7.0 Appendices

Appendix A – Contact Lists and Call Tree - SCDHHS DR Systems

Appendix B – Line of Succession - SCDHHS Non-Mainframe DR Systems

Appendix D – Application Damage Assessment Checklist

Appendix E – Vital Records Checklist

Appendix F – Risk Analysis of Potential Disaster Events

Disaster Recovery Plan June 2021 V6.0 Page 6 of 70

Appendix G – Vendor Contact List - SCDHHS Non-Mainframe DR Systems

Appendix H –DR Hardware List - SCDHHS Non-mainframe DR Systems

Appendix I – Notification and Activation Procedures Checklist

Appendix J – Recovery Action Plan Checklist

Appendix K – Redundant Link Overview

Appendix L – Master Copy Dr Testing Framework

Appendix M: MMS Environments

Disaster Recovery Plan June 2021 V6.0 Page 7 of 70

1. INTRODUCTION

1.1 OVERVIEW AND PURPOSE

This Disaster Recovery Plan (DR) documents the strategies, personnel, procedures, and resources that will be used to respond to short-term or long-term interruptions to regular operations of the SCDHHS systems, in the event of a disaster or continuous interruption of service.

This plan represents collaborative planning between SCDHHS, IBM and MITS to implement processes and procedures to provide multi-site redundancy, and to provide the ability to fail services over between the respective sites, with a contractually defined recovery time objective

(RTO) and recovery point objective (RPO). The following actions are considered part of this DR

Plan:

• Establish and maintain a multi-site hosting site to house and operate the DR system(s).

• Establish and maintain a secondary processing system that will act as a failover production environment in the event of a disaster or continuous interruption to service.

• Document DR site access for all resources, including external vendors.

• Establish and maintain applicable telecom and network connections between MITS and the DR site, including redundancy on those connections.

• Maintain current documentation of the systems architecture for both the current production environments for the system(s) covered by this plan and the related DR processing environment.

• Establish and maintain comprehensive processes and procedures to follow in the event of a disaster or continuous interruption to service.

• Identify key personnel to engage in the event of a disaster or continuous interruption of service.

• Develop and maintain a testing schedule to ensure that the DR site is in a continuous state of readiness.

All applicable MITS, IBM and SCDHHS personnel are to be made aware of the existence and location of this document and are to be advised to read and become familiar with it. Key staff identified with primary or secondary responsibility in a disaster scenario are listed in Appendix

A – Contact Lists and Call Tree. These staff are responsible for ensuring access to the most current hardcopy version of this document, and that it is stored in a secure and accessible location. The most current version of this document is also available in the SCDHHS MMRP

SharePoint site as an Approved CDML Deliverable.

Disaster Recovery Plan June 2021 V6.0 Page 8 of 70

1.2 DEFINITION OF TERMS AND ACRONYMS

Acronym Description

CCTV Closed Circuit TV

CIS Core Infrastructure Services

DNS Domain Name System

DR Disaster Recovery – Disaster recovery is the process, policies, and procedures related to preparing for recovery or continuation of technology infrastructure that are vital to an organization after a natural or human-induced disaster.

DRD Disaster Recovery Director

DRT Disaster Recovery Team

DTO Division of Technology (SCDHHS)

MMRS

(EEMMS)

Member Management Replacement System

(Eligibility and Enrollment Member Management System)

HA High Availability – High availability is a system design approach and associated service implementation that ensures a prearranged level of operational performance will be met during a contractual measurement period

HCIS Healthcare Infrastructure Services Team

HWA Hardware Architecture Team

IBM International Business Machines

ISA Individual Service Agreement

ISO Infrastructure Systems and Operations

ITC Information Technology Center

MEDS Medicaid Eligibility Determination System

MITS Medicaid Information Technology Services

MMIS Medicaid Management Information System

MMS Member Management System

MMRP Member Management Replacement Program

MSA Master Service Agreement

NOC Network Operation Center

Pod Generalized term used to refer to high-density infrastructure racks supporting SCDHHS systems

RPO Recovery Point Objective - Point to which information used by an activity must be restored to enable the activity to operate on resumption.

Can also be referred to as “maximum data loss”.

RTE Recovery Time Estimate - Estimated period of time required to restore a particular level of functionality after taking into account any uncertainties.

Disaster Recovery Plan June 2021 V6.0 Page 9 of 70

Acronym Description

RTO Recovery Time Objective - Time goal for the restoration and recovery of functions or resources based on the acceptable down time and acceptable level of performance in case of a disruption of operations.

SCDHHS South Carolina Department of Health and Human Services

VPN Virtual Private Network

1.3 SCOPE /DESIGN

The scope of this DR Plan is to provide the tools, procedures, guidelines, and preventive measures to avoid or minimize the impact of an event that would otherwise result in the loss of data or an impact to SCDHHS system availability.

The DR Plan will be updated annually prior to the annual MMRP DR Test.

Key Vendor personnel have been identified and will be trained in their emergency response and recovery roles as part of the execution of this DR Plan.

The DR Plan addresses the establishment of a failover site for the MMS application, and addresses the implementation of the applicable processing environment and connectivity points for the non-mainframe components of MMS, which consists of the following:

• MMS production environment which includes

• OnBase

• Person Composite Service

Disaster Recovery Plan June 2021 V6.0 Page 10 of 70

For the purposes of this document, the term disaster is to mean natural and man-made disasters and other major events that would likely result in the damage to or interruption of services from the MMS production system, such as the following:

• Utility outage

• Infrastructure Failure

• Earthquake

• Fire

• Flood

• Hurricane or severe storm

• Criminal attack (e.g., cyber, bomb threat, terrorism)

This DR Plan establishes procedures to recover the above systems following a major disruption by addressing the following topics:

• Identification of responsible authority to direct emergency response

• Communication plans

• Inventory and classification of resources

• Emergency response procedures, including specification of teams of personnel assigned to respond in emergency situations

• Provisions in contracts with external vendors and service providers that ensure their preparedness for prompt emergency response

This plan has the following objectives:

• Maximize the effectiveness of disaster recovery operations through an established plan that consists of the following phases:

1. Notification and Activation to activate the plan and alert recovery plan team members about their responsibilities.

2. Damage Assessment to determine the nature and extent of the damage.

3. Recovery and Restoration to ensure that the SCDHHS systems are validated through testing, and that normal operations are resumed.

• Ensure coordination with other personnel at Clemson University responsible for emergency management beyond the scope of the SCDHHS systems, such as campus security and local municipal emergency services.

• Ensure coordination with any stakeholders (SCDHHS, IBM, MITS, HCIS, Keymark, and

NOC) and other vendors providing materials and services to the systems.

• Maintain SLAs (RTO, RPO)

Disaster Recovery Plan June 2021 V6.0 Page 11 of 70

This plan shall be tested minimally on an annual basis. The testing plan and details can be found in Appendix L – Master Copy DR Testing Framework.

Testing the plan shall include the following:

• Identify and validate the personnel responsible for recovery

• Confirm contact information for the personnel

• Evaluate their abilities to implement the plan

• Identify disaster recovery plan deficiencies

• Ensure that validated personnel have hardcopies of the plan to refer to at work and home in the event electronic copies are not available

As part of the testing phase, Tabletop tests will be performed, and any results impacting procedures and processes for disaster recovery will be updated in this DR Plan. Tabletop tests consist of disaster recovery personnel discussing the procedures and processes that each role will follow during an emergency. The tests validate that all requisite steps are followed in a particular emergency situation. Gaps identified during the tests are remediated and integrated into the DR Plan.

Disaster Recovery Plan June 2021 V6.0 Page 12 of 70

1.4 ROLES AND RESPONSIBILITIES

TABLE 1.4.1. ROLES AND RESPONSIBILITIES

Roles Responsibilities

SCDHHS Business

Owner(s) - SCDHHS DR

Systems

The responsibilities of the SCDHHS Business Owner during normal operations are as follows:

• Provide technical and administrative guidance to ensure efficient and secure operation of SCDHHS systems

• Work with the Clemson/IBM teams to address problems

• Participate in the Change Control Board (CCB) to ensure effective, secure changes to the production system

• Provide feedback from the user community in terms of system problems and desired enhancements

Responsibilities of the SCDHHS Business Owner during business disruption are as follows:

• Activate established agency business continuity/disaster recovery protocols.

• Establish and maintain communications with the identified

Clemson disaster recovery points of contact.

System

Developers/Maintainers -

SCDHHS DR Systems

The responsibilities of the IBM Developer/Maintainer include the following:

• Design the system

• Operate and maintain the system

• Document the system operations

• Propose design enhancements and implement those that are approved and funded

• Provide user support

• Develop, support, and train on user applications as requested and funded by CMS and SCDHHS

• Report system problems and security breaches

• Conduct, at minimum, an annual disaster recovery testing exercise

• Maintain hard copies of current disaster recovery processes, procedures and associated documentation

The responsibilities of the System Developer/Maintainer during business disruption is as follows:

• Follow the disaster recovery processes and procedures outlined in this DR Plan for the SCDHHS DR systems.

Disaster Recovery Plan June 2021 V6.0 Page 13 of 70

Roles Responsibilities

Disaster Recovery

Director (DRD)

The responsibilities of the Clemson University Disaster Recovery

Director include the following:

• Participate in and execute the annual disaster recovery testing exercise

• Plan, architect and implement DR solution

• Convene the DRT in the event of a disaster event

• Serve as the liaison between SCDHHS, Vendors and the CU DRT as well as with outside disaster responders (police, fire fighters at

Clemson University and in the Anderson, SC, community)

• Oversee infrastructure recovery operations for MMRP

• Oversee infrastructure and database recovery for OnBase

• Work closely with the Disaster Recovery testing team

• Communicate with the business owner(s), as applicable, as the recovery progresses

• Provide updates to the communications plans for disaster event response

Clemson University and

DTO Infrastructure

Support/Data Center

The responsibilities of the Clemson University and DTO Infrastructure

Support/Data Centers include the following:

• Report system problems and security breaches

• Conduct, at minimum, an annual disaster recovery testing exercise

• Maintain hard copies of current disaster recovery processes, procedures and associated documentation

• House the SCDHHS production and DR Systems

• Ensure the physical integrity of the facilities

• Guard against inappropriate access to the SCDHHS systems

• Provide appropriate power, cooling, etc., for efficient operation

• Ensure complete evacuation of either facility in the event it suffers loss of physical integrity

Disaster Recovery Plan June 2021 V6.0 Page 14 of 70

1.5 DISASTER RECOVERY TEAM

The Disaster Recovery Team (DRT) is responsible for recovery of SCDHHS systems, applications, and data. Members of the team include personnel who are responsible for daily operations and maintenance, and personnel who have been trained to respond to disaster events.

The DRT will lead the overall recovery effort. Each of these roles has designated a second in command (alternate member) to participate in the DRT in the event the primary resource is unavailable (See Appendix A – Contact Lists and Call Tree).

In addition, the DRT will head up teams in designated areas for closer coordination and will gather frequent progress updates to share with fellow DRT members. Communication will go up and down this chain of command to keep all personnel informed, and to facilitate decision-making based on the facts, priorities, and dependencies of the situation as it evolves.

In the event of a declared major disaster or disruption, DRT members, if on site at ITC and if the facility is still viable, should report to the ITC Canteen. If they are not on site at Clemson

University, they should not proceed to the disaster site but, instead, check their Clemson

University email account (listed in Appendix A –Contact Lists and Call Tree) for instructions.

In addition to email notification, the DRT will be notified per the call sequence specified in the call tree (See Appendix A – Contact Lists and Call Tree).

1.6 ASSUMPTIONS

The following assumptions and constraints are the basis for this plan:

• This DR Plan focuses on the Phase 1 approach for the design and implementation of recovering from a disaster applicable to SCDHHS systems.

• The Clemson University plan utilizes multi-site DR locations. Providing DR as a service will be wholly administered by Clemson in compliance with the programmatic and technical requirements outlined in the Master Service Agreement (MSA) and the Individual Service

Agreements (ISAs).

• IBM is the primary system integrator for the application and is governed by a separate contract with SCDHHS.

• Continuous communication between the Clemson and DR sites will be maintained via a direct or redundant connection.

• This DR Plan will be maintained in accordance with the SCDHHS Change Management Plan and process.

• Preventive controls (e.g., generators, environmental controls, sprinkler systems, fire extinguishers, and fire department assistance) are fully operational at the time of the disaster.

• All of the SCDHHS systems are connected to an uninterrupted power system (UPS) that provides temporary power during an electrical power outage.

• Systems, applications software, and data are replicated to the Columbia DR location (4430 Broad

River Road).

• Current infrastructure (services, utilities, and facilities) are intact and available to support normal operations.

Disaster Recovery Plan June 2021 V6.0 Page 15 of 70

• Clemson and IBM staff will attempt to recover and bring the system back into operation as quickly as possible. The RTO for this plan is 24 hours from the time of a disaster declaration. The

RPO is 1 hour from the event.

• Measures to protect vital records and personally identifiable information will be in accordance with CMS requirements, the alternate site that provides DR resources must conform to the same security controls as the primary Data Center.

Disaster Recovery Plan June 2021 V6.0 Page 16 of 70

2.0 CONCEPT OF OPERATIONS

This section provides system and operational details about the preparedness, general emergency response, mitigation, and an overview of the three phases of the DR Plan: (1) Notification and

Activation; (2) Damage Assessment; and (3) Recovery and Restoration.

Preparedness is critical to disaster recovery and involves all actions required to implement emergency response plans and conduct disaster recovery operations. Preparedness is implemented through a continual cycle of planning, training and equipping, conducting emergency exercises, and evaluating the results, then taking actions to correct deficiencies, mitigate vulnerabilities, and update relevant documentation.

2.1 EMERGENCY RESPONSE

For the general emergency response in the event of a disaster, policies are maintained by

Clemson University for the Anderson Datacenter (ITC).

DIAGRAM 2.1.1 CLEMSON ITC FLOOR PLAN

2.1.2 Evacuation drills

Evacuation drills are performed as needed, but at a minimum once a year. The drills will be pre-announced to minimize work disruption and scheduling conflicts, as well as to enable personnel to review evacuation procedures, routes, and appropriate gathering place(s) prior to the drill. In order to validate the effectiveness of the planned drills, non-announced drills will be conducted intermittently during the year.

Planned evacuation drill schedules will be coordinated with the Clemson University Emergency

Management Department, and Clemson Campus Security, with due consideration given for the needs of meetings, visitors, etc.

Evacuation drills shall consist of the following:

1. Security initiates the alarm to start the drill.

2. A timed evacuation of the building

3. The watch standers check their assigned areas and report “all clear” to the Clemson senior manager who, in turn, reports to the Facilities Management Representative with the same “all clear.”

4. General response emergency efforts shall follow pre-planned standard operating procedures (as shown in the following table). Contingency planning shall have determined the priority of activities to address both immediate and longer-term effects of the emergency.

Disaster Recovery Plan June 2021 V6.0 Page 17 of 70

TABLE 2.1.3 OVERVIEW OF GENERAL EMERGENCY RESPONSES

Type of

Emergency Generalized Response Estimated Downtime

Medical

Emergency

Call emergency number 911 for assistance.

Render first aid to the extent the organization-sponsored first aid training allows (as applicable). If victim's life is threatened or has the possibility of deterioration during transport, call the emergency number 911 to request an ambulance.

0 hours

Fire Call 911. Commence evacuation as necessary, using standard evacuation procedures. Meet at the designated emergency evacuation assembly area to account for all personnel and get updated information on the situation.

A few days to a few months, depending on the severity of the fire and what systems are affected

Severe

Storm

Secure the area. Move to a safe location within the interior of the building (interior stairwells, conference rooms, bathrooms, non-windowed rooms). Tune radio or television, if available, to weather station for local conditions and advice on taking cover. Be prepared to evacuate the building, if needed, when it is safe to do so.

A few days to a few weeks or months, depending on the severity of the storm, extent of the damage to the facility, and whether any hardware is affected

Tornado Employees should know where the “safer” areas are located in any building that they are

in. Things to remember about finding a “safer area”:

• Try and get to the lowest level of any structure in which you are located.

• Stay away from areas where windows and glass are present.

• Put as many walls as you can between yourself and the outside.

A few days to a few months, depending on the severity of the tornado.

Flooding Evacuate to higher ground if flooding is threatened or in progress.

A few days to a few months, depending on the severity of the flooding.

Disaster Recovery Plan June 2021 V6.0 Page 18 of 70

Type of

Emergency Generalized Response Estimated Downtime

Earthquake Take refuge under a desk, table, or stand within a doorframe. After the quake, evacuate to a place that is clear of buildings, trees, and power lines. Do not enter a structurally damaged building. Anticipate aftershocks.

A few days to a few months, depending on the severity of the earthquake, extent of the damage to facility, and whether any hardware is affected

Bomb

Threat

Until proven otherwise, treat all threats as real.

Call 911 and report threat to Clemson and

Anderson police departments. Commence evacuation as necessary, using standard evacuation procedures. Meet at designated emergency evacuation assembly area to account for all personnel and get updated information on situation.

0 hours to one day (assuming the threat is resolved with no damage and the systems continue to function even in the event of a facility evacuation)

Criminal

Attack

If the building structure is compromised, evacuate as per emergency evacuation plan.

Meet at designated emergency evacuation assembly area to account for all personnel and get updated information on situation. If building structure remains sound, await information through notification chain described earlier in this document.

Several weeks to a few months, depending on extent of damage to facility, whether any hardware is affected, and whether staff performing critical functions are injured.

Terrorist

Attack

If the building structure is compromised, evacuate as per emergency evacuation plan.

Meet at designated emergency evacuation assembly area to account for all personnel and receive updated information on the situation. If building structure remains sound, await information through notification chain described earlier in this document.

Several weeks to a few months, depending on extent of damage to facility, whether any hardware is affected, and whether staff performing critical functions are injured.

Disaster Recovery Plan June 2021 V6.0 Page 19 of 70

2.2 MITIGATION

The Clemson team (MITS, HCIS, and NOC) has put precautionary measures in place to mitigate the potential for a disaster. These include the following:

• Local mirrors of systems and/or data and use of disk protection technology such as

RAID.

• Surge protectors to minimize the effect of power surges on delicate electronic equipment.

• Use of an uninterruptible power supply and/or a backup generator to keep systems going in the event of a power failure.

• Fire suppression systems

• Fire alarms and accessible fire extinguishers

• General security measures — anti-virus software, system monitoring, and implementation of vulnerability scanners.

• Controlled, redundant temperature, and humidity environment.

• Secure network connectivity

• Encryption transmission for sensitive information

• System and data are replicated to DTO, a secure, off-site location that provides protection against common hazards, such as fire, flood, earthquake, and the theft and decay of storage media.

• Controlled physical access to computer systems for authorized personnel only.

• 7x24 staff to monitor and respond during emergencies.

• Secure access from remote locations during emergencies.

• Segra protected network link between point of presence (POP) in Anderson and

Columbia added

Electronic information is stored on professionally managed servers rather than individual desktop computers. The servers and data are replicated to the disaster recovery hardware located at Broad River Road, in Columbia, SC. Replication ensures full and efficient recovery in the event of resource destruction or modification.

Disaster Recovery Plan June 2021 V6.0 Page 20 of 70

2.3 RECOVERY PHASES

To coordinate preparedness, general emergency response, and mitigation strategies for the complete recovery of the SCDHHS Systems, this DR Plan uses a three-phase approach. This approach ensures that system recovery efforts are performed in a methodical sequence to maximize the efficiency and effectiveness of the recovery and minimize system outage due to errors and omissions.

The three (3) disaster recovery phases are as follows:

1. Notification and Activation – Activation of the DR Plan occurs after a disruption or outage that may reasonably extend beyond the RTO (24 hours) established for the system. Once the DR Plan is activated, the System Owner, Business Owner, and users are notified of the severity and estimated length of time of the outage.

2. Damage Assessment – A thorough outage assessment is performed of the system, presented to the System Owner and Business Owner, and used to modify recovery procedures specific to the cause of the outage.

3. Recovery and Restoration – This phase includes the activities to recover the affected system(s) and to implement notification procedures to communicate recovery status to the System Owner and Business Owner, Clemson University personnel, users, and other stakeholders. This phase consists of three major activities: validating successful restoration, deactivating the disaster recovery plan, and notifying all affected personnel that the system is once again operational.

During validation, the system is tested and validated as operational prior to returning operation to its normal state. Validation procedures include functionality or regression testing, concurrent processing, data validation, and user testing. The system is declared recovered and operational upon successful completion of validation testing.

This phase also documents the recovery effort, finalizes the activity log, and compiles lessons learned, which are then used to update this DR Plan and to expand preparation for future emergency events.

The Data Center, as part of the Clemson University campus, subscribes to the Clemson

Emergency Action/Operations Plan, which describes how the campus will respond to any type of emergency or disaster (http://www.clemson.edu/cusafety/EmergencyManagement/).

2.4 SYSTEM DESCRIPTION AND ARCHITECTURE

Clemson University, as contracted by SCDHHS, maintains the hardware, software, and provides services required for operating the SCDHHS DR Systems.

IBM, as contracted by SCDHHS, provides the following services: system development life cycle, design, testing, implementation, and operations and maintenance (O&M).

The below Table 2.4.1 System/Application Owners provides a list of the systems and application owners which are part of the DR Plan.

http://www.clemson.edu/cusafety/EmergencyManagement/

Disaster Recovery Plan June 2021 V6.0 Page 21 of 70

TABLE 2.4.1 SYSTEM/APPLICATION OWNERS

Official System Name Member Management

System Acronym MMS

Name of Organization Clemson University / IBM

Address 1801 Main Street

City, State, Zip Columbia, SC 29201

ISA Number A20181719A

Name of Business Owner Chris Cranford

Title MMRP O&M Manager

Address 1801 Main Street

City, State, Zip Columbia, South Carolina, 29201

E-mail Address chris.cranford@scdhhs.gov

Phone Number (803) 898-3407

Official System Name OnBase

System Acronym OnBase

Name of Organization Medicaid IT Services (MITS)/ Clemson University

Address 340 Computer Court

City, State, Zip Anderson, SC 29625

RISA Number A202514096A

Name of Business Owner Peter Schlies

Title Eligibility and Enrollment Application Management

Manager

Address 1801 Main Street Suite 1100

City, State, Zip Columbia, South Carolina, 29201

E-mail Address Peter.Schlies@scdhhs.gov

Phone Number 803-898-3082, 803-704-5185 (mobile)

For detailed system descriptions, see the respective System Design Documents for each of the above systems located on the MMRP SharePoint site under the Project Management Planning

Documents library as a CDML Deliverable).

The production systems are housed in the ITC Data Center or 340 Computer Court, Anderson, SC 29625.

The majority of users are located offsite and are from the following organizations:

mailto:chris.cranford@scdhhs.gov

Disaster Recovery Plan June 2021 V6.0 Page 22 of 70

• SCDHHS Staff and Field Offices

• South Carolina Medicaid Recipients

• Medicaid Service

Additionally, the user community includes SCDHHS, IBM, and Clemson staff who perform system and database administration, applications development, user support, documentation, and management roles.

In the context of the above systems, Clemson partners with the following:

• SCDHHS

• Individual Users

• DTO provides colocation services for Clemson University at an off-site location more than 100 miles from Clemson’s ITC data center in Anderson, SC.

• Various vendors, such as IBM, Keymark (OnBase), and Oracle, through whom hardware and systems and applications software have been procured. A full list of vendors can be found in Appendix G

2.5 DOCUMENT APPLICATION INTERDEPENDENCIES

The table below lists system interdependencies with references to more detailed documentation.

TABLE 2.5.1. DOCUMENT INTERDEPENDENCIES

Description Data

Name of system(s) MMS

Type of interconnection (Transmission

Control Protocol/Internet Protocol (TCP/IP), Domain Trust, Dial, etc.)

See the MMRP System Design Document

(located as an Approved CDML Deliverable in the Project Management Planning

Documents library on the MMRP SharePoint site)

Detailed descriptions of interconnections to other systems/applications outside the scope of the documented system

See the MMRP System Design Document

Describe the connectivity between systems See the MMRP System Design Document

Physical and logical security controls between systems

See the MMRP System Security Plan

Information/data that is shared and transmitted between systems, and the security

See the MMS System Security Plan (located in SERVIS on CMS zONE.

Disaster Recovery Plan June 2021 V6.0 Page 23 of 70 controls that are implemented to safeguard the confidentiality, integrity and availability such as encryption, virtual private network

(VPN), etc.

Name of system(s) OnBase

Type of interconnection (Transmission

Control Protocol/Internet Protocol (TCP/IP), Domain Trust, Dial, etc.)

See the OnBase System Design Document.

Detailed descriptions of interconnections to other systems/applications outside the scope of the documented system

See the OnBase System Design Document.

Describe the connectivity between systems See the OnBase System Design Document.

Physical and logical security controls between systems

See the OnBase System Security Plan.

Information/data that is shared and transmitted between systems, and the security controls that are implemented to safeguard the confidentiality, integrity and availability such as encryption, virtual private network

(VPN), etc.

See the OnBase System Security Plan.

Disaster Recovery Plan June 2021 V6.0 Page 24 of 70

3.0 NOTIFICATION AND ACTIVATION PHASE

This phase addresses the initial actions taken to detect and assess damage due to a disruption to any SCDHHS systems included in this DR Plan. The Notification and Activation Phase defines initial actions taken once a disruption has been detected or appears to be imminent. This phase includes activities to activate the plan and notify members of the DRT of their responsibilities.

Contact information for key personnel is located in Appendix A –Contact Lists and Call Tree.

The line of succession can be found in Appendix B – Line of Succession. At the completion of this phase, the DRT will be prepared to conduct damage assessment.

If a disaster occurs and the facility is still viable, all DR staff will follow the Communications

Plan.

The most senior member of the DRT present at the time of a disaster occurrence is the one to activate the DR Plan. See Appendix B – Line of Succession. The Disaster Recovery Director

(DRD) should convene the DRT, work with User Services to initiate notifications, assist/direct first responders, and lead the process for ascertaining the extent of damage.

The DR Plan must be activated if either of the following criteria is met:

• The type of outage indicates the SCDHHS systems will be down for more than 24 hours.

• The facility housing the SCDHHS systems is so badly damaged that it is likely not to be available within 24 hours.

The DRD should immediately contact the SCDHHS business owner(s) about the incident and expectations for the recovery process/timeframe, and take a DRT roll call to identify who is present and, assuming that some members of the DRT are not present, confirm that someone has responsibility for each area.

3.0.1 NOTIFICATION AND ACTIVATION CHECKLIST

The SCDHHS Systems Notification and Activation procedures are documented in Appendix I.

This table describes the actions to be taken by the DRD in the event of an incident, which include conducting an initial status meeting with technical and business leaders, activating the

DR Plan, communicating with the system owner and business owner, determining the extent of response and recovery actions to be performed, establishing the frequency of communications with the DRT and users affected by the outage, authorizing the necessary resources to effect an efficient recovery, and maintaining a log of recovery activities to be used to for lessons learned and to update this DRP.

Disaster Recovery Plan June 2021 V6.0 Page 25 of 70

Notifications will be accomplished through e-mail and telephone. MITS also has a toll-free line to use for the purpose of emergency communications:

Toll-free number: 888-289-4573 (U.S. & Canada)

Access code: 2628566

Host code: 5006001

An event may occur with or without prior notice. For example, advance notice is often available when a computer virus is expected on a certain date. Prompt notification is important to reduce the effects on the SCDHHS systems; and in some cases, it may provide enough time to allow systems personnel to shut down the system to avoid a hard crash.

However, no advanced notice can be expected of severe equipment failure, a criminal act, severe storm or tornado.

For a problem that is not obvious to all personnel, notification should be made to the senior manager present so that he/she can contact the DRD to invoke the DR Plan and begin notifications accordingly.

The amount and detail of information relayed in notifications may depend on the specific team being notified. Notification information should include the following (See Table 3.0.1):

• Date/time/nature of the emergency that has occurred (or is anticipated)

• Situation description

• Loss of life or injuries

• Actions being taken

• Any known physical damage to building and computer systems, and system outage

• Involvement of outside responders (e.g., fire, police, campus disaster response personnel)

• Plan (even if initial) for response and recovery

• Where and when to convene for a briefing or further instruction

• Instructions to extend notifications to a wider group of people

The first responder should notify the senior manager present who, in turn, should contact the

DRD to convene the DRT. All personnel shall be trained to notify the senior manager first in the event of a disaster. All known information about the disaster must be relayed to the DRD.

In activating the plan, the DRD will notify all personnel, including the system owner(s). A

SCDHHS MMS Disaster Recovery distribution list, will be established expressly for disaster recovery purposes, including primary and alternate email addresses for all personnel (in case the

LAN goes down). In addition to email notification, the DRT will be notified per the call sequence specified in the call tree (See Appendix A Contact List and Call Tree).

Disaster Recovery Plan June 2021 V6.0 Page 26 of 70

Table 3.0.1 NOTIFICATION FORM

Notification Form – Initial Report Date: Time:

1. Contact information for person providing notification

Name: Position: Title:

Address:

Office/cell phone: E-mail: Fax no.:

2. Contact information for person being notified

Address:

Office/cell phone: E-mail: Fax no.:

3. Nature of the Emergency/Situation Description

4. Date/Time

Date: Time:

5. Loss of Life or Injuries

Loss of Life Summary:

Injuries Summary:

6. Any known physical damage to building/computer systems/system outage

Physical Damage to Office Building:

Other Physical Damage:

Is Data Center Accessible:

7. Involvement of Outside Responders (fire, police, campus police, disaster response personnel)

Fire Department: Police: Campus Police:

DR Response Team: Campus Emergency Manager: Other Responders:

8. Initial Actions Being Taken

9. Where and When to Convene for Next Briefing

Date: Time: Where:

10. Instructions to extend notifications to a wider group of people

Disaster Recovery Plan June 2021 V6.0 Page 27 of 70

3.1 LINE OF SUCCESSION

Clemson has set forth an order of succession for managing the disaster recovery activities. Disaster management roles have been established with a line of succession, making authority for the recovery clear and pre-determined. The line of succession for the SCDHHS, and IBM can be found in Appendix B – Line of Succession.

3.2 KEY PERSONNEL

Contact information for key SCDHHS, IBM and CU personnel is provided in Appendix A –

Contact Lists and Call Tree.

Disaster Recovery Plan June 2021 V6.0 Page 28 of 70

4.0 DAMAGE ASSESSMENT PHASE

After the DRT has been notified of the disaster, a thorough damage assessment will be conducted to determine the extent of the damage, and to determine the expected time to recovery. This damage assessment should be completed as quickly as conditions permit. In conducting this assessment, personnel safety should always be the highest priority.

The Damage Assessment Team will determine the extent of damage that has occurred. Personnel with damage assessment responsibilities should be familiar enough with this DR Plan that they can perform their responsibilities even if this document is unavailable. However, they have been advised to retain a hard copy of this DR Plan both at work and at home in the event of a disaster.

See Appendix A – Contact Lists and Call Tree for a list of the Damage Assessment Team members.

The assessment should consist of the following components:

• Determine the cause of the outage.

• Identify potential, additional disruption or damage (e.g., due to additional lightning strike if the building is still standing but heavily damaged).

• Assess the physical facility (e.g., structural integrity of the Data Center, the condition of electric power, telecommunications, heating, ventilation, and air conditioning [HVAC]).

• Determine the functional status of the SCDHHS systems (e.g., fully functional, partially functional, or nonfunctional; type of damage to the equipment or data [e.g., water/fire damage, physical destruction, electrical malfunction due to power surge]).

• Identify items that need to be replaced (e.g., hardware, software, firmware, cabling, server racks, etc.).

• Estimate time to restoration of normal service.

The form provided in Table 4.0.1 should be used for members of the DRT to report their findings in an initial assessment of the disaster, and for use in compiling lessons learned to update the DR Plan after the system has been recovered.

Once the relevant technical leads have assessed the impact to the SCDHHS systems in their respective areas, they will provide the results to the DRD who, with the Business Owners and

System Owners and DRT, will formulate a plan of action (with dependencies, milestones, and timetable if possible) to recover the SCDHHS systems to full operation. The Recovery Action

Plan must be approved by the DRD, and the co-director, and documented as a common reference point as the situation evolves. The team leaders should discuss the planned response with their respective teams and the specific roles of individuals in the recovery effort. The Recovery Action

Plan is documented in Appendix J.

In the event of a power outage, the first step is to evaluate the physical system, as they will be in an unknown state. Servers and disks should be brought back up in the order that supports critical systems first. Firewalls, switches, and disks should power on automatically with the resumption

Disaster Recovery Plan June 2021 V6.0 Page 29 of 70 of power. The servers and border firewall/VPN should be configured not to power on automatically.

Disaster Recovery Plan June 2021 V6.0 Page 30 of 70

TABLE 4.0.1 DISASTER REPORT FORM

DISASTER REPORT FORM

Date: Report #:

1. Contact information for this report

Name: Position: Title:

Address:

Office/cell phone: E-mail: Fax no.:

2. Physical location of affected computer/network

(Include building number, room number):

3. Date and time event occurred

Date (mm/dd/yy): Time (hh:mm:ss am/pm-use local time):

4. Type of disaster

Describe with known specifics:

Provide any Internet URL with information relevant to this event. Provide a synopsis of the event and actions taken to mitigate the loss/damage.

5. Information on affected system

IP Address Computer/Host Name Operating System Other Applications

6. Initial assessment

What is the impact of this event?

Sensitivity of the data residing on the system:

Damage or observations resulting from disaster:

7. Notifications:

Who has been notified?

8. Additional Information:

Any amplifying information?

Disaster Recovery Plan June 2021 V6.0 Page 31 of 70

4.1 ROOT CAUSE ANALYSIS (RCA)

A Root Cause Analysis must be provided to SCDHHS documenting any issues that occurred during the DR Testing. The RCA must include the following:

• Description of DR Issue (written narrative of the issues)

• Action Taken (bulleted list of all actions taken during the DR test (to reach a resolution)

• Research findings (bulleted list of findings from RCA)

• Preventative Measures (bulleted list of recommended preventative measure steps)

• Conclusions (written narrative and summary of high/leadership level)

• Lessons Learned

• Revision History

The DR RCA Template must be completed and provided to SCDHHS after the completion of the

Annual DR Test.

5.0 RECOVERY AND RESTORATION

The Recovery and Restoration begins after the DR Plan has been activated, stakeholders have been notified, damage assessments have been completed, and the appropriate technical teams have been mobilized. Recovery activities include implementing strategies to repair damage, acquiring replacement hardware (if necessary), requesting and restoring backup files (if necessary) to restore system capabilities, testing and validating the restored system, returning to full operation, re-enabling outside secure access, notifying the user community that normal operations have been restored, cleaning up, backing up the system, documenting the experience, and lessons learned that will be used to update this DR Plan.

Periodically, Clemson staff members who are likely to be involved in system restoration should discuss the process that would be used if they were presented with a particular type of disaster scenario. Examples of the types of issues that might arise include situations in which the network information might not be correct if the system administrators need to use a different machine than was used before for a particular purpose, what happens if the systems come back but the database does not, etc.

SCDHHS operating systems and applications software and data are stored in two (2) locations for geographic replication: at the Clemson University ITC Data Center and at the DR site located on Broad River Road in Columbia, SC

(https://team.scdhhs.gov/PMO/ProjectRepository/1211207/SitePages/Home.aspx).

Tabletop Tests will be conducted in the DR testing exercises referenced in section 1.3 of this document. The results of the Tabletop Test will be used to update and alleviate any discovered deficiencies of procedures documented within this DR Plan.

https://schhs.sharepoint.com/:w:/r/sites/1211207/_layouts/15/Doc.aspx?sourcedoc=%7B72C94D2B-DA08-4928-9EF5-0ECDFD58A194%7D&file=DR%20Root%20Cause%20Analysis%20Template.docx&action=default&mobileredirect=true https://team.scdhhs.gov/PMO/ProjectRepository/1211207/SitePages/Home.aspx

Disaster Recovery Plan June 2021 V6.0 Page 32 of 70

5.1 RECOVERY TEAM RESPONSIBILITY

The following procedures are provided for recovery and restoration of the SCDHHS Systems at

Clemson University. These procedures and responsibilities are outlined per team to ensure that the teams stay in sync with each other and maintain an efficient, coordinated recovery effort.

Disaster Recovery Plan June 2021 V6.0 Page 33 of 70

Once SCDHHS, the DRD, and the DRD co-director have approved the DR plan, the various teams will take the necessary actions, and report back to the DRD and co-DRD as progress is made.

Activity / Responsible

Party

CU SCDHHS

System

Integrator Notes

R=Responsible, A=Accountable, C=Consulted, I=Informed

1. Architecture, Design & Build

Infrastructure architecture – design & build

R / A C / I I

Data center & network architecture

(drawings, diagrams, server lists)

R / A C / I I

LAN / WAN architecture R / C / I R / A C / I

Application technical documentation

C / I C / I R / A

Application business documentation

C / I R / A C / I

2. Hardware & Software Provisioning, Configuration & Installation

Provisioning of servers, infrastructure, load-balancers, DNS, & firewalls

R / A C / I C / I While CU is ultimately responsible and accountable for hardware provisioning, the MMS SI must be consulted and informed to assess and react to impacts to the

MMS.

Installation of OS software and patches including security patches

R / A C / I C / I

Installation of database software and patches

C / I C / I R / A

Procurement and licensing hardware to provide infrastructure

R / A C / I I

Installation of MMS application software

I C / I R / A IBM Cúram, ESB, etc.

Disaster Recovery Plan June 2021 V6.0 Page 34 of 70

Party

CU SCDHHS

System

Integrator Notes

Installation of CU-owned supporting software

R / A C / I I

3. Security

Conduct weekly vulnerability scanning of all open systems.

R / A I I

Distribute scanning results to appropriate teams

R / A C / I C / I Results will be sent to

SCDHHS OIA who then distributes to the appropriate application team (SI for MMS).

Intrusion detection services from external threats – servers and infrastructure

R / A C / I…