RFP 2025HCA5 Att 7 - Disaster Plan.docx

[System Name]

Security Categorization: {High/Medium/Low}

Washington State Health Care Authority (HCA)

Disaster Recovery Plan (DRP)

Version [#]

[Date]

Prepared by

[Technical Service Owner] Health Care Authority 626 8th Ave SE Olympia, WA 98501

TABLE OF CONTENTS

Plan Approval…………………………………………………….………..….……….……A.3-3

1. Introduction ………………………………………………….……..……….…….……..A.3-4

2. Concept of Operations ………………………….……..…………………………..……A.3-5

3. Activation and Notification………………....………………………..………….……..A.3-6

4. Recovery……………………….……………....…………………………………………..A.3-7

5. Reconstitution..……………….……………....………………….………………..………A.3-8

APPENDICES

Plan Approval Provide a statement in accordance with the agency’s Disaster Recovery Planning Policy (6-##) to affirm that the DRP is complete and has been tested sufficiently. The statement should also affirm that the designated authority is responsible for continued maintenance and testing of the DRP. This statement should be approved and signed by the designated system service owner. Space should be provided for the designated system service owner to sign, along with any other applicable approving signatures. A sample language is provided below:

As the designated system service owner for {system name}, I hereby certify that the disaster recovery plan (DRP) is complete, and that the information contained in this DRP provides an accurate representation of the application, its hardware, software, and telecommunication components. I further certify that this document identifies the criticality of the system as it relates to the mission of the HCA, and that the recovery strategies identified will provide the ability to recover the system functionality in the most expedient and cost-beneficial method in keeping with its level of criticality.

I further attest that this DRP for {system name} will be tested at least annually. This plan was last tested on {insert exercise date}; the test, training, and exercise (TT&E) material associated with this test can be found {TT&E results appendix or location}. This document will be modified as changes occur and will remain under version control, in accordance with HCA’s policy.

{Technical Service Owner Title}

1. Introduction Information systems are vital to HCA’s mission/business processes; therefore, it is critical that services provided by {system name} are able to operate effectively without excessive interruption. This Disaster Recovery Plan (DRP) establishes comprehensive procedures to recover {system name} quickly and effectively following a service disruption.

1.1 Background

This {system name} DRP establishes procedures to recover {system name} following a disruption. The following recovery plan objectives have been established:

· Maximize the effectiveness of contingency operations through an established plan that consists of the following phases:

· Activation and Notification phase to activate the plan and determine the extent of damage;

· Recovery phase to restore {system name} operations; and

· Reconstitution phase to ensure that {system name} is validated through testing and that normal operations are resumed.

· Identify the activities, resources, and procedures to carry out {system name} processing requirements during prolonged interruptions to normal operations.

· Assign responsibilities to designated HCA personnel and provide guidance for recovering {system name} during prolonged periods of interruption to normal operations.

· Ensure coordination with other personnel responsible for HCA’s contingency planning strategies. Ensure coordination with external points of contact and vendors associated with {system name} and execution of this plan.

1.2 Scope

This DRP has been developed for {system name}, which is classified as a {high/medium/low}-impact system, in accordance with Federal Information Processing Standards (FIPS) 199 – Standards for Security Categorization of Federal Information and Information Systems. Procedures in this DRP are for {high/medium/low}- impact systems and designed to recover {system name} within {RTO hours}. This plan does not address replacement or purchase of new equipment, short-term disruptions lasting less than {RTO hours}, or loss of data at the onsite facility or at the user-desktop levels.

1.3 Assumptions

The following assumptions were used when developing this DRP:

· {System name} has been established as a {high/medium/low}-impact system, in accordance with FIPS 199.

· Alternate processing services and offsite storage are required and have been established for this system.

· Current backups of the system software and data are intact and available.

· Alternate services have been established at AWS Availability Zone {us-west-1/2) and are available if needed for {system name}.

· The {system name} is inoperable at the HCA and cannot be recovered within {RTO hours}.

· Key {system name} personnel have been identified and trained in their emergency response and recovery roles; they are available to activate the {system name} Disaster Recovery Plan.

· Additional assumptions as appropriate.

The {system name} DRP does not apply to the following situations:

· Overall recovery and continuity of mission/business operations. The Business Continuity Plan (BCP) and Continuity of Operations Plan (COOP) address continuity of business operations.

· Emergency evacuation of personnel. The Occupant Emergency Plan (OEP) addresses employee evacuation.

· Any additional constraints and associated plans should be added to this list.

2. Concept of Operations The Concept of Operations section provides details about {system name}, an overview of the three phases of the DRP (Activation and Notification, Recovery, and Reconstitution), and a description of roles and responsibilities of HCA’s personnel during a plan activation.

2.1 System Description

Provide a general description of system architecture and functionality.

Indicate the operating environment, physical location, general location of users, and partnerships with external organizations/systems. Include information regarding any other technical considerations that are important for recovery purposes, such as backup procedures.

2.2 Overview of Three Phases

This DRP has been developed to recover the {system name} using a three-phased approach. This approach ensures that system recovery efforts are performed in a methodical sequence to maximize the effectiveness of the recovery effort and minimize system outage time due to errors and omissions.

The three system recovery phases are:

Activation and Notification Phase – Activation of the DRP occurs after a disruption or outage that may reasonably extend beyond the RTO established for a system. The outage event may result in severe damage to the facility that houses the system, severe damage or loss of equipment, or other damage that typically results in long-term loss.

Once the DRP is activated, system owners and users are notified of a possible long-term outage, and a thorough outage assessment is performed for the system. Information from the outage assessment is presented to system owners and may be used to modify recovery procedures specific to the cause of the outage.

Recovery Phase – The Recovery phase details the activities and procedures for recovery of the affected system. Activities and procedures are written at a level that an appropriately skilled technician can recover the system without intimate system knowledge. This phase includes notification and awareness escalation procedures for communication of recovery status to system owners and users.

Reconstitution – The Reconstitution phase defines the actions taken to test and validate system capability and functionality at the original or new permanent location. This phase consists of two major activities: validating successful reconstitution and deactivation of the plan.

During validation, the system is tested and validated as operational prior to returning operation to its normal state. Validation procedures may include functionality or regression testing, concurrent processing, and/or data validation. The system is declared recovered and operational by system owners upon successful completion of validation testing.

Deactivation includes activities to notify users of system operational status. This phase also addresses recovery effort documentation, activity log finalization, incorporation of lessons learned into plan updates, and readying resources for any future events.

2.3 Roles and Responsibilities

The DRP establishes several roles for {system name} recovery and reconstitution support. Persons or teams assigned DRP roles have been trained to respond to a contingency event affecting {system name}.

Describe each team and role responsible for executing or supporting system recovery and reconstitution. Include responsibilities for each team/role, leadership roles, and coordination with other recovery and reconstitution teams, as applicable. At a minimum, a role should be established for a technical service owner or business unit point of contact, a recovery coordinator, and a technical recovery point of contact.

Leadership roles should include the agency CIO or Deputy CIO, who has overall management responsibility to the agency, and an technical service owner, who is responsible for the plan and oversees recovery and reconstitution progress, initiate any needed escalations or awareness communications, and establish coordination with other recovery and reconstitution teams as appropriate.

3. Activation and Notification The Activation and Notification Phase defines initial actions taken once a {system name} disruption has been detected or appears to be imminent. This phase includes activities to notify recovery personnel, conduct an outage assessment, and activate the DRP. At the completion of the Activation and Notification Phase, {system name} DRP staff will be prepared to perform recovery measures.

3.1 Activation Criteria and Procedure

The {system name} DRP may be activated if one or more of the following criteria are met:

1. The type of outage indicates {system name} will be down for more than {RTO hours};

2. The facility housing {system name} is inaccessible and may not be available within {RTO hours}; and

3. Other criteria, as appropriate.

The following persons or roles may activate the ISCP if one or more of these criteria are met:

Establish one or more roles that may activate the plan based on activation criteria. Authorized persons may include the system or business owner, or the operations point of contact (POC) for system support.

3.2 Notification

The first step upon activation of the {system name} DRP is notification of appropriate business and system support personnel. Contact information for appropriate POCs is included in {Contact List Appendix name}.

For {system name}, the following method and procedure for notifications are used:

Describe established notification procedures. Notification procedures should include who makes the initial notifications, the sequence in which personnel are notified (e.g., agency leadership, business owner, technical service owner, business unit or user unit, and recovery team), and the method of notification (e.g., email blast, call tree, automated notification system, etc.).

3.3 Outage Assessment

Following notification, a thorough outage assessment is necessary to determine the extent of the disruption, any damage, and expected recovery time. This outage assessment is conducted by {name of recovery team}. Assessment results are provided to the technical service owner to assist in the coordination of the recovery of {system name}.

Outline detailed procedures to include how to determine the cause of the outage; identification of potential for additional disruption or damage; assessment of affected physical area(s); and determination of the physical infrastructure status, IS equipment functionality, and inventory. Procedures should include notation of items that will need to be replaced and estimated time to restore service to normal operations.

4. Recovery The Recovery Phase provides formal recovery operations that begin after the DRP has been activated, outage assessments have been completed (if possible), personnel have been notified, and appropriate teams have been mobilized. Recovery Phase activities focus on implementing recovery strategies to restore system capabilities, repair damage, and resume operational capabilities at the original or an alternate location. At the completion of the Recovery Phase, {system name} will be functional and capable of performing the functions identified in Section 2.1 of this plan.

4.1 Sequence of Recovery Activities

The following activities occur during recovery of {system name}:

Modify the following list as appropriate for the selected system recovery strategy:

1. Identify recovery location (if not at original location);

2. Identify required resources to perform recovery procedures;

3. Retrieve backup and system installation media;

4. Recover hardware and operating system (if required); and

5. Recover system from backup and system installation media.

4.2 Recovery Procedures

The following procedures are provided for recovery of {system name} at the original or established alternate location. Recovery procedures are outlined per team and should be executed in the sequence presented to maintain an efficient recovery effort.

Provide general procedures for the recovery of the system from backup solution. Specific keystroke-level procedures may be provided in an appendix. If specific procedures are provided in an appendix, a reference to that appendix should be included in this section. Teams or persons responsible for each procedure should be identified.

4.3 Recovery Escalation Notices/Awareness

Provide appropriate procedures for escalation notices during recovery efforts. Notifications during recovery include problem escalation to leadership and status awareness to system owners and users. Teams or persons responsible for each escalation/awareness procedure should be identified.

5. Reconstitution Reconstitution is the process by which recovery activities are completed and normal system operations are resumed. If the original facility is unrecoverable, the activities in this phase can also be applied to preparing a new permanent location to support system processing requirements. A determination must be made on whether the system has undergone significant change and will require reassessment and reauthorization. The phase consists of two major activities: validating successful reconstitution and deactivation of the plan.

5.1 Concurrent Processing (High-Impact Systems)

High-impact systems are not required to have concurrent processing as part of the validation effort. If concurrent processing does occur for the system prior to making it operational, procedures should be inserted here. Procedures should include length of time for concurrent processing, processing information on both concurrent systems, and validating information on the new permanent system.

For high-impact systems without concurrent processing and Medium/Low – impact systems, this section may be removed.

5.2 Validation Data Testing

Validation data testing is the process of testing and validating recovered data to ensure that data files or databases have been recovered completely. The following procedures will be used to determine that the recovered data is complete and current to the last available backup:

Provide procedures for testing or validation of recovered data to ensure that data is correct and up to date. This section may be combined with the Functionality Testing section if procedures test both the functionality and data validity. Teams or persons responsible for each procedure should be identified. An example of a validation data test for a high-impact system would be to log into the system database and check the audit logs to determine that all transactions and updates are current. Detailed data test procedures may be provided in Appendix E, System Validation Test Plan.

5.3 Validation Functionality Testing

Validation functionality testing is the process of verifying that {system name} functionality has been tested, and the system is ready to return to normal operations.

Provide system functionality testing and validation procedures to ensure that the system is operating correctly. This section may be combined with the Data Testing section if procedures test both the functionality and data validity. Teams or persons responsible for each procedure should be identified. An example of a functional test for a high-impact system may be logging into the system and running a series of operations as a test or real user to ensure that all parts of the system are operating correctly. Detailed functionality test procedures may be provided in Appendix E, System Validation Test Plan.

5.4 Recovery Declaration

Upon successfully completing testing and validation, the technical service owner will formally declare recovery efforts complete, and that {system name} is in normal operations. {System name} business and technical POCs will be notified of the declaration by the DRP Coordinator identified in Appendix A.

5.5 Notifications (users)

Upon return to normal system operations, {system name} users will be notified by {role} using predetermined notification procedures (e.g., email, broadcast message, phone calls, etc.).

5.6 Cleanup

Cleanup is the process of cleaning up or dismantling any temporary recovery locations, restocking supplies used, returning manuals or other documentation to their original locations, and readying the system for a possible future contingency event.

Provide any specific cleanup procedures for the system, including preferred locations for manuals and documents and returning backup or installation media to its original location.

5.7 Offsite Data Storage

It is important that all backup and installation media used during recovery be returned to the offsite data storage location is applicable. The following procedures should be followed to return backup and installation media to its offsite data storage location.

Provide procedures for returning retrieved backup or installation media to its offsite data storage location. This may include proper logging and packaging of backup and installation media, preparing for transportation, and validating that media is securely stored at the offsite location.

5.8 Data Backup

As soon as reasonable following recovery, the system should be fully backed up and a new copy of the current operational system stored for future recovery efforts. This full backup is then kept with other system backups. The procedures for conducting a full system backup are:

Provide appropriate procedures for ensuring that a full system backup is conducted within a reasonable time frame, ideally at the next scheduled backup period. This backup should go offsite with the other media in Section 5.7.

5.9 Event Documentation

It is important that all recovery events be well-documented, including actions taken and problems encountered during the recovery and reconstitution effort, and lessons learned for inclusion and update to this DRP. It is the responsibility of each DRP team or person to document their actions during the recovery and reconstitution effort, and to provide that documentation to the technical service owner.

Provide details about the types of information each DRP team member is required to provide or collect for updating the DRP with lessons learned. Types of documentation that should be generated and collected after a contingency activation include:

· Activity logs (including recovery steps performed and by whom, the time the steps were initiated and completed, and any problems or concerns encountered while executing activities);

· Functionality and data testing results;

· Lessons learned documentation; and

· After Action Report.

Event documentation procedures should detail responsibilities for development, collection, approval, and maintenance.

5.10 Deactivation

Once all activities have been completed and documentation has been updated, the technical service owner will formally deactivate the DRP recovery and reconstitution effort. Notification of this declaration will be provided to all business and technical POCs.

SUGGESTED APPENDICES

DRP appendices included should be based on system and plan requirements. The following appendices are recommended:

APPENDIX A PERSONNEL CONTACT LIST

Provide contact information for each person with a role or responsibility for activation or implementation of the DRP, or coordination with the DRP. For each person listed, at least one office and one non-office contact number is recommended. Note: Information may contain personally identifiable information and should be protected.

{System name} DRP Key Personnel

Home

Cellular

Email

Home

Cellular

Email

Home

Cellular

Email

Home

Cellular

Email

Home

Cellular

Email

APPENDIX B VENDOR CONTACT LIST

Contact information for all key maintenance or support vendors should be included in this appendix. Contact information, such as emergency phone numbers, contact names, contract numbers, and contractual response and onsite times should be included.

APPENDIX C DETAILED RECOVERY PROCEDURES

This appendix includes the detailed recovery procedures for the system, which may include items such as:

· Keystroke-level recovery steps;

· System installation instructions from Marketplace, tape, CD, or other media;

· Required configuration settings or changes;

· Recovery of data from backups and audit logs; and

· Other system recovery procedures, as appropriate.

If the system relies totally on another group or system for its recovery and reconstitution (such as WaTech), information provided should include contact information and locations of detailed recovery and reconstitution procedures for that supporting team.

APPENDIX D ALTERNATE PROCESSING PROCEDURES

This section should identify any alternate manual or technical processing procedures available that allow the business unit to continue some processing of information that would normally be done by the affected system. Examples of alternate processes include manual forms processing, input into workstations to store data until it can be uploaded and processed, or queuing of data input.

APPENDIX E SYSTEM VALIDATION TEST PLAN

This appendix includes system acceptance procedures that are performed after the system has been recovered and prior to putting the system into full operation and returned to users. The system validation test plan may include the regression or functionality testing conducted prior to implementation of a system upgrade or change.

An example of a system validation test plan:

Once the system has been recovered, the following steps will be performed to validate system data and functionality:

- Select Current Date Report

- Select Weekly

- Select To Screen Report is generated on screen with last successful transaction included

APPENDIX F ALTERNATE STORAGE, SITE, AND TELECOMMUNICATIONS

This appendix provides information for alternate storage, alternate processing site, and alternate telecommunications for the system. Alternate storage, site, and telecommunications information is required for high-impact systems, per NIST SP 800-53 Rev. 3. Refer to NIST SP 800-53 Rev. 3, for details on control specifics. Information that should be provided for each area includes:

Alternate Storage:

· City and state of alternate storage facility, and distance from primary facility;

· Whether the alternate storage facility is owned by the organization or is a third-party storage provider;

· Name and points of contact for the alternate storage facility;

· Delivery schedule and procedures for packaging media to go to alternate storage facility;

· Procedures for retrieving media from the alternate storage facility;

· Names and contact information for those persons authorized to retrieve media;

· Alternate storage configuration features that facilitate recovery operations (such as keyed or card reader access by authorized retrieval personnel);

· Any potential accessibility problems to the alternate storage site in the event of a widespread disruption or disaster;

· Mitigation steps to access alternate storage site in the event of a widespread disruption or disaster;

· Types of data located at alternate storage site, including databases, application software, operating systems, and other critical information system software; and

· Other information as appropriate.

Alternate Processing Site:

· City and state of alternate processing site, and distance from primary facility;

· Whether the alternate processing site is owned by the organization or is a third-party site provider;

· Name and points of contact for the alternate processing site;

· Procedures for accessing and using the alternate processing site, and access security features of alternate processing site;

· Names and contact information for those persons authorized to go to alternate processing site;

· Type of alternate processing site, and equipment available at site;

· Alternate processing site configuration information (such as available power, floor space, office space, telecommunications availability, etc.);

· Any potential accessibility problems to the alternate processing site in the event of a widespread disruption or disaster;

· Mitigation steps to access alternate processing site in the event of a widespread disruption or disaster;

· SLAs or other agreements of use of alternate processing site, available office/support space, setup times, etc.; and

· Other information as appropriate.

Alternate Telecommunications:

· Name and contact information of alternate telecommunications vendors;

· Geographic locations of alternate telecommunications vendors facilities (such as central offices, switch centers, etc.);

· Contracted capacity of alternate telecommunications;

· SLAs or other agreements for implementation of alternate telecommunications capacity;

· Information on alternate telecommunications vendor contingency plans;

· Names and contact information for those persons authorized to implement or use alternate telecommunications capacity; and

· Other information as appropriate.

APPENDIX G DIAGRAMS (SYSTEM AND INPUT/OUTPUT)

NOTE: Information for this section should be available from the Enterprise Architecture System Portfolio (EASSP) and can be copied from the EASP, or reference the applicable section in the EASP and attach the latest version of the EASP to this contingency plan. Include any system architecture, input/output, or other technical or logical diagrams that may be useful in recovering the system. Diagrams may also identify information about interconnection with other systems.

APPENDIX H HARDWARE AND SOFTWARE INVENTORY

Provide the hardware and software inventory for the system. Inventory information should include type of server or hardware on which the system runs, processors and memory requirements, storage requirements, and any other pertinent details. The software inventory should identify the operating system (including service pack or version levels, and any other applications necessary to operate the system, such as database software).

APPENDIX I INTERCONNECTIONS TABLE

Interconnection information should include the type of connection, information transferred, and contact person for that system.

If the system does not have any direct interconnections, then this appendix may be removed, or the following statement may be used:

{System name} does not directly interconnect with any other systems.

APPENDIX J TEST AND MAINTENANCE SCHEDULE

All DRPs should be reviewed and tested annually or whenever there is a significant change to the system. Provide information and a schedule for the testing of the system. The full functional test should include all DRP points of contact and be facilitated by the agency Chief Information Security Officer (CISO). A formal test plan is developed prior to the functional test, and test procedures are developed to include key sections of the DRP, including the following:

· Notification procedures;

· System recovery on an alternate platform from backup media;

· Internal and external connectivity; and

· Reconstitution procedures.

Results of the test are documented in an After Action Report, and Lessons Learned are developed for updating information in the DRP.

NOTE: Full functional tests of systems normally are failover tests to the alternate locations, and may be very disruptive to system operations if not planned well. Other systems located in the same physical location may be affected by or included in the full functional test. It is highly recommended that several functional tests be conducted and evaluated prior to conducting a full functional (failover) test.

Examples of functional tests that may be performed prior to a full functional test include:

· Full notification and response of key personnel to recovery location;

· Recovery of a server or database from backup; and

· Setup and processing from a server at an alternate location.

The following is a sample of a yearly test and maintenance schedule for a high-impact system:

APPENDIX K ASSOCIATED PLANS AND PROCEDURES

The most current version of the DRP, location of DRP, and primary point of contact (such as the technical service owner) should be noted.

APPENDIX L BUSINESS IMPACT ANALYSIS

The Business Impact Analysis results should be included in this appendix.

APPENDIX M DOCUMENT CHANGE PAGE

Modifications made to this plan since the last printing are as follows:

Record of Changes