Attachment L_ITOPS DRP Test Script Template.docx

2020 Disaster Recovery (DR) Exercise Plan

Directives

Relevant Controls Information:

CP-1 CONTINGENCY PLANNING POLICY AND PROCEDURES - Establishes policy and procedures for the effective implementation of selected security controls and control enhancements.

CP-2 CONTINGENCY PLAN - Addresses contingency roles, responsibilities, and assigned individuals with contact information.

CP-3 CONTINGENCY TRAINING - Training provided by organizations is linked to the assigned roles and responsibilities of organizational personnel to ensure that the appropriate content and level of detail.

CP-4 CONTINGENCY PLAN TESTING - Methods for testing contingency plans to determine the effectiveness of the plans and to identify potential weaknesses.

CP-6 ALTERNATE STORAGE SITE - An alternate storage site maintains duplicate copies of information and data in the event that the primary storage site is not available.

CP-7 ALTERNATE PROCESSING SITE - Establishes an alternate processing site including necessary agreements to permit the transfer and resumption of essential missions/business functions within recovery time and recovery point objectives when the primary processing capabilities are unavailable.

CP-8 TELECOMMUNICATIONS SERVICES – Alternate telecommunications services reflect the continuity requirements in contingency plans to maintain essential missions/business functions despite the loss of primary telecommunications services.

CP-9 INFORMATION SYSTEM BACKUP - Information system backups reflect the requirements in contingency plans as well as other organizational requirements for backing up information.

CP-10 INFORMATION SYSTEM RECOVERY AND RECONSTITION - The organization provides for the recovery and reconstitution of the information system to a known state after a disruption, compromise, or failure.

DR Test Script 2020 Disaster Recovery Exercise Plan The Exercise Plan gives officials, observers, and players from participating organizations information they need to observe or participate in the exercise. Some exercise material is intended for the exclusive use of exercise planners, controllers, and evaluators, but players may view other materials that are necessary to their performance.

The following terms/acronyms used in this plan are:

ESL – Enterprise Service Line CP – Contingency Plan OBC – Office of Business Continuity ISCP – Information System Contingency Plan EP – Emergency Preparedness OI&T – Office of Information & Technology NIST – National Institute of Standards and Technology EOC – Enterprise Operations Center VANS – Veterans Notification System NSD – National Service Desk COOP – Continuity of Operations Plan VISTA – Veterans Health Information Systems and Technology Architecture VISTA RO – Veterans Health Information Systems and Technology Architecture Read Only ANR – Automated Notification Reports CASD – Associates Service Desk IT – Information Technology MAS – Medical Administration Support SITREPS – Situation Reports VHA – Veterans Health Admiration POC – Point of Contact CIO – Chief Information Officer ISO – Information Systems Officer AAR – After Action Report OS – Operating System IS – Information System DECC – Defense Enterprise Computing Center Exercise Plan ITOPS Test Plan

TR – Test Results FOUO – Official Use Only – Department of Veteran Affairs

EXERCISE OVERVIEW

The test is to validate OIT disaster recovery plans. Procedures and organizational relationships will be evaluated, all aspects of the plans tested with lessons learned from this test used as a basis to update and strengthen our plans.

This test will measure the critical IS service outage response, recovery, and reconstitution as defined in VA Handbook 6500.8, the Federal Information Security Management Act (FISMA), and NIST Special Publications.

· Functionally test plan(s) to ensure the plan meets operational requirements based upon realistic scenario.

· Validate activation and notification process.

· Demonstrate ability to follow plan

· Validate data recovery and reconstitution process.

· Critically review performance post annual testing, capture lessons learned and plan disconnects

· Make corrections/modifications to the plan based upon lessons learned

· Test the ability and communication process for the newly formed ECC

· Test the communication process at the stakeholder level to pass on critical recovery information

· Encourage users to test their respective COOP plans during a test or actual outage

· Test the process for data validation before switching over to recovery dataset

· Test the ability of newly formed ITOPS to function as an enterprise team

· Test the communication process involved in making a disaster recovery decision

· Test full notification and response of key personnel to the recovery location

· Recovery of a server or database from backup media or setup, and processing from a server at an alternate location

· Ensure that the recovery capabilities of the systems are fully tested and that any weaknesses or shortcomings are identified and addressed

· Test the ability to failover to the alternate site to ensure that recovery efforts can be accomplished within the Recovery Time Objective stated in the plan

· Testing the contingency plan at the alternate processing site

· Full recovery and reconstitution of the information system to a known state

· Exercise the roles and responsibilities of specific team members, procedures, and assets involved in one or more functional aspects of a plan (e.g., communications, emergency notifications, IT equipment setup)

· The ECC establishes a call line to start the recovery process and each of the regions will test their respective data center DRP

· The checklist which is used for the migration is the same checklist used during an emergent DR event

· The Health, Benefits, Memorial, Corporate Support team will recover in the DR location

· The team will ensure sites are ready to migrate to alternate site and test their respective DRPs

· Data validations will be accomplished prior to migration

Players should take necessary action(s)

· Lync will be used so we can record the events

· ITOPS personnel, ECC, DR-COOP, and the Recovery team will join the call to recover affected system(s)

· During the event site personnel will involve and provide period updates to VA customers/leadership

· An ANR will be created by the ECC

· SITREPS to information senior OIT leadership will be prepared and sent to leadership

· A CASD ticket will be generated to engage appropriate recovery staff

· Sites will provide periodic feedback to customers

· OIT planners will test gaps in their respective plans

· Site will return to normal operations; and notify users

Sponsor

DR-COOP Team:

ECC

· Email: OIT ITOPS SO ECO ECC DR COOP All Staff <OITITOPSSPECOECCDRCOOPAllStaff@va.gov>

Exercise Plan ITOPS Test Plan

FOUO – Official Use Only – Department of Veteran Affairs

GENERAL INFORMATION

Exercise Objectives and Core Capabilities The following exercise objectives in Table 1 describe the expected outcomes for the exercise. The objectives are linked to core capabilities, which are distinct critical elements necessary to achieve the specific mission area(s). The objectives and aligned core capabilities are guided by elected and appointed officials and selected by the Exercise Planning Team.

Table 1. Exercise Objectives and Associated Core Capabilities Participant Roles and Responsibilities The term participant encompasses many groups of people, not just those playing in the exercise. Groups of participants involved in the exercise, and their respective roles and responsibilities, are as follows:

Players. Players are personnel who have an active role in discussing or performing their regular roles and responsibilities during the exercise. Players discuss or initiate actions in response to the simulated emergency.

Area Manager and staff ITOPS recovery teams Facilitators. Facilitators provide situation updates and moderate discussions. They also provide additional information or resolve questions as required. Key Exercise Planning Team members also may assist with facilitation as subject matter experts (SMEs) during the exercise.

DR-COOP Division Evaluators. Evaluators evaluate and provide feedback on a designated functional area of the exercise. Evaluators observe and document performance against expected outcomes and construction of the plan.

Evaluator appointed by AM, ISO or lead technician would be a good candidate as an observer Support Staff. The exercise support staff includes individuals who perform administrative and logistical support tasks during the exercise ECC personnel will provide assistance to the team during the exercise and actual events by initiating ANRs, CASD tickets, simulating calls to regional leadership, and providing an events log.

Exercise Assumptions and Artificialities In any exercise, assumptions and artificialities may be necessary to complete play in the time allotted and/or account for logistical limitations. Exercise participants should accept that assumptions and artificialities are inherent in any exercise and should not allow these considerations to negatively impact their participation.

Assumptions Assumptions constitute the implied factual foundation for the exercise and, as such, are assumed to be present before the exercise starts. The following assumptions apply to the exercise:

The exercise is conducted in a no-fault learning environment wherein capabilities, plans, systems, and processes will be evaluated.

The exercise scenario is plausible, and events occur as they are presented.

Exercise simulation contains sufficient detail to allow players to react to information and situations as they are presented as if the simulated incident were real.

Participating participants may need to balance exercise play with real-world emergencies. Real-world emergencies take priority.

Each facility will test their plan using this scenario.

For the purpose of this exercise the simulated loss of VISTA will affect at least 100 users and last over four hours.

No actual interruptions to VISTA will occur; the exercise will have no impact on normal business operations.

The local POC for the exercise will bring all stakeholders together for the exercise.

A facility driven tabletop with the VA staff prior to the region exercise would ensure a smooth event and stronger collaboration with business partners, however is not required.

The facility POC will encourage the VA stakeholders to test their contingency plans during the testing of this scenario. What are the backup plans for business operations if VISTA were lost?

Artificialities During this exercise, the following artificialities apply:

Only one CASD ticket and one ANR will be created per testing date.

Exercise Plan ITOPS Test Plan

No actual system will experience downtime.

EXERCISE LOGISTICS

Safety Exercise participant safety takes priority over exercise events.

Security If entry control is required both physical and logical for the exercise; the owner is responsible for arranging appropriate security measures.

POST-EXERCISE AND EVALUATION ACTIVITIES

Debriefings Post-exercise debriefings aim to collect sufficient relevant data to support effective evaluation, improvement planning, update and correct plans based on lessons learned.

Hot Wash At the conclusion of this exercise, controllers facilitate a Hot Wash to allow players to discuss strengths and areas for improvement, and evaluators to seek clarification regarding player actions and decision-making processes. All participants are strongly encouraged to attend.

Test Results (TR) The TR summarizes key information related to evaluation. The TR primarily focuses on the analysis of gaps in planning as well as performance, strengths, and areas for improvement. TRs also include basic exercise information, including the exercise name, type of exercise, dates, location, participating organizations, mission area(s), specific threat or hazard, a brief scenario description, name of participants and capture all related artifacts such as data integrity, AARs, and internal communications relative to the exercise.

Improvement Plan The TR identifies specific corrective actions and serves as a baseline to update the plan.

PARTICIPANT INFORMATION AND GUIDANCE

Exercise Rules The following general rules govern exercise play:

Real-world emergency actions take priority over exercise actions.

Exercise players will comply with real-world emergency procedures, unless otherwise directed by the control staff.

All communications (including written, telephone, and e-mail) during the exercise will begin and end with the statement “This is an exercise.”

Players Instructions Players should follow certain guidelines before, during, and after the exercise to ensure a safe and effective exercise.

Before the Exercise Review appropriate organizational plans, procedures, and exercise support documents.

During the Exercise Respond to exercise events and information as if the emergency were real, unless otherwise directed by an exercise controller.

Facilitators will give you only information they are specifically directed to disseminate.

Maintain a log of your activities. Many times, this log may include documentation of activities that were missed by a facilitator or evaluator.

After the Exercise Participate in the Hot Wash.

Include any notes or materials generated from the exercise in your TR document.

image1.png