Attachment H_Azure Disaster Recovery Plan Associate Template.docx

DOCX document 484 KB Posted

Attached to
R499--Social Security Number Reduction Tool and Database Management_VA-21-00004967
Solicitation number
36C10B21R0039
Issued by

About this file

This document is a solicitation for a sole source task order to maintain the Authority To Operate for the Social Security Number Reduction tool in the VA Enterprise Cloud environment. The contractor shall provide upgrades and maintenance to the VAEC platform as needed to support the functioning of the SSNR tool. Additionally, the contractor shall provide database management, troubleshooting, and user support for VA officials utilizing the SSNR. The task order will be awarded under the authority of the Small Business Act Section 8(a) and FAR Subpart 19.8. The Department of Veterans Affairs Technology Acquisition Center Austin is the contracting agency.

View the file

Other files

Show all 15

On GovTribe

Work with this file

  • Download the original file
  • Contacts named in this file
  • Similar government files
  • Ask GovTribe AI about this file

Text version

VA Enterprise Cloud (VAEC) Microsoft Azure Government (MAG) Security Classification: High GRC Boundary Alignment: VAEC MS AZURE GOVT

And [Organization 2 name/acronym] Security Categorization: Low/Moderate/High GRC Boundary Alignment: XYZ

Version X.X

MONTH YYYY

Disaster Recovery Plan

Entries in GREEN are instructions (to be replaced with instruction request) Entries in RED are to be replaced/completed by the System/Application Team

Entries in BLACK should NOT to be changed, they are boiler plate changed (if changes are necessary, please highlight)

[Organization 2] is the organization/system/application hosted in the VAEC.

1. All colored texts should be removed or converted to black when finalized for signature.

2. This box/page should be removed when finalized.

3. Send to VAECSecurityTeam@va.gov for review PRIOR to final VAEC sign off/approval.

i OFFICE OF INFORMATION SECURITY Table of Contents

Table of Contents2
Table of Figures4
List of Tables4
Document Change Control Record5
Disaster Recovery Plan (DRP) Approval6
DRP Distribution7
DRP Distribution [Organization 2 name/acronym]8
INTRODUCTION9
1.1Objective10
1.2Scope10
1.2.1System Description10
1.2.2VAEC Services12
1.3DRP Assumptions and Constraints13
Enter your organization DRP Assumptions and Constraints here:14
1.4DRP Considerations14
Enter your organization DRP Considerations here:14
1.5DRP Roles and Responsibilities14
Enter your organization DRP Roles and Responsibilities in the table below:16
1.6Threats and Vulnerabilities16
CONCEPT OF OPERATIONS17
2.1Activation and Notification18
2.1.1DRP Activation Criteria and Procedures18
2.1.2Notification Procedures19
2.1.3Accountability of OIT Recovery Personnel19
2.2Recovery Site(s)19
RECOVERY21
3.1Sequence of Recovery Activities21
3.2Escalation Notices / Awareness21
RECONSTITUTION21
4.1Concurrent Processing – IS Services21
4.2Data Validation and Functionality Testing21
4.3Reconstitution Declaration22
4.4Notifications (Users)22
4.5Cleanup22
4.6Event Documentation22
4.7Deactivation22
TEST, TRAINING AND EXERCISE23
DOCUMENT MANAGEMENT26
6.1Document Ownership26
6.2Plan Review and Maintenance26
6.3Document Distribution26

Table of Figures

Figure 1: VAEC MAG High Architecture11
Figure 2: VAEC MAG High Architecture Diagram11
Figure 3: DRP Call TreeB-1
Figure 4: DRP Call Tree [Organization 2 name/acronym]B-1

List of Tables

Table 1: Document Change Control5
Table 2: DRP Distribution List7
Table 3: DRP Distribution List [Organization 2 name/acronym]8
Figure 1: VAEC MAG High Architecture11
Figure 2: VAEC MAG High Architecture Diagram11
Table 4: IS Services/Applications/IS Support Services12
Table 5: [Organization 2 name/acronym] IS Services/Applications/IS Support Services13
Table 6: Roles and Responsibilities16
Table 7: Roles and Responsibilities [Organization 2 name/acronym]16
Table 8: VAEC MAG Threat Assessment16
Table 9: [Organization 2 name/acronym] Threat Assessment17
Table 10: VAEC MAG Vulnerability Assessment17
Table 11: [Organization 2 name/acronym] Vulnerability Assessment17
Table 12: TT&E Calendar24
Table 13: TT&E Calendar [Organization 2 name/acronym25
Table 14: DRP Personnel Contact Data – VAA-1
Table 15: DRP Personnel Contact Data – [Organization 2 name/acronym]A-2
Table 16: Primary Site VAEC MAG Key Personnel Contact DataA-2
Table 17: Primary Site [Organization 2 name/acronym] Key Personnel Contact DataA-3
Table 18: Recovery Site VAEC MAG Key Personnel Contact DataA-3
Table 19: Recovery Site [Organization 2 name/acronym] Key Personnel Contact DataA-3
Figure 3: DRP Call TreeB-1
Figure 4: DRP Call Tree [Organization 2 name/acronym]B-1
Table 21: Vendor Contact DataC-2
Table 22: [Organization 2 name/acronym] Contact DataC-3
Table 23: Alternate Storage SiteD-3
Table 24: Alternate Storage Site [Organization 2 name/acronym]D-4
Table 25: TelecommunicationsE-4
Table 26: Telecommunications [Organization 2 name/acronym]E-5
Table 27: Recovery SiteF-5
Table 28: Recovery Site [Organization 2 name/acronym]F-6
Table 29: Associated PlansG-6
Table 30: Associated Plans [Organization 2 name/acronym]G-7
Table 31: IS Services [Application/IS Support Services] Recovery Priority RankingG-7
Table 32: IS Services [Application/IS Support Services] Recovery Priority Ranking [Organization 2 name/acronym]G-7
Table 33: Acronym ListI-1

Document Change Control Record

Version
Release Date
Summary of Changes
Author
1.0
06/02/2017
Initial Draft
Sathishbabu

Sandu/Ji Jung

1.2
10/30/2017
Various Updates
Daniel Beaver/ Wilbert Francis, MPG/CSRA
1.3
10/30/2017
Review and update
Peter Davies, MPG/CSRA
1.4
09/07/2018
Revised for current VA template

Updated Information across full document Kevin Fanberg

1.5
06/12/2019
Updated template
Larry Do | CAS Security
1.6
11/08/2019
Updated contact info with VAEC Help Desk info
Kristen Hill-VAEC COMS Security Team/Cognosante
1.7
02/28/2020
Updated Template 2020
Larry Do | CAS Security
1.8
04/03/2020
Updated for April 2020 Release.
Larry Do | CAS Security
1.9
10/30/2020
Updated for October 2020 Release
Larry Do | CAS Security
1.10
01/10/2021
Updated for Jan 2021 Release.
Larry Do | CAS Security
1.11
05/18/2021
Updated for May 2021 Release
Larry Do | CAS Security
2.0
MM/DD/YYYY
DRP Updated with Organization 2 Information.
Organization 2 Author name - [System Acronym]

Table 1: Document Change Control Provide organization Author name and System Acronym in the document change control record table.

Disaster Recovery Plan (DRP) Approval As the designated authority for [Organization 2 name/acronym] hosted on VA Enterprise Cloud (VAEC) (Microsoft Azure Government (MAG) High, hereafter known as VAEC MAG, I hereby certify that the Disaster Recovery Plan (DRP) is complete and that the information contained provides an accurate representation of the recovery requirements for this application/system. I further certify that this document identifies the criticality of VAEC, and that the recovery strategies identified will provide the ability to recover VAEC and each information system in the most expedient and cost-beneficial method in keeping with their individual level of criticality.

I further attest that this DRP for VAEC will be tested at least annually. This plan was last tested on MONTH DD, YYYY. The test, training and exercise material associated with this plan are found in the VA plan repository. This document will be modified as changes occur and will remain under version control, in accordance with Federal regulations and guidance, and VA Handbook 6500.8 Information System Contingency Planning guidance.

Enter your Organizations signatory authorities below (May include but not limited to: Director, SO, ISSO, Manager, and/or PM).

[Organization 2 name/acronym] signatory authorities:

<Plan Approver Name> <Job Title>

<Plan Approver Name> <Job Title>

<Plan Approver Name> <Job Title>

<Plan Approver Name> <Job Title>

VAEC Review:

DRP Distribution Distribution of the DRP should be restricted to personnel involved in, or responsible for, the activities for the continued operations of the site/facility, the information systems, and system owners. Update this table with key personnel required to receive and hold a copy of this plan, as well as plan updates when they are issued.

VAEC:

Name
Title
Submitted via eMASS
VA AO via eMASS
Joseph Stevenson
DRP Coordinator
Christopher Cardella
DRP Coordinator (Alternate)
David Catanoso
DRP Director
Sue Speck
DRP Director (Alternate)
Christopher Cardella
Outage Assessment Team POC
Joseph Stevenson
Outage Assessment Team POC (Alternate)
David Catanoso
Business / Service Line POC
Sue Speck
Business / Service Line POC (Alternate)
Joseph Fourcade
Cyber Security
Albert Estacio
Information System Security Officer (ISSO)
William McDonough
Cloud System Architect
MAG COMS
ComsRequests@va.gov
VAEC Ops Team
VAEC-OPS-Team@va.gov

VAEC-Azure-Ops@va.gov

VAEC Security
VAECSecurityTeam@va.gov

All Disaster Recovery Activation Teams Cognosante Department of Veterans Affairs (VA) Microsoft Azure Government (MAG) Amazon Web Services (AWS) Disaster Recovery Activation Team

Table 2: DRP Distribution List

DRP Distribution [Organization 2 name/acronym] Distribution of the DRP should be restricted to personnel involved in, or responsible for, the activities for the continued operations of the site/facility, the information systems, and system owners. Update this table with key personnel required to receive and hold a copy of this plan, as well as plan updates when they are issued.

Enter your organization DRP distribution in the table below:

Name
Title

Table 3: DRP Distribution List [Organization 2 name/acronym]

INTRODUCTION

Information Systems (IS) are vital to the Department of Veterans Affairs (VA) business processes. This disaster recovery plan (DRP) for the VAEC Microsoft Azure Government High and [Organization 2 name/acronym] establishes comprehensive procedures to recover the site critical IS Services quickly and effectively following a disaster or extended critical disruption. It is important that IS Services are able to effectively operate at a recovery facility independent from the primary facility to ensure continued operations. The DRP is one plan within a suite of security and emergency management-related plans that provides guidance for fast recovery when a disaster impacts a VA facility.

As per NIST SP 800-34, the DRP applies to a major, usually physical disruption to service that deny access to the primary facility infrastructure for an extended period. A DRP is an information system-focused plan designed to restore operability of the target system, application, or computer facility infrastructure at an alternate site after an emergency. The DRP may be supported by multiple information system contingency plans to address recovery of impacted individual systems once the alternate facility has been established.

VAEC is a cloud based GSS, hosted on Microsoft Azure Government (MAG); FedRAMP package #: F1603087869. As per MAG’s FedRAMP authorization, MAG is responsible for all DRP activities within this environment. MAG’s DRP can be found in FedRAMP package #: F1603087869. The MAG is designed utilizing separate data centers (zones), and geographically separated regions.

Based on the foregoing, the VAEC system DRP responsibilities are inherited from MAG FedRAMP package #F1603087869.

The VAEC MAG’s primary location is multiple regions. All core services functions are replicated from MAG throughout these regions. VAEC Information System Contingency Plan (ISCP) documents the recovery and reconstitution procedures for the VAEC core services and hosted applications within the VAEC system boundary.

This DRP is, in applicable parts, compliant with the following guidance and directives:

Federal Information Security Modernization Act of 2014 (FISMA 2014), December 2014 OMB Circular A-130, Managing Federal Information as a Strategic Resource, July 2016 Department of Homeland Security (DHS), National Security Presidential Directive 51 / Homeland Security Presidential Directive 20, National Continuity Policy, May 2007 DHS, Federal Continuity Directive 1, Federal Executive Branch National Continuity Program and Requirements, January 2017 DHS, National Response Framework, October 2019 DHS, Homeland Security Exercise and Evaluation Program (HSEEP), January 2020 Homeland Security Council, National Continuity Policy Implementation Plan, August 2007 National Institute of Standards and Technology (NIST) Special Publication (SP) 800-34, Revision 1, Contingency Planning Guide for Information Technology Systems, November 2010 NIST SP 800-53, Revision 4, Security and Privacy Controls for Federal Information Systems and Organizations, January 2015 NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities, September 2006 VA Handbook 6500.8, Information Technology Contingency Planning, April 2011 OIT Comprehensive Emergency Management Homeland Security Test, Training & Exercise Program Strategy (Draft), January 2010

Objective The purpose of the VAEC MAG and [Organization 2 name/acronym] DRP is to provide a documented plan to address the restoration of mission critical IS Services and operations from a recovery site following an event that prevents the normal continuation of those services from the organization’s primary site. The DRP is supported by the Information System Contingency Plans (ISCP) for each critical IS Service at the affected facility and describes the coordination activities between the primary, and recovery site(s) that are required to recover and continue IS service operations.

Scope The VAEC MAG and [Organization 2 name/acronym] DRP has been developed for increasing the organizations’ resiliency posture in the face of an emergency that jeopardizes IS Services and operations in a specific facility. It was written in accordance with federal guidance NIST 800-34, Rev 1 and VA Handbook 6500.8. Specific IS Service procedures and instructions described within this DRP are for systems hosted within the VAEC MAG and [Organization 2 name/acronym]. The DRP does not address disruptions that can be resolved at the primary site either through the electronic transfer of data to an alternate location or through the acquisition and delivery of necessary backups to a different processing location. This plan does not address replacement or purchase of new equipment, short-term disruptions, and loss of data at the onsite facility or at the user-desktop levels.

ISCPs for each IS Service are referenced in the DRP in order to assist in the restoration of critical systems or transfer of critical systems’ data to the recovery site after it has been appropriately configured.

System Description The VA Enterprise Cloud (VAEC) MAG High is a General Support System that provides a secure application and hosting environment for VA applications, content, and utilities. These applications and services are used to deliver content to an audience made up of employees, veterans, contractors, partners across all VA medical centers and component facilities, Federal government, and the general public. Content and applications are provided by Veterans Benefits Administration (VBA), Veterans Health Administration (VHA), National Cemetery Administration (NCA) and Support Offices. VAEC provides the following services: Content delivery, Application Hosting and Management Services.

The VAEC infrastructure is hosted by MAG, a cloud service provider. The MAG platform is used to provide a variety of hosting environments to suit a variety of needs. MAG can support applications categorized up to High as rated in accordance with Federal Information Processing Standard (FIPS) 199. VA applications available to the public are hosted in MAG.

A dedicated private data link (MAG Express Route) provides all connectivity for on-premise VA resources communicating to the VAEC. Virtual Networks (VNets) wrap the applications within MAG to encapsulate network access. Access from the applications to VA internal resources such as Identity, Credential, and Access Management (ICAM) and Active Directory (AD) Services are conducted over the encrypted private data link to the VA network.

VAEC MAG is in multiple regions designed to allow U.S. government agencies, contractors and customers to move sensitive workloads into the cloud for addressing specific regulatory and compliance requirements. MAG does not manage logical access controls within the VAEC system boundary. VAEC offers the same level of security as other VA physical technology centers and supports existing VA security controls and certification requirements such as FISMA, HIPAA, HITECH, SAS-70, ISO 27001, FIPS 140-2 compliant end points, and PCI DSS.

The VAEC MAG High Architecture is provided in the Figure below:

VAEC-MAG Diagram LINK Figure 1: VAEC MAG High Architecture

Figure 2: VAEC MAG High Architecture Diagram i OFFICE OF INFORMATION SECURITY

Enter your organization System Description below:

VAEC Services The information above documents Service Level Agreements (SLAs) provided by the VAEC.

Enter your organization Service / Business Line in the table below:

The information below documents Service Level Agreements (SLAs) provided by the [Organization 2 name/acronym].

Service / Business Line
MTD
VAEC MAG High
6 hours
Service / Business Line
MTD
[Organization 2 name/acronym] [Security Categorization]
[MTD]

The following IS Services are maintained at VAEC MAG:

Service LineIS Service [Applications/IS Support Services]MTDRTO
Authentication Services6 hours6 hours
Server Configuration Management Service6 hours6 hours
Vulnerability Scanning Service6 hours6 hours
Monitoring Service6 hours6 hours
JumpBox Service6 hours6 hours
Auditing Service6 hours6 hours
Code Configuration and Release Management Services6 hours6 hours

Table 4: IS Services/Applications/IS Support Services

The following IS Services are maintained at [Organization 2 name/acronym].

Enter your organization IS Services/Applications/IS Support Services in the table below:

Service Line IS Service [Applications/IS Support Services] MTD RTO

Table 5: [Organization 2 name/acronym] IS Services/Applications/IS Support Services

DRP Assumptions and Constraints The following assumptions were used when developing this DRP:

The MAG is responsible for maintaining the DR site at the infrastructure level only.

· Applications that will leverage DR capabilities toolset within VAEC will require separate coordination and confirmation.

The MAG is responsible for annual DR testing at the infrastructure level only.

Personnel well versed in the day-to-day IS service operating procedures are available to operate at the recovery site.

Relocated IS Service recovery teams are prepared to deal with emergency procedures without the need to reference detailed written steps.

Management personnel with authority will be available to make decisions.

System and supporting component restoration priorities have been documented within each IS Service’s ISCP.

Alternate processing procedures (or workarounds) for critical business processes have been established by Service/Business lines and are listed in the ISCP for each system.

Current backups of the system software and data are intact and available at the offsite data storage facility or facilities.

The IS Service(s) at MAG are inoperable and cannot be recovered within the recovery time objective (RTO) required to allow the facility to continue to operate normally.

Additional assumptions:

Applications can inherit the VAEC MAG DRP. An application level DRP is required for application-specific Disaster Recovery processes.

This plan does not apply to the situations described below:

Disruptions deemed recoverable at the primary site.

Emergency evacuation of personnel addressed by the occupant evacuation plan.

Overall recovery of business operations. Service/Business line owners should address recovery of business operations in a separate business recovery plan.

Enter your organization DRP Assumptions and Constraints here:

DRP Considerations The MAG is responsible for backup and recovery of data needed to restore operations to the requisite sites.

As depicted in Figure 1, “VAEC MAG High Architecture Diagram”, the VAEC Infrastructure is designed to be highly available, fault tolerant and scalable by using the MAG Regions. The components of the VAEC Virtual Network (VNet) are designed to be hosted in designated MAG Regions.

The VAEC is designed with a comprehensive strategy to back up data, by first identifying the failure or disaster situation that can occur and the potential business impact. The VAEC leverages many built-in backup and recovery features of MAG. The VAEC also implements backup processes that offer the appropriate level of granularity to meet pre-defined RTO and RPO (listed in Table 3) that includes:

· File-level recovery

· Volume-level recovery

· Image-level recovery

· Object-level recovery

· Container-level recovery

The VAEC DRP takes into consideration the existing capabilities of the recovery site locations and documents them along with any associated limitations or restrictions that would hinder data and access for a data center, medical facility, benefits center, or national cemetery from being recovered as soon as possible after a major disruption.

The expectation is that critical IS services will continue by using strategies for continuing operations, including coordinating with continuity site personnel, processing data from remote locations, using cached data, transferring backups to another facility, and employing full-service restoration after the disruption by utilizing the recovery teams identified in this plan.

The DRP addresses all critical IS services as identified in the ISCPs. Facilities that are co-located or share infrastructure with other administrations should seek to ensure alignment with the continuity planning efforts developed for the other administrative or business units located in the facility.

Enter your organization DRP Considerations here:

DRP Roles and Responsibilities Roles and responsibilities are provided by MAG personnel and inherited from FedRAMP package #: F1603087869. The following table includes responsibilities that describe each role and identifies the individual or team responsible for executing or supporting system recovery.

DRP Role
Job Title
Responsibilities
DRP Director
Director, Business Continuity Management
· Overall responsibility for the development, execution, maintenance, activation, and deactivation of the DRP

· Authorizes all changes to the DRP

· Determines if manual or secondary processing should be initiated as a temporary method of maintaining business operations

· Ensures that an alternate DRP Director is designated

DRP Director (Alternate)
Business Continuity IT Specialist
· Acts on behalf of the DRP Director should the need arise
DRP Coordinator
Program Director
· Monitors recovery team activities until the systems are recovered at the recovery site

· Ensures that recovery operations are being performed consistent with service level agreements / service level requirements

· Provides periodic status updates to the DRP Director

· Files an after-action report (AAR) upon resumption of normal operations

DRP Coordinator (Alternate)
Deputy Project Manager
· Acts on behalf of the DRP Coordinator should the need arise
Cybersecurity Engineering Team
Cybersecurity Team
· Provides cybersecurity oversight during DRP activities
VA POC
Infrastructure Manager
· Represents the recovery and restoration interests of the affected Service/Business line
VA POC (Alternate)
Technical Architect
· Represents the recovery and restoration interests of the affected Service/Business line
IS Services POC (Alternate)
Manager, Cloud

Infrastructure

· Represents the recovery and restoration interests of the affected service/business line

Disaster Recovery Activation Team
Disaster Recovery Activation Team includes the following groups:

· Cognosante

· AWS/MAG

· Other Affiliated and/or Personnel as needed

· VA

· Determines the expected duration of the failover to the recovery site

· Prioritizes the sequence of resource recovery

· Performs all system recovery and resumption activities

· Powers on / off systems

· May retrieve backup tapes

· Configures systems

· Ensures voice and data communications are functioning, activate pagers, sat phones

· Provides IP numbers and network routing information

· Includes validation testing teams or personnel

· Responsible for coordinating recovery activities between each site’s teams

Table 6: Roles and Responsibilities

Roles and responsibilities are provided by [Organization 2 name/acronym] personnel and inherited from FedRAMP package #: F1603047866. The following table includes responsibilities that describe each role and identifies the individual or team responsible for executing or supporting system recovery.

Enter your organization DRP Roles and Responsibilities in the table below:

DRP Role
Job Title
Responsibilities

Table 7: Roles and Responsibilities [Organization 2 name/acronym]

Threats and Vulnerabilities The data gathered through the Information System Contingency Planning Assessment (ISCPA) process is used in the completion of a Business Impact Analysis (BIA), a Vulnerability assessment, and a Threat assessment. The results of the Threat and Vulnerability assessments for VAEC MAG are shown in Tables 5 and 6.

ThreatLikelihoodCapacityThreat Rating
Unpatched operating systems create a potential threat to the systemLowLowLow
Network misconfigurations create a potential threat to the systemLowLowLow

Table 8: VAEC MAG Threat Assessment Enter your Organization Threat Assessment here:

Threat Likelihood Capacity Threat Rating

Table 9: [Organization 2 name/acronym] Threat Assessment

IS ServiceThreatVulnerabilityExploit PotentialMitigation EffectivenessMitigation Strength
System AdministrationUnpatched operating systems create a potential threat to the systemOperating Systems Required Patch Service UpdatesLowHighHigh
System AdministrationNetwork misconfigurations create a potential threat to the systemNetwork daemons not managed by the package systemLowHighHigh

Table 10: VAEC MAG Vulnerability Assessment Enter your Organization Vulnerability Assessment here:

IS Service Threat Vulnerability Exploit Potential Mitigation Effectiveness Mitigation Strength

Table 11: [Organization 2 name/acronym] Vulnerability Assessment

CONCEPT OF OPERATIONS

The Concept of Operations section provides an overview of the VAEC MAG DRP activation and notification, recovery and, reconstitution phases, and supporting roles and responsibilities of VAEC MAG personnel during disaster activation.

The MAG is responsible for backup and recovery of data needed to restore operations to the requisite sites. VAEC inherits these responsibilities from MAG’s FedRAMP package #: F1603087869.

Disaster recovery (DR) is the function of preparing for and recovering from a disaster, that is, any event having a negative impact on your business continuity or security. This could include hardware or software failure, a network outage, a power outage, physical damage to a building such as fire or flooding, human error, malicious occurrence, or some other significant disaster. Several Federal Information Processing Standards (FIPS) and National Institute of Standards and Technology (NIST) policies are relevant to VA in this context: FIPS 199, FIPS 200, NIST SP 800-34, and NIST SP 800-53. Additionally, VA Handbooks 6500, 6500.5, and 6500.8 address how the 800- 53 controls are applied and implemented for systems and applications developed for, or used by, VA. Regular testing of DR plans is critical to train staff and to assess and validate DR components. Without regular, rigorous, and realistic testing, DR plans and solutions are almost certain to fail in the event of a disaster. A key concept is the difference between high availability (HA) and DR. As a practical matter, HA addresses routine component, sub-system or system failures, while DR addresses catastrophic failures or events that could take down an entire site.

Proper DR planning requires understanding of the following industry terms:

• Recovery Time Objective (RTO) – The time it takes after a disruption to restore a business process to its service level, as defined by the DR plan with the program’s accreditation package.

• Recovery Point Objective (RPO) – The acceptable amount of data loss measured in time.

In addition to these industry standards, VA also uses and requires the following:

• Business Impact Analysis (BIA) – A BIA, as defined by NIST Special Publication (SP) 800-34, Rev. 1, Contingency Planning Guide for Federal Information Systems, is an analysis of an information system’s requirements, functions, and interdependencies, and must correlate information systems with critical mission/business processes (e.g. VA’s Mission Essential Functions) and respective services.

• Maximum Tolerable Downtime (MTD) – Total amount of time that is acceptable for a mission/business process outage or disruption, determined by utilizing BIA constructs.

Enter your organization concept of operation here:

Activation and Notification The Activation and Notification Phase defines initial actions taken once a disruption has been detected and recovery activities appear to be imminent. The DRP Activation and Notification Phase defines the activities required to activate the DRP and notify supporting recovery personnel. These phases include preparing for deployment of support personnel to the specified recovery site or coordinating activities between the primary site and the site where data or operations will be temporarily processed. The following sub sections further describe the activation and notification activities for sites that will transfer data or processing to another facility or implement a full mobilization.

Enter your organization activation and notification here:

DRP Activation Criteria and Procedures The VAEC MAG DRP may be activated if one or more of these criteria are met:

The VAEC MAG facility is rendered unusable;

Any or all of the critical IS Services residing within the facility will not be available within the accepted recovery time objective (RTO) set forth in the individual ISCPs.

Additionally, the decision to activate the VAEC MAG DRP may require the DRP director to consult with the service/business Line POC’s listed in Appendix A.

Enter your organization DRP activation criteria and procedure here.

Notification Procedures The first step after activation of the VAEC MAG DRP is the notification of key personnel, stakeholders and IS Service(s) recovery support teams. This includes teams whose operations will be transferred to the recovery facility. However, this may be accomplished by notifying the Regional Operations Center who will coordinate and carry out notifications.

Notification procedures may include:

Identification of who makes the initial notifications The sequence in which personnel are notified (e.g., system owner, technical POC, contingency plan coordinator, business/service line POC, and Recovery Team POC);

The method of internal and external notifications (e.g., email, mobile phone, automated notification system; etc.);

What to do if any single person in the notification sequence cannot be reached; and Alert/notification messages.

Enter your organization notification procedure here:

For a full list of all DRP specific key personnel and contact information, refer to Appendix A.

For a list identifying leadership, recovery personnel and any facility POCs that are to be alerted of the DRP activation, refer to Appendix B.

Call trees are an effective means of conveying the communication sequence in which leadership, recovery personnel, and facility POCs should be alerted. Refer to Appendix B.

Accountability of OIT Recovery Personnel In the event of a natural or man-made disaster, MAG Package #: F1603087869 states that all requisite infrastructure and services required to restore site operations are replicated across MAG data centers (zones) and/or regions.

In the event of a natural or man-made disaster, DoD directives stipulate that all agencies achieve 100% personnel accountability. Normally, personnel accountability begins with the activation of your Directorate phone trees/recall roster. Additionally, the Veterans Administration has employed a mass notification system referred to as the VA-Personnel Accountability System (VA-PAS). VA-PAS is a bidirectional mass alert system that provides proactive assessment of personnel status and safety following any emergency, and maintaining communications while incidents are resolved.

Enter your organization accountability of OIT recovery personnel here.

Recovery Site(s) VAEC MAG is responsible for recovery of data needed to restore operations to the requisite sites (application/system teams are responsible for leveraging the appropriate Cloud Services to ensure appropriate recover). VAEC inherits recovery from MAG Package #: F1603087869. VAEC documents recovery and reconstitution procedures for IS services in the VAEC ISCP. This information is in Appendix F.

Enter your organization recovery site(s) here.

RECOVERY

The recovery phase provides formal recovery operations that begin after the DRP has been activated, outage assessments have been completed (if possible), personnel have been notified, and in some situations, appropriate teams have been mobilized.

Recovery Phase activities focus on implementing recovery strategies to restore the site / facilities IS Services and data capabilities at a recovery facility. At the completion of the recovery phase, VAEC MAG and [Organization 2 name/acronym] IS Services and data capabilities will be operational at the recovery site and capable of performing the functions identified in the plan. These functions will continue until IS Services and components are restored, repair of damage is completed, and the operational capabilities at the original or new permanent location are resumed.

Sequence of Recovery Activities High-level activities that occur during the recovery of VAEC MAG’s and [Organization 2 name/acronym] ‘s systems, data and operations are identified in their associated ISCPs listed in Appendix G.

Escalation Notices / Awareness Notifications include problem escalation to leadership and status awareness to system owners and users. Call trees are an effective means of conveying the communication sequence in which leadership, recovery personnel and facility points of contact should be alerted.

RECONSTITUTION

Reconstitution is the process by which a facility is tested to validate that all its system capabilities and functionalities are fully restored and operational. During the Reconstitution phase, recovery activities are completed, and normal systems operations are resumed, and operations are transitioned back to the primary site (if needed). If the original facility is “unrecoverable”, the activities in this phase can be applied to preparing the recovery site as a new permanent location to support system processing requirements.

The MAG is responsible for reconstitution of data needed to restore operations to the requisite sites. VAEC inherits recovery from MAG Package #: F1603087869. VAEC documents recovery and reconstitution procedures for IS services in the VAEC ISCP.

Enter your Organization reconstitution here:

Concurrent Processing – IS Services In concurrent processing, an IS Service operates at separate locations concurrently until there is a level of assurance that the recovered system is operating correctly. Please refer to the individual ISCPs for details regarding concurrent processing strategies for each IS Service resident within VAEC MAG. For a list of associated plans, see Appendix G.

Enter your Organization concurrent processing IS services here:

Data Validation and Functionality Testing

Data validation and functionality testing is the process of validating recovered data, data files or databases and testing to confirm that functionality have been recovered completely. Please refer to the individual ISCPs for details for each IS Service resident within VAEC MAG.

Enter your Organization data validation and functionality testing here:

For a list of associated plans, see Appendix G.

Reconstitution Declaration

Upon successfully completing testing and validation, the DRP director will formally declare recovery efforts complete. The DRP director will notify facility leadership, business / service line and technical POCs of the declaration.

Notifications (Users)

Upon return to normal system operations, VAEC MAG users will be notified by DRP Director using predetermined notification procedures (e.g., VA-PAS, email, broadcast message, phone calls, etc.).

Enter your Organization notifications (Users) here:

Cleanup

Cleanup is the process of restocking supplies used, returning manuals or other documentation to their original locations, and readying the system for a possible future contingency event. Please refer to the individual ISCPs for details for each IS Service resident within VAEC MAG.

Enter your Organization cleanup here:

For a list of associated plans, see Appendix G.

Event Documentation It is important that all recovery events throughout the transition of operations to the continuity site and back to the primary site, be well documented and lessons learned for inclusion and update to the DRP. It is the responsibility of each recovery team or person to document their actions during the recovery effort and to provide that documentation to the DRP coordinator. Alternatively, one of the recovery teams may be appointed the task of tracking the events. Event documentation should include:

Activity logs (including recovery steps performed and by whom, the time the steps were initiated and completed, and any problems or concerns encountered while executing activities);

Functionality and data testing results;

Lessons learned documentation; and After Action Report (AAR).

Deactivation

Once all activities have been completed and documentation has been updated, the DRP director or designee will formally deactivate the DRP recovery efforts. Notification of this declaration will be provided to all business and technical POCs.

TEST, TRAINING AND EXERCISE

The MAG is responsible for reconstitution of data needed to restore operations to the requisite sites. VAEC inherits recovery from MAG Package #: F1603087869. VAEC documents recovery and reconstitution procedures for IS services in the VAEC ISCP. Comment by Hill, Kristen A. (Cognosante): This statement needs to be clarified to include GSS/Infrastructure and should be made clear that it’s the application’s responsibility to have a plan or make arrangements to leverage the MAG capabilities.

All DRPs should be reviewed and tested at least yearly or whenever there is a significant change to the system.

NOTE: Full functional tests of systems are normally failover tests to the alternate locations and may be very disruptive to system operations if not planned well. Other systems located in the same physical location may be affected by or included in the full functional test. It is highly recommended that several functional tests be conducted and evaluated prior to conducting a full functional (failover) test.

A formal test plan is developed prior to the tabletop or functional test and exercise. Functional test procedures are developed to include key sections of the DRP, including a walk-through of the following:

Notification procedures;

System recovery on an alternate platform from backup media;

Internal and external connectivity; and Reconstitution procedures.

Results of the test are documented in an AAR, and lessons learned are developed for updating information in the DRP.

Persons or teams assigned DRP roles must be trained to efficiently and correctly respond to an event affecting VAEC MAG. VA OIT has developed a test, training, and exercise (TT&E) program compliant with the Homeland Security Exercise and Evaluation Program (HSEEP) which supports the following objectives:

Ensure that VAEC MAG’s personnel are familiar with the DRP and its associated activation, recovery and reconstitution procedures.

Validate DRP policies and procedures.

Exercise procedures through the use of tabletop and functional exercises, as appropriate.

Ensure that hardware, software, backup data, and records required to support recovery at a continuity site are available.

Exercises should be designed for the complete loss of a facility. The exercise should be a functionality test of application and infrastructure to ensure ability to continue critical business processes meeting mission essential functions of the VA enterprise performed by the primary site.

Activities | Tests | Training | Exercises
Frequency
Internal and External Interdependency Testing of internal and external interdependencies identified in the OI&T CEMP plans, with respect to performance of, and other agencies’ MEFs.
Annually
Test DRP’s communications.
Annually
Continuity communications testing of communications equipment (both secure and non-secure) to ensure the internal and external interoperability and viability of continuity communications systems and capabilities.
Annually
Test primary and backup infrastructure systems and services at alternate operating facilities, (e.g., power, water, fuel).
Annually
Alert, Notification, and Activation Procedures Testing for mission critical/emergency personnel.
Annually
DHS test reporting of the formal reporting processes of test results as directed by the Department of Homeland Security (DHS) / Federal Emergency Management Agency (FEMA). This report is prepared by the Office of Operations, Security and Preparedness (OSP) with input from the Administrations and Staff Offices to include OIT.
Annually
Alternate Facility Logistics Testing and exercising of required physical security capabilities at the identified alternate facility(ies).
Annually
Test recovery of vital classified and unclassified records, critical information systems, services, and data.
Annually
Documenting and reporting testing of the internal processes for formally documenting and reporting tests and their results.
Annually
Test DRP’s notification/activation procedures.
Annually
CEMP Awareness/Orientation training: a high-level overview presentation of CEMP concepts for all OIT staff (both mission critical/emergency personnel and non-mission critical/emergency personnel, to include contractors).
Annually
Relocation exercise to the recovery site.
Annually

Table 12: TT&E Calendar

Enter your Organization deactivation here:

Enter test, training, and exercise (TT&E) results in the table below:
Activities | Tests | Training | Exercises
Frequency

Table 13: TT&E Calendar [Organization 2 name/acronym

DOCUMENT MANAGEMENT

Document Ownership The contents of this document are the responsibility of VAEC MAG, which has assigned the DRP coordinator responsibility for its content, modifications, currency, distribution to stakeholders, and its presence in the VA document repository.

Enter your Organization document ownership here:

Plan Review and Maintenance To ensure currency, this document will be reviewed annually in conjunction with the annual DRP test / exercise and when systems at this facility incur significant modifications.

The DRP is a component of the overall VAEC Management Plan. This plan, and all related project management plans, can be found in the GRC tool.

The Project Management Team will review the DRP every 365 days and as necessary to address project conditions, client input, or contract changes.

This document will be subject to formal change control after approval by the VA contracting officer. Subsequent changes to this document will be requested, approved, and implemented in accordance with the change management processes defined in the Change Management Plan. Changes to this document will be recorded in the revision history table.

The most current version of the Privacy and Security Management Plan will be posted to the VAEC instance in the GRC tool.

Enter your Organization plan review and maintenance here:

Document Distribution A copy of the MAG DRP is in the MAG FedRAMP package #: F1603087869. A copy of the VAEC MAG DRP will be:

Provided to facility stakeholders (listed in Plan Distribution table at the front of this document) who have an interest or responsibility for the development or testing of this plan.

Held electronically or in hard copy or both by every member of the recovery teams where it is easily accessible in an emergency.

Stored in the VA document repository.

Stored in an off-site location in both soft and hard copy format for ease of use under a wide range of circumstances

Enter your Organization document distribution here:

: DRP Personnel Contact Data - VA DRP Leadership

Key Personnel
Contact Information
DRP Director
Work #: 732-440-9583
Name: David Catanoso
VA Cellular #: 732-440-9583
Title: Director, Enterprise Cloud Solutions Office (ECSO)
VA E-mail: david.catanoso@va.gov
DRP Director (Alternate)
Work #: 801-588-5013
Name: Sue Speck
VA Cellular #: 801-209-4839
Title: Deputy Director, Enterprise Cloud Solutions Office (ECSO)
VA E-mail: sue.speck@va.gov
Cybersecurity Engineering Team
Work #: (561)291-2175
Name: Joseph Fourcade
VA Cellular #: (561)291-2175
Title: Cyber Security Analyst
VA E-mail: Joseph.Fourcade@va.gov
DRP Coordinator
Work #: (202) 461-0447
Name: Joseph Stevenson
VA Cellular #:
Title: VAEC Implementation Team Leader & COMS PM
VA E-mail: Joseph.Stevenson@va.gov
DRP Coordinator (Alternate)
Work #: TBD
Name: TBD
VA Cellular #: TBD
Title: MAG Program Manager Lead
VA E-mail: TBD
Outage Assessment Team POC
Work #: TBD
Name: Christopher Cardella
VA Cellular #:512-590-9414
Title: Manager, Cloud Infrastructure
VA E-mail: Christopher.Cardella@va.gov
Alternate Outage Assessment Team POC
Work #: 512-981-4792
Name: LaDonn Wittenburg
VA Cellular #:
Title: Implementation Manager, Service Management and Planning
VA E-mail: ladonn.wittenburg@va.gov

Table 14: DRP Personnel Contact Data – VA

Enter your Organization DRP Personnel Contact data here:

[Organization 2 name/acronym] DRP Leadership

Key Personnel
Contact Information
DRP Director
Work #:
Name:
VA Cellular #:
Title
VA E-mail:
DRP Director (Alternate)
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
Cybersecurity Engineering Team
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
Cybersecurity Engineering Team (Alternate)
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
Title:
VA E-mail:
DRP Coordinator
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
DRP Coordinator (Alternate)
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
Outage Assessment Team POC
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
Alternate Outage Assessment Team POC
Work #:
Name:
VA Cellular #:
Title: Implementation Manager, Service Management and Planning
VA E-mail:

Table 15: DRP Personnel Contact Data – [Organization 2 name/acronym]

Primary Site VAEC MAG Key Personnel

Key Personnel
Contact Information
Team Lead
Work #: 512-326-6180
Name: Christopher Cardella
VA Cellular #: 512-590-9414
Title: Manager, Cloud Infrastructure
E-mail: Christopher.Cardella@va.gov
Team Member
Work #: 512-981-4792
Name: LaDonn Wittenburg
VA Cellular #:512-364-3517
Title: IT Specialist, Core Systems Engineering
E-mail: LaDonn.Wittenburg@va.gov

Table 16: Primary Site VAEC MAG Key Personnel Contact Data

Enter your Organization primary site key personnel in the table below.

Primary Site [Organization 2 name/acronym] Key Personnel

Key Personnel
Contact Information
Team Lead
Work #:
Name:
VA Cellular #:
Title:
E-mail:
Team Member
Work #:
Name:
VA Cellular #:
Title:
E-mail:

Table 17: Primary Site [Organization 2 name/acronym] Key Personnel Contact Data

Recovery Site VAEC MAG Key Personnel

Key Personnel
Contact Information
Team Lead
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
Team Member
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:

Table 18: Recovery Site VAEC MAG Key Personnel Contact Data

Enter your Organization recovery site key personnel in the table below.

Recovery Site [Organization 2 name/acronym] Key Personnel

Key Personnel
Contact Information
Team Lead
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:
Team Member
Work #:
Name:
VA Cellular #:
Title:
VA E-mail:

Table 19: Recovery Site [Organization 2 name/acronym] Key Personnel Contact Data

: Call Tree

Name
Role
Telephone
Email
Priority
David Catanoso
DRP Director
Work #: 732-440-9583

Cell #: 732-440-9583

David.Catanoso@va.gov
1
Joseph Stevenson
DRP Director (Alternate)
Work #: 202-461-0447

Cell #:

Joseph.Stevenson@va.gov
2
Joseph Stevenson
DRP Coordinator
Work #: 202-461-0447

Cell #:

Joseph.Stevenson@va.gov
3
Christopher Cardella
DRP Coordinator (Alternate)
Work #: (512)-981-4792
Christopher.Cardella@va.gov
4

Figure 3: DRP Call Tree

1.The help desk team calls the DRP Director (or alternate), opens a communications bridge over Skype, and distributes a meeting invitation to all participants
2.The DRP Director calls the Program Director (or Deputy Project Manager)
3.The Program Director calls the ISO, the Cybersecurity Engineer, the Deputy Project Manager, the Project Coordinator, and the DRP Coordinator
4.The DRP Coordinator notifies all infrastructure team members as necessary
5.The ISO notifies the VA security team as necessary
6.The Program Director notifies VA program management in accordance with notification protocols and requirements

Enter your Organization call tree in the table below:

Name
Role
Telephone
Email
Priority

Figure 4: DRP Call Tree [Organization 2 name/acronym]

1.The help desk team calls the DRP Director (or alternate), opens a communications bridge over Skype, and distributes a meeting invitation to all participants
2.The DRP Director calls the Program Director (or Deputy Project Manager)
3.The Program Director calls the ISO, the Cybersecurity Engineer, the Deputy Project Manager, the Project Coordinator, and the DRP Coordinator
4.The DRP Coordinator notifies all infrastructure team members as necessary
5.The ISO notifies the VA security team as necessary
6.The Program Director notifies VA program management in accordance with notification protocols and requirements

: Personnel Contact Data – Vendors

Vendor Contact Data
Details
Vendor Name
Microsoft Azure Government
Vendor Type
Cloud Service Provider
Address
One Microsoft Way
City, State, and ZIP Code
Redmond, Washington 98052
Primary Contact Name
Scott Goad
Office Phone Number
(469) 775-2001

Emergency Phone Number

Email Address
sgoad@microsoft.com
Secondary Contact Name
Ryan Holt
Office Phone Number
(240) 620-6585
Email Address
Ryan.Holt@microsoft.com

SLA/MOU

Special Instructions

Table 21: Vendor Contact Data

Enter your organization personnel contact data in the table below:

Vendor Contact Data
Details

Vendor Name

Vendor Type

Address

City, State, and ZIP Code

Primary Contact Name

Office Phone Number

Emergency Phone Number

Email Address

Secondary Contact Name

Office Phone Number

Email Address

SLA/MOU

Special Instructions

Table 22: [Organization 2 name/acronym] Contact Data

: Alternate Storage Site VAEC is a highly available system that automatically fails over to MAG Region 2. Addresses for these sites are not published.

Alternate Storage Site
Details
City
Not published
State
Not published
Distance from Primary Facility
Not published
Is VA Owned?
No; however, the data is located in a VA-controlled environment within the FedRAMP System High-approved MAG
POC Name
Windows Azure Live Site (WALS) Team

wals@microsoft.com https://azure.microsoft.com/en-us/overview/clouds/government/contact/

This is the start of the file's text. The full file is on GovTribe.

File details come from the government source that posted it. Updated .