Attachment H_Azure Disaster Recovery Plan Associate Template.docx
DOCX document 484 KB Posted
- Solicitation number
- 36C10B21R0039
About this file
This document is a solicitation for a sole source task order to maintain the Authority To Operate for the Social Security Number Reduction tool in the VA Enterprise Cloud environment. The contractor shall provide upgrades and maintenance to the VAEC platform as needed to support the functioning of the SSNR tool. Additionally, the contractor shall provide database management, troubleshooting, and user support for VA officials utilizing the SSNR. The task order will be awarded under the authority of the Small Business Act Section 8(a) and FAR Subpart 19.8. The Department of Veterans Affairs Technology Acquisition Center Austin is the contracting agency.
View the file
Other files
Show all 15
On GovTribe
Work with this file
- Download the original file
- Contacts named in this file
- Similar government files
- Ask GovTribe AI about this file
Text version
VA Enterprise Cloud (VAEC) Microsoft Azure Government (MAG) Security Classification: High GRC Boundary Alignment: VAEC MS AZURE GOVT
And [Organization 2 name/acronym] Security Categorization: Low/Moderate/High GRC Boundary Alignment: XYZ
Version X.X
MONTH YYYY
Disaster Recovery Plan
Entries in GREEN are instructions (to be replaced with instruction request) Entries in RED are to be replaced/completed by the System/Application Team
Entries in BLACK should NOT to be changed, they are boiler plate changed (if changes are necessary, please highlight)
[Organization 2] is the organization/system/application hosted in the VAEC.
1. All colored texts should be removed or converted to black when finalized for signature.
2. This box/page should be removed when finalized.
3. Send to VAECSecurityTeam@va.gov for review PRIOR to final VAEC sign off/approval.
i OFFICE OF INFORMATION SECURITY Table of Contents
| Table of Contents | 2 | |
| Table of Figures | 4 | |
| List of Tables | 4 | |
| Document Change Control Record | 5 | |
| Disaster Recovery Plan (DRP) Approval | 6 | |
| DRP Distribution | 7 | |
| DRP Distribution [Organization 2 name/acronym] | 8 | |
| INTRODUCTION | 9 | |
| 1.1 | Objective | 10 |
| 1.2 | Scope | 10 |
| 1.2.1 | System Description | 10 |
| 1.2.2 | VAEC Services | 12 |
| 1.3 | DRP Assumptions and Constraints | 13 |
| Enter your organization DRP Assumptions and Constraints here: | 14 | |
| 1.4 | DRP Considerations | 14 |
| Enter your organization DRP Considerations here: | 14 | |
| 1.5 | DRP Roles and Responsibilities | 14 |
| Enter your organization DRP Roles and Responsibilities in the table below: | 16 | |
| 1.6 | Threats and Vulnerabilities | 16 |
| CONCEPT OF OPERATIONS | 17 | |
| 2.1 | Activation and Notification | 18 |
| 2.1.1 | DRP Activation Criteria and Procedures | 18 |
| 2.1.2 | Notification Procedures | 19 |
| 2.1.3 | Accountability of OIT Recovery Personnel | 19 |
| 2.2 | Recovery Site(s) | 19 |
| RECOVERY | 21 | |
| 3.1 | Sequence of Recovery Activities | 21 |
| 3.2 | Escalation Notices / Awareness | 21 |
| RECONSTITUTION | 21 | |
| 4.1 | Concurrent Processing – IS Services | 21 |
| 4.2 | Data Validation and Functionality Testing | 21 |
| 4.3 | Reconstitution Declaration | 22 |
| 4.4 | Notifications (Users) | 22 |
| 4.5 | Cleanup | 22 |
| 4.6 | Event Documentation | 22 |
| 4.7 | Deactivation | 22 |
| TEST, TRAINING AND EXERCISE | 23 | |
| DOCUMENT MANAGEMENT | 26 | |
| 6.1 | Document Ownership | 26 |
| 6.2 | Plan Review and Maintenance | 26 |
| 6.3 | Document Distribution | 26 |
Table of Figures
| Figure 1: VAEC MAG High Architecture | 11 |
| Figure 2: VAEC MAG High Architecture Diagram | 11 |
| Figure 3: DRP Call Tree | B-1 |
| Figure 4: DRP Call Tree [Organization 2 name/acronym] | B-1 |
List of Tables
| Table 1: Document Change Control | 5 |
| Table 2: DRP Distribution List | 7 |
| Table 3: DRP Distribution List [Organization 2 name/acronym] | 8 |
| Figure 1: VAEC MAG High Architecture | 11 |
| Figure 2: VAEC MAG High Architecture Diagram | 11 |
| Table 4: IS Services/Applications/IS Support Services | 12 |
| Table 5: [Organization 2 name/acronym] IS Services/Applications/IS Support Services | 13 |
| Table 6: Roles and Responsibilities | 16 |
| Table 7: Roles and Responsibilities [Organization 2 name/acronym] | 16 |
| Table 8: VAEC MAG Threat Assessment | 16 |
| Table 9: [Organization 2 name/acronym] Threat Assessment | 17 |
| Table 10: VAEC MAG Vulnerability Assessment | 17 |
| Table 11: [Organization 2 name/acronym] Vulnerability Assessment | 17 |
| Table 12: TT&E Calendar | 24 |
| Table 13: TT&E Calendar [Organization 2 name/acronym | 25 |
| Table 14: DRP Personnel Contact Data – VA | A-1 |
| Table 15: DRP Personnel Contact Data – [Organization 2 name/acronym] | A-2 |
| Table 16: Primary Site VAEC MAG Key Personnel Contact Data | A-2 |
| Table 17: Primary Site [Organization 2 name/acronym] Key Personnel Contact Data | A-3 |
| Table 18: Recovery Site VAEC MAG Key Personnel Contact Data | A-3 |
| Table 19: Recovery Site [Organization 2 name/acronym] Key Personnel Contact Data | A-3 |
| Figure 3: DRP Call Tree | B-1 |
| Figure 4: DRP Call Tree [Organization 2 name/acronym] | B-1 |
| Table 21: Vendor Contact Data | C-2 |
| Table 22: [Organization 2 name/acronym] Contact Data | C-3 |
| Table 23: Alternate Storage Site | D-3 |
| Table 24: Alternate Storage Site [Organization 2 name/acronym] | D-4 |
| Table 25: Telecommunications | E-4 |
| Table 26: Telecommunications [Organization 2 name/acronym] | E-5 |
| Table 27: Recovery Site | F-5 |
| Table 28: Recovery Site [Organization 2 name/acronym] | F-6 |
| Table 29: Associated Plans | G-6 |
| Table 30: Associated Plans [Organization 2 name/acronym] | G-7 |
| Table 31: IS Services [Application/IS Support Services] Recovery Priority Ranking | G-7 |
| Table 32: IS Services [Application/IS Support Services] Recovery Priority Ranking [Organization 2 name/acronym] | G-7 |
| Table 33: Acronym List | I-1 |
Document Change Control Record
| Version |
| Release Date |
| Summary of Changes |
| Author |
| 1.0 |
| 06/02/2017 |
| Initial Draft |
| Sathishbabu |
Sandu/Ji Jung
| 1.2 |
| 10/30/2017 |
| Various Updates |
| Daniel Beaver/ Wilbert Francis, MPG/CSRA |
| 1.3 |
| 10/30/2017 |
| Review and update |
| Peter Davies, MPG/CSRA |
| 1.4 |
| 09/07/2018 |
| Revised for current VA template |
Updated Information across full document Kevin Fanberg
| 1.5 |
| 06/12/2019 |
| Updated template |
| Larry Do | CAS Security |
| 1.6 |
| 11/08/2019 |
| Updated contact info with VAEC Help Desk info |
| Kristen Hill-VAEC COMS Security Team/Cognosante |
| 1.7 |
| 02/28/2020 |
| Updated Template 2020 |
| Larry Do | CAS Security |
| 1.8 |
| 04/03/2020 |
| Updated for April 2020 Release. |
| Larry Do | CAS Security |
| 1.9 |
| 10/30/2020 |
| Updated for October 2020 Release |
| Larry Do | CAS Security |
| 1.10 |
| 01/10/2021 |
| Updated for Jan 2021 Release. |
| Larry Do | CAS Security |
| 1.11 |
| 05/18/2021 |
| Updated for May 2021 Release |
| Larry Do | CAS Security |
| 2.0 |
| MM/DD/YYYY |
| DRP Updated with Organization 2 Information. |
| Organization 2 Author name - [System Acronym] |
Table 1: Document Change Control Provide organization Author name and System Acronym in the document change control record table.
Disaster Recovery Plan (DRP) Approval As the designated authority for [Organization 2 name/acronym] hosted on VA Enterprise Cloud (VAEC) (Microsoft Azure Government (MAG) High, hereafter known as VAEC MAG, I hereby certify that the Disaster Recovery Plan (DRP) is complete and that the information contained provides an accurate representation of the recovery requirements for this application/system. I further certify that this document identifies the criticality of VAEC, and that the recovery strategies identified will provide the ability to recover VAEC and each information system in the most expedient and cost-beneficial method in keeping with their individual level of criticality.
I further attest that this DRP for VAEC will be tested at least annually. This plan was last tested on MONTH DD, YYYY. The test, training and exercise material associated with this plan are found in the VA plan repository. This document will be modified as changes occur and will remain under version control, in accordance with Federal regulations and guidance, and VA Handbook 6500.8 Information System Contingency Planning guidance.
Enter your Organizations signatory authorities below (May include but not limited to: Director, SO, ISSO, Manager, and/or PM).
[Organization 2 name/acronym] signatory authorities:
<Plan Approver Name> <Job Title>
<Plan Approver Name> <Job Title>
<Plan Approver Name> <Job Title>
<Plan Approver Name> <Job Title>
VAEC Review:
DRP Distribution Distribution of the DRP should be restricted to personnel involved in, or responsible for, the activities for the continued operations of the site/facility, the information systems, and system owners. Update this table with key personnel required to receive and hold a copy of this plan, as well as plan updates when they are issued.
VAEC:
| Name |
| Title |
| Submitted via eMASS |
| VA AO via eMASS |
| Joseph Stevenson |
| DRP Coordinator |
| Christopher Cardella |
| DRP Coordinator (Alternate) |
| David Catanoso |
| DRP Director |
| Sue Speck |
| DRP Director (Alternate) |
| Christopher Cardella |
| Outage Assessment Team POC |
| Joseph Stevenson |
| Outage Assessment Team POC (Alternate) |
| David Catanoso |
| Business / Service Line POC |
| Sue Speck |
| Business / Service Line POC (Alternate) |
| Joseph Fourcade |
| Cyber Security |
| Albert Estacio |
| Information System Security Officer (ISSO) |
| William McDonough |
| Cloud System Architect |
| MAG COMS |
| ComsRequests@va.gov |
| VAEC Ops Team |
| VAEC-OPS-Team@va.gov |
VAEC-Azure-Ops@va.gov
| VAEC Security |
| VAECSecurityTeam@va.gov |
All Disaster Recovery Activation Teams Cognosante Department of Veterans Affairs (VA) Microsoft Azure Government (MAG) Amazon Web Services (AWS) Disaster Recovery Activation Team
Table 2: DRP Distribution List
DRP Distribution [Organization 2 name/acronym] Distribution of the DRP should be restricted to personnel involved in, or responsible for, the activities for the continued operations of the site/facility, the information systems, and system owners. Update this table with key personnel required to receive and hold a copy of this plan, as well as plan updates when they are issued.
Enter your organization DRP distribution in the table below:
| Name |
| Title |
Table 3: DRP Distribution List [Organization 2 name/acronym]
INTRODUCTION
Information Systems (IS) are vital to the Department of Veterans Affairs (VA) business processes. This disaster recovery plan (DRP) for the VAEC Microsoft Azure Government High and [Organization 2 name/acronym] establishes comprehensive procedures to recover the site critical IS Services quickly and effectively following a disaster or extended critical disruption. It is important that IS Services are able to effectively operate at a recovery facility independent from the primary facility to ensure continued operations. The DRP is one plan within a suite of security and emergency management-related plans that provides guidance for fast recovery when a disaster impacts a VA facility.
As per NIST SP 800-34, the DRP applies to a major, usually physical disruption to service that deny access to the primary facility infrastructure for an extended period. A DRP is an information system-focused plan designed to restore operability of the target system, application, or computer facility infrastructure at an alternate site after an emergency. The DRP may be supported by multiple information system contingency plans to address recovery of impacted individual systems once the alternate facility has been established.
VAEC is a cloud based GSS, hosted on Microsoft Azure Government (MAG); FedRAMP package #: F1603087869. As per MAG’s FedRAMP authorization, MAG is responsible for all DRP activities within this environment. MAG’s DRP can be found in FedRAMP package #: F1603087869. The MAG is designed utilizing separate data centers (zones), and geographically separated regions.
Based on the foregoing, the VAEC system DRP responsibilities are inherited from MAG FedRAMP package #F1603087869.
The VAEC MAG’s primary location is multiple regions. All core services functions are replicated from MAG throughout these regions. VAEC Information System Contingency Plan (ISCP) documents the recovery and reconstitution procedures for the VAEC core services and hosted applications within the VAEC system boundary.
This DRP is, in applicable parts, compliant with the following guidance and directives:
Federal Information Security Modernization Act of 2014 (FISMA 2014), December 2014 OMB Circular A-130, Managing Federal Information as a Strategic Resource, July 2016 Department of Homeland Security (DHS), National Security Presidential Directive 51 / Homeland Security Presidential Directive 20, National Continuity Policy, May 2007 DHS, Federal Continuity Directive 1, Federal Executive Branch National Continuity Program and Requirements, January 2017 DHS, National Response Framework, October 2019 DHS, Homeland Security Exercise and Evaluation Program (HSEEP), January 2020 Homeland Security Council, National Continuity Policy Implementation Plan, August 2007 National Institute of Standards and Technology (NIST) Special Publication (SP) 800-34, Revision 1, Contingency Planning Guide for Information Technology Systems, November 2010 NIST SP 800-53, Revision 4, Security and Privacy Controls for Federal Information Systems and Organizations, January 2015 NIST SP 800-84, Guide to Test, Training, and Exercise Programs for IT Plans and Capabilities, September 2006 VA Handbook 6500.8, Information Technology Contingency Planning, April 2011 OIT Comprehensive Emergency Management Homeland Security Test, Training & Exercise Program Strategy (Draft), January 2010
Objective The purpose of the VAEC MAG and [Organization 2 name/acronym] DRP is to provide a documented plan to address the restoration of mission critical IS Services and operations from a recovery site following an event that prevents the normal continuation of those services from the organization’s primary site. The DRP is supported by the Information System Contingency Plans (ISCP) for each critical IS Service at the affected facility and describes the coordination activities between the primary, and recovery site(s) that are required to recover and continue IS service operations.
Scope The VAEC MAG and [Organization 2 name/acronym] DRP has been developed for increasing the organizations’ resiliency posture in the face of an emergency that jeopardizes IS Services and operations in a specific facility. It was written in accordance with federal guidance NIST 800-34, Rev 1 and VA Handbook 6500.8. Specific IS Service procedures and instructions described within this DRP are for systems hosted within the VAEC MAG and [Organization 2 name/acronym]. The DRP does not address disruptions that can be resolved at the primary site either through the electronic transfer of data to an alternate location or through the acquisition and delivery of necessary backups to a different processing location. This plan does not address replacement or purchase of new equipment, short-term disruptions, and loss of data at the onsite facility or at the user-desktop levels.
ISCPs for each IS Service are referenced in the DRP in order to assist in the restoration of critical systems or transfer of critical systems’ data to the recovery site after it has been appropriately configured.
System Description The VA Enterprise Cloud (VAEC) MAG High is a General Support System that provides a secure application and hosting environment for VA applications, content, and utilities. These applications and services are used to deliver content to an audience made up of employees, veterans, contractors, partners across all VA medical centers and component facilities, Federal government, and the general public. Content and applications are provided by Veterans Benefits Administration (VBA), Veterans Health Administration (VHA), National Cemetery Administration (NCA) and Support Offices. VAEC provides the following services: Content delivery, Application Hosting and Management Services.
The VAEC infrastructure is hosted by MAG, a cloud service provider. The MAG platform is used to provide a variety of hosting environments to suit a variety of needs. MAG can support applications categorized up to High as rated in accordance with Federal Information Processing Standard (FIPS) 199. VA applications available to the public are hosted in MAG.
A dedicated private data link (MAG Express Route) provides all connectivity for on-premise VA resources communicating to the VAEC. Virtual Networks (VNets) wrap the applications within MAG to encapsulate network access. Access from the applications to VA internal resources such as Identity, Credential, and Access Management (ICAM) and Active Directory (AD) Services are conducted over the encrypted private data link to the VA network.
VAEC MAG is in multiple regions designed to allow U.S. government agencies, contractors and customers to move sensitive workloads into the cloud for addressing specific regulatory and compliance requirements. MAG does not manage logical access controls within the VAEC system boundary. VAEC offers the same level of security as other VA physical technology centers and supports existing VA security controls and certification requirements such as FISMA, HIPAA, HITECH, SAS-70, ISO 27001, FIPS 140-2 compliant end points, and PCI DSS.
The VAEC MAG High Architecture is provided in the Figure below:
VAEC-MAG Diagram LINK Figure 1: VAEC MAG High Architecture
Figure 2: VAEC MAG High Architecture Diagram i OFFICE OF INFORMATION SECURITY
Enter your organization System Description below:
VAEC Services The information above documents Service Level Agreements (SLAs) provided by the VAEC.
Enter your organization Service / Business Line in the table below:
The information below documents Service Level Agreements (SLAs) provided by the [Organization 2 name/acronym].
| Service / Business Line |
| MTD |
| VAEC MAG High |
| 6 hours |
| Service / Business Line |
| MTD |
| [Organization 2 name/acronym] [Security Categorization] |
| [MTD] |
The following IS Services are maintained at VAEC MAG:
| Service Line | IS Service [Applications/IS Support Services] | MTD | RTO |
| Authentication Services | 6 hours | 6 hours | |
| Server Configuration Management Service | 6 hours | 6 hours | |
| Vulnerability Scanning Service | 6 hours | 6 hours | |
| Monitoring Service | 6 hours | 6 hours | |
| JumpBox Service | 6 hours | 6 hours | |
| Auditing Service | 6 hours | 6 hours | |
| Code Configuration and Release Management Services | 6 hours | 6 hours |
Table 4: IS Services/Applications/IS Support Services
The following IS Services are maintained at [Organization 2 name/acronym].
Enter your organization IS Services/Applications/IS Support Services in the table below:
Service Line IS Service [Applications/IS Support Services] MTD RTO
Table 5: [Organization 2 name/acronym] IS Services/Applications/IS Support Services
DRP Assumptions and Constraints The following assumptions were used when developing this DRP:
The MAG is responsible for maintaining the DR site at the infrastructure level only.
· Applications that will leverage DR capabilities toolset within VAEC will require separate coordination and confirmation.
The MAG is responsible for annual DR testing at the infrastructure level only.
Personnel well versed in the day-to-day IS service operating procedures are available to operate at the recovery site.
Relocated IS Service recovery teams are prepared to deal with emergency procedures without the need to reference detailed written steps.
Management personnel with authority will be available to make decisions.
System and supporting component restoration priorities have been documented within each IS Service’s ISCP.
Alternate processing procedures (or workarounds) for critical business processes have been established by Service/Business lines and are listed in the ISCP for each system.
Current backups of the system software and data are intact and available at the offsite data storage facility or facilities.
The IS Service(s) at MAG are inoperable and cannot be recovered within the recovery time objective (RTO) required to allow the facility to continue to operate normally.
Additional assumptions:
Applications can inherit the VAEC MAG DRP. An application level DRP is required for application-specific Disaster Recovery processes.
This plan does not apply to the situations described below:
Disruptions deemed recoverable at the primary site.
Emergency evacuation of personnel addressed by the occupant evacuation plan.
Overall recovery of business operations. Service/Business line owners should address recovery of business operations in a separate business recovery plan.
Enter your organization DRP Assumptions and Constraints here:
DRP Considerations The MAG is responsible for backup and recovery of data needed to restore operations to the requisite sites.
As depicted in Figure 1, “VAEC MAG High Architecture Diagram”, the VAEC Infrastructure is designed to be highly available, fault tolerant and scalable by using the MAG Regions. The components of the VAEC Virtual Network (VNet) are designed to be hosted in designated MAG Regions.
The VAEC is designed with a comprehensive strategy to back up data, by first identifying the failure or disaster situation that can occur and the potential business impact. The VAEC leverages many built-in backup and recovery features of MAG. The VAEC also implements backup processes that offer the appropriate level of granularity to meet pre-defined RTO and RPO (listed in Table 3) that includes:
· File-level recovery
· Volume-level recovery
· Image-level recovery
· Object-level recovery
· Container-level recovery
The VAEC DRP takes into consideration the existing capabilities of the recovery site locations and documents them along with any associated limitations or restrictions that would hinder data and access for a data center, medical facility, benefits center, or national cemetery from being recovered as soon as possible after a major disruption.
The expectation is that critical IS services will continue by using strategies for continuing operations, including coordinating with continuity site personnel, processing data from remote locations, using cached data, transferring backups to another facility, and employing full-service restoration after the disruption by utilizing the recovery teams identified in this plan.
The DRP addresses all critical IS services as identified in the ISCPs. Facilities that are co-located or share infrastructure with other administrations should seek to ensure alignment with the continuity planning efforts developed for the other administrative or business units located in the facility.
Enter your organization DRP Considerations here:
DRP Roles and Responsibilities Roles and responsibilities are provided by MAG personnel and inherited from FedRAMP package #: F1603087869. The following table includes responsibilities that describe each role and identifies the individual or team responsible for executing or supporting system recovery.
| DRP Role |
| Job Title |
| Responsibilities |
| DRP Director |
| Director, Business Continuity Management |
| · Overall responsibility for the development, execution, maintenance, activation, and deactivation of the DRP |
· Authorizes all changes to the DRP
· Determines if manual or secondary processing should be initiated as a temporary method of maintaining business operations
· Ensures that an alternate DRP Director is designated
| DRP Director (Alternate) |
| Business Continuity IT Specialist |
| · Acts on behalf of the DRP Director should the need arise |
| DRP Coordinator |
| Program Director |
| · Monitors recovery team activities until the systems are recovered at the recovery site |
· Ensures that recovery operations are being performed consistent with service level agreements / service level requirements
· Provides periodic status updates to the DRP Director
· Files an after-action report (AAR) upon resumption of normal operations
| DRP Coordinator (Alternate) |
| Deputy Project Manager |
| · Acts on behalf of the DRP Coordinator should the need arise |
| Cybersecurity Engineering Team |
| Cybersecurity Team |
| · Provides cybersecurity oversight during DRP activities |
| VA POC |
| Infrastructure Manager |
| · Represents the recovery and restoration interests of the affected Service/Business line |
| VA POC (Alternate) |
| Technical Architect |
| · Represents the recovery and restoration interests of the affected Service/Business line |
| IS Services POC (Alternate) |
| Manager, Cloud |
Infrastructure
· Represents the recovery and restoration interests of the affected service/business line
| Disaster Recovery Activation Team |
| Disaster Recovery Activation Team includes the following groups: |
· Cognosante
· AWS/MAG
· Other Affiliated and/or Personnel as needed
· VA
· Determines the expected duration of the failover to the recovery site
· Prioritizes the sequence of resource recovery
· Performs all system recovery and resumption activities
· Powers on / off systems
· May retrieve backup tapes
· Configures systems
· Ensures voice and data communications are functioning, activate pagers, sat phones
· Provides IP numbers and network routing information
· Includes validation testing teams or personnel
· Responsible for coordinating recovery activities between each site’s teams
Table 6: Roles and Responsibilities
Roles and responsibilities are provided by [Organization 2 name/acronym] personnel and inherited from FedRAMP package #: F1603047866. The following table includes responsibilities that describe each role and identifies the individual or team responsible for executing or supporting system recovery.
Enter your organization DRP Roles and Responsibilities in the table below:
| DRP Role |
| Job Title |
| Responsibilities |
Table 7: Roles and Responsibilities [Organization 2 name/acronym]
Threats and Vulnerabilities The data gathered through the Information System Contingency Planning Assessment (ISCPA) process is used in the completion of a Business Impact Analysis (BIA), a Vulnerability assessment, and a Threat assessment. The results of the Threat and Vulnerability assessments for VAEC MAG are shown in Tables 5 and 6.
| Threat | Likelihood | Capacity | Threat Rating |
| Unpatched operating systems create a potential threat to the system | Low | Low | Low |
| Network misconfigurations create a potential threat to the system | Low | Low | Low |
Table 8: VAEC MAG Threat Assessment Enter your Organization Threat Assessment here:
Threat Likelihood Capacity Threat Rating
Table 9: [Organization 2 name/acronym] Threat Assessment
| IS Service | Threat | Vulnerability | Exploit Potential | Mitigation Effectiveness | Mitigation Strength |
| System Administration | Unpatched operating systems create a potential threat to the system | Operating Systems Required Patch Service Updates | Low | High | High |
| System Administration | Network misconfigurations create a potential threat to the system | Network daemons not managed by the package system | Low | High | High |
Table 10: VAEC MAG Vulnerability Assessment Enter your Organization Vulnerability Assessment here:
IS Service Threat Vulnerability Exploit Potential Mitigation Effectiveness Mitigation Strength
Table 11: [Organization 2 name/acronym] Vulnerability Assessment
CONCEPT OF OPERATIONS
The Concept of Operations section provides an overview of the VAEC MAG DRP activation and notification, recovery and, reconstitution phases, and supporting roles and responsibilities of VAEC MAG personnel during disaster activation.
The MAG is responsible for backup and recovery of data needed to restore operations to the requisite sites. VAEC inherits these responsibilities from MAG’s FedRAMP package #: F1603087869.
Disaster recovery (DR) is the function of preparing for and recovering from a disaster, that is, any event having a negative impact on your business continuity or security. This could include hardware or software failure, a network outage, a power outage, physical damage to a building such as fire or flooding, human error, malicious occurrence, or some other significant disaster. Several Federal Information Processing Standards (FIPS) and National Institute of Standards and Technology (NIST) policies are relevant to VA in this context: FIPS 199, FIPS 200, NIST SP 800-34, and NIST SP 800-53. Additionally, VA Handbooks 6500, 6500.5, and 6500.8 address how the 800- 53 controls are applied and implemented for systems and applications developed for, or used by, VA. Regular testing of DR plans is critical to train staff and to assess and validate DR components. Without regular, rigorous, and realistic testing, DR plans and solutions are almost certain to fail in the event of a disaster. A key concept is the difference between high availability (HA) and DR. As a practical matter, HA addresses routine component, sub-system or system failures, while DR addresses catastrophic failures or events that could take down an entire site.
Proper DR planning requires understanding of the following industry terms:
• Recovery Time Objective (RTO) – The time it takes after a disruption to restore a business process to its service level, as defined by the DR plan with the program’s accreditation package.
• Recovery Point Objective (RPO) – The acceptable amount of data loss measured in time.
In addition to these industry standards, VA also uses and requires the following:
• Business Impact Analysis (BIA) – A BIA, as defined by NIST Special Publication (SP) 800-34, Rev. 1, Contingency Planning Guide for Federal Information Systems, is an analysis of an information system’s requirements, functions, and interdependencies, and must correlate information systems with critical mission/business processes (e.g. VA’s Mission Essential Functions) and respective services.
• Maximum Tolerable Downtime (MTD) – Total amount of time that is acceptable for a mission/business process outage or disruption, determined by utilizing BIA constructs.
Enter your organization concept of operation here:
Activation and Notification The Activation and Notification Phase defines initial actions taken once a disruption has been detected and recovery activities appear to be imminent. The DRP Activation and Notification Phase defines the activities required to activate the DRP and notify supporting recovery personnel. These phases include preparing for deployment of support personnel to the specified recovery site or coordinating activities between the primary site and the site where data or operations will be temporarily processed. The following sub sections further describe the activation and notification activities for sites that will transfer data or processing to another facility or implement a full mobilization.
Enter your organization activation and notification here:
DRP Activation Criteria and Procedures The VAEC MAG DRP may be activated if one or more of these criteria are met:
The VAEC MAG facility is rendered unusable;
Any or all of the critical IS Services residing within the facility will not be available within the accepted recovery time objective (RTO) set forth in the individual ISCPs.
Additionally, the decision to activate the VAEC MAG DRP may require the DRP director to consult with the service/business Line POC’s listed in Appendix A.
Enter your organization DRP activation criteria and procedure here.
Notification Procedures The first step after activation of the VAEC MAG DRP is the notification of key personnel, stakeholders and IS Service(s) recovery support teams. This includes teams whose operations will be transferred to the recovery facility. However, this may be accomplished by notifying the Regional Operations Center who will coordinate and carry out notifications.
Notification procedures may include:
Identification of who makes the initial notifications The sequence in which personnel are notified (e.g., system owner, technical POC, contingency plan coordinator, business/service line POC, and Recovery Team POC);
The method of internal and external notifications (e.g., email, mobile phone, automated notification system; etc.);
What to do if any single person in the notification sequence cannot be reached; and Alert/notification messages.
Enter your organization notification procedure here:
For a full list of all DRP specific key personnel and contact information, refer to Appendix A.
For a list identifying leadership, recovery personnel and any facility POCs that are to be alerted of the DRP activation, refer to Appendix B.
Call trees are an effective means of conveying the communication sequence in which leadership, recovery personnel, and facility POCs should be alerted. Refer to Appendix B.
Accountability of OIT Recovery Personnel In the event of a natural or man-made disaster, MAG Package #: F1603087869 states that all requisite infrastructure and services required to restore site operations are replicated across MAG data centers (zones) and/or regions.
In the event of a natural or man-made disaster, DoD directives stipulate that all agencies achieve 100% personnel accountability. Normally, personnel accountability begins with the activation of your Directorate phone trees/recall roster. Additionally, the Veterans Administration has employed a mass notification system referred to as the VA-Personnel Accountability System (VA-PAS). VA-PAS is a bidirectional mass alert system that provides proactive assessment of personnel status and safety following any emergency, and maintaining communications while incidents are resolved.
Enter your organization accountability of OIT recovery personnel here.
Recovery Site(s) VAEC MAG is responsible for recovery of data needed to restore operations to the requisite sites (application/system teams are responsible for leveraging the appropriate Cloud Services to ensure appropriate recover). VAEC inherits recovery from MAG Package #: F1603087869. VAEC documents recovery and reconstitution procedures for IS services in the VAEC ISCP. This information is in Appendix F.
Enter your organization recovery site(s) here.
RECOVERY
The recovery phase provides formal recovery operations that begin after the DRP has been activated, outage assessments have been completed (if possible), personnel have been notified, and in some situations, appropriate teams have been mobilized.
Recovery Phase activities focus on implementing recovery strategies to restore the site / facilities IS Services and data capabilities at a recovery facility. At the completion of the recovery phase, VAEC MAG and [Organization 2 name/acronym] IS Services and data capabilities will be operational at the recovery site and capable of performing the functions identified in the plan. These functions will continue until IS Services and components are restored, repair of damage is completed, and the operational capabilities at the original or new permanent location are resumed.
Sequence of Recovery Activities High-level activities that occur during the recovery of VAEC MAG’s and [Organization 2 name/acronym] ‘s systems, data and operations are identified in their associated ISCPs listed in Appendix G.
Escalation Notices / Awareness Notifications include problem escalation to leadership and status awareness to system owners and users. Call trees are an effective means of conveying the communication sequence in which leadership, recovery personnel and facility points of contact should be alerted.
RECONSTITUTION
Reconstitution is the process by which a facility is tested to validate that all its system capabilities and functionalities are fully restored and operational. During the Reconstitution phase, recovery activities are completed, and normal systems operations are resumed, and operations are transitioned back to the primary site (if needed). If the original facility is “unrecoverable”, the activities in this phase can be applied to preparing the recovery site as a new permanent location to support system processing requirements.
The MAG is responsible for reconstitution of data needed to restore operations to the requisite sites. VAEC inherits recovery from MAG Package #: F1603087869. VAEC documents recovery and reconstitution procedures for IS services in the VAEC ISCP.
Enter your Organization reconstitution here:
Concurrent Processing – IS Services In concurrent processing, an IS Service operates at separate locations concurrently until there is a level of assurance that the recovered system is operating correctly. Please refer to the individual ISCPs for details regarding concurrent processing strategies for each IS Service resident within VAEC MAG. For a list of associated plans, see Appendix G.
Enter your Organization concurrent processing IS services here:
Data Validation and Functionality Testing
Data validation and functionality testing is the process of validating recovered data, data files or databases and testing to confirm that functionality have been recovered completely. Please refer to the individual ISCPs for details for each IS Service resident within VAEC MAG.
Enter your Organization data validation and functionality testing here:
For a list of associated plans, see Appendix G.
Reconstitution Declaration
Upon successfully completing testing and validation, the DRP director will formally declare recovery efforts complete. The DRP director will notify facility leadership, business / service line and technical POCs of the declaration.
Notifications (Users)
Upon return to normal system operations, VAEC MAG users will be notified by DRP Director using predetermined notification procedures (e.g., VA-PAS, email, broadcast message, phone calls, etc.).
Enter your Organization notifications (Users) here:
Cleanup
Cleanup is the process of restocking supplies used, returning manuals or other documentation to their original locations, and readying the system for a possible future contingency event. Please refer to the individual ISCPs for details for each IS Service resident within VAEC MAG.
Enter your Organization cleanup here:
For a list of associated plans, see Appendix G.
Event Documentation It is important that all recovery events throughout the transition of operations to the continuity site and back to the primary site, be well documented and lessons learned for inclusion and update to the DRP. It is the responsibility of each recovery team or person to document their actions during the recovery effort and to provide that documentation to the DRP coordinator. Alternatively, one of the recovery teams may be appointed the task of tracking the events. Event documentation should include:
Activity logs (including recovery steps performed and by whom, the time the steps were initiated and completed, and any problems or concerns encountered while executing activities);
Functionality and data testing results;
Lessons learned documentation; and After Action Report (AAR).
Deactivation
Once all activities have been completed and documentation has been updated, the DRP director or designee will formally deactivate the DRP recovery efforts. Notification of this declaration will be provided to all business and technical POCs.
TEST, TRAINING AND EXERCISE
The MAG is responsible for reconstitution of data needed to restore operations to the requisite sites. VAEC inherits recovery from MAG Package #: F1603087869. VAEC documents recovery and reconstitution procedures for IS services in the VAEC ISCP. Comment by Hill, Kristen A. (Cognosante): This statement needs to be clarified to include GSS/Infrastructure and should be made clear that it’s the application’s responsibility to have a plan or make arrangements to leverage the MAG capabilities.
All DRPs should be reviewed and tested at least yearly or whenever there is a significant change to the system.
NOTE: Full functional tests of systems are normally failover tests to the alternate locations and may be very disruptive to system operations if not planned well. Other systems located in the same physical location may be affected by or included in the full functional test. It is highly recommended that several functional tests be conducted and evaluated prior to conducting a full functional (failover) test.
A formal test plan is developed prior to the tabletop or functional test and exercise. Functional test procedures are developed to include key sections of the DRP, including a walk-through of the following:
Notification procedures;
System recovery on an alternate platform from backup media;
Internal and external connectivity; and Reconstitution procedures.
Results of the test are documented in an AAR, and lessons learned are developed for updating information in the DRP.
Persons or teams assigned DRP roles must be trained to efficiently and correctly respond to an event affecting VAEC MAG. VA OIT has developed a test, training, and exercise (TT&E) program compliant with the Homeland Security Exercise and Evaluation Program (HSEEP) which supports the following objectives:
Ensure that VAEC MAG’s personnel are familiar with the DRP and its associated activation, recovery and reconstitution procedures.
Validate DRP policies and procedures.
Exercise procedures through the use of tabletop and functional exercises, as appropriate.
Ensure that hardware, software, backup data, and records required to support recovery at a continuity site are available.
Exercises should be designed for the complete loss of a facility. The exercise should be a functionality test of application and infrastructure to ensure ability to continue critical business processes meeting mission essential functions of the VA enterprise performed by the primary site.
| Activities | Tests | Training | Exercises |
| Frequency |
| Internal and External Interdependency Testing of internal and external interdependencies identified in the OI&T CEMP plans, with respect to performance of, and other agencies’ MEFs. |
| Annually |
| Test DRP’s communications. |
| Annually |
| Continuity communications testing of communications equipment (both secure and non-secure) to ensure the internal and external interoperability and viability of continuity communications systems and capabilities. |
| Annually |
| Test primary and backup infrastructure systems and services at alternate operating facilities, (e.g., power, water, fuel). |
| Annually |
| Alert, Notification, and Activation Procedures Testing for mission critical/emergency personnel. |
| Annually |
| DHS test reporting of the formal reporting processes of test results as directed by the Department of Homeland Security (DHS) / Federal Emergency Management Agency (FEMA). This report is prepared by the Office of Operations, Security and Preparedness (OSP) with input from the Administrations and Staff Offices to include OIT. |
| Annually |
| Alternate Facility Logistics Testing and exercising of required physical security capabilities at the identified alternate facility(ies). |
| Annually |
| Test recovery of vital classified and unclassified records, critical information systems, services, and data. |
| Annually |
| Documenting and reporting testing of the internal processes for formally documenting and reporting tests and their results. |
| Annually |
| Test DRP’s notification/activation procedures. |
| Annually |
| CEMP Awareness/Orientation training: a high-level overview presentation of CEMP concepts for all OIT staff (both mission critical/emergency personnel and non-mission critical/emergency personnel, to include contractors). |
| Annually |
| Relocation exercise to the recovery site. |
| Annually |
Table 12: TT&E Calendar
Enter your Organization deactivation here:
| Enter test, training, and exercise (TT&E) results in the table below: |
| Activities | Tests | Training | Exercises |
| Frequency |
Table 13: TT&E Calendar [Organization 2 name/acronym
DOCUMENT MANAGEMENT
Document Ownership The contents of this document are the responsibility of VAEC MAG, which has assigned the DRP coordinator responsibility for its content, modifications, currency, distribution to stakeholders, and its presence in the VA document repository.
Enter your Organization document ownership here:
Plan Review and Maintenance To ensure currency, this document will be reviewed annually in conjunction with the annual DRP test / exercise and when systems at this facility incur significant modifications.
The DRP is a component of the overall VAEC Management Plan. This plan, and all related project management plans, can be found in the GRC tool.
The Project Management Team will review the DRP every 365 days and as necessary to address project conditions, client input, or contract changes.
This document will be subject to formal change control after approval by the VA contracting officer. Subsequent changes to this document will be requested, approved, and implemented in accordance with the change management processes defined in the Change Management Plan. Changes to this document will be recorded in the revision history table.
The most current version of the Privacy and Security Management Plan will be posted to the VAEC instance in the GRC tool.
Enter your Organization plan review and maintenance here:
Document Distribution A copy of the MAG DRP is in the MAG FedRAMP package #: F1603087869. A copy of the VAEC MAG DRP will be:
Provided to facility stakeholders (listed in Plan Distribution table at the front of this document) who have an interest or responsibility for the development or testing of this plan.
Held electronically or in hard copy or both by every member of the recovery teams where it is easily accessible in an emergency.
Stored in the VA document repository.
Stored in an off-site location in both soft and hard copy format for ease of use under a wide range of circumstances
Enter your Organization document distribution here:
: DRP Personnel Contact Data - VA DRP Leadership
| Key Personnel |
| Contact Information |
| DRP Director |
| Work #: 732-440-9583 |
| Name: David Catanoso |
| VA Cellular #: 732-440-9583 |
| Title: Director, Enterprise Cloud Solutions Office (ECSO) |
| VA E-mail: david.catanoso@va.gov |
| DRP Director (Alternate) |
| Work #: 801-588-5013 |
| Name: Sue Speck |
| VA Cellular #: 801-209-4839 |
| Title: Deputy Director, Enterprise Cloud Solutions Office (ECSO) |
| VA E-mail: sue.speck@va.gov |
| Cybersecurity Engineering Team |
| Work #: (561)291-2175 |
| Name: Joseph Fourcade |
| VA Cellular #: (561)291-2175 |
| Title: Cyber Security Analyst |
| VA E-mail: Joseph.Fourcade@va.gov |
| DRP Coordinator |
| Work #: (202) 461-0447 |
| Name: Joseph Stevenson |
| VA Cellular #: |
| Title: VAEC Implementation Team Leader & COMS PM |
| VA E-mail: Joseph.Stevenson@va.gov |
| DRP Coordinator (Alternate) |
| Work #: TBD |
| Name: TBD |
| VA Cellular #: TBD |
| Title: MAG Program Manager Lead |
| VA E-mail: TBD |
| Outage Assessment Team POC |
| Work #: TBD |
| Name: Christopher Cardella |
| VA Cellular #:512-590-9414 |
| Title: Manager, Cloud Infrastructure |
| VA E-mail: Christopher.Cardella@va.gov |
| Alternate Outage Assessment Team POC |
| Work #: 512-981-4792 |
| Name: LaDonn Wittenburg |
| VA Cellular #: |
| Title: Implementation Manager, Service Management and Planning |
| VA E-mail: ladonn.wittenburg@va.gov |
Table 14: DRP Personnel Contact Data – VA
Enter your Organization DRP Personnel Contact data here:
[Organization 2 name/acronym] DRP Leadership
| Key Personnel |
| Contact Information |
| DRP Director |
| Work #: |
| Name: |
| VA Cellular #: |
| Title |
| VA E-mail: |
| DRP Director (Alternate) |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| Cybersecurity Engineering Team |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| Cybersecurity Engineering Team (Alternate) |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| Title: |
| VA E-mail: |
| DRP Coordinator |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| DRP Coordinator (Alternate) |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| Outage Assessment Team POC |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| Alternate Outage Assessment Team POC |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: Implementation Manager, Service Management and Planning |
| VA E-mail: |
Table 15: DRP Personnel Contact Data – [Organization 2 name/acronym]
Primary Site VAEC MAG Key Personnel
| Key Personnel |
| Contact Information |
| Team Lead |
| Work #: 512-326-6180 |
| Name: Christopher Cardella |
| VA Cellular #: 512-590-9414 |
| Title: Manager, Cloud Infrastructure |
| E-mail: Christopher.Cardella@va.gov |
| Team Member |
| Work #: 512-981-4792 |
| Name: LaDonn Wittenburg |
| VA Cellular #:512-364-3517 |
| Title: IT Specialist, Core Systems Engineering |
| E-mail: LaDonn.Wittenburg@va.gov |
Table 16: Primary Site VAEC MAG Key Personnel Contact Data
Enter your Organization primary site key personnel in the table below.
Primary Site [Organization 2 name/acronym] Key Personnel
| Key Personnel |
| Contact Information |
| Team Lead |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| E-mail: |
| Team Member |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| E-mail: |
Table 17: Primary Site [Organization 2 name/acronym] Key Personnel Contact Data
Recovery Site VAEC MAG Key Personnel
| Key Personnel |
| Contact Information |
| Team Lead |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| Team Member |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
Table 18: Recovery Site VAEC MAG Key Personnel Contact Data
Enter your Organization recovery site key personnel in the table below.
Recovery Site [Organization 2 name/acronym] Key Personnel
| Key Personnel |
| Contact Information |
| Team Lead |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
| Team Member |
| Work #: |
| Name: |
| VA Cellular #: |
| Title: |
| VA E-mail: |
Table 19: Recovery Site [Organization 2 name/acronym] Key Personnel Contact Data
: Call Tree
| Name |
| Role |
| Telephone |
| Priority |
| David Catanoso |
| DRP Director |
| Work #: 732-440-9583 |
Cell #: 732-440-9583
| David.Catanoso@va.gov |
| 1 |
| Joseph Stevenson |
| DRP Director (Alternate) |
| Work #: 202-461-0447 |
Cell #:
| Joseph.Stevenson@va.gov |
| 2 |
| Joseph Stevenson |
| DRP Coordinator |
| Work #: 202-461-0447 |
Cell #:
| Joseph.Stevenson@va.gov |
| 3 |
| Christopher Cardella |
| DRP Coordinator (Alternate) |
| Work #: (512)-981-4792 |
| Christopher.Cardella@va.gov |
| 4 |
Figure 3: DRP Call Tree
| 1. | The help desk team calls the DRP Director (or alternate), opens a communications bridge over Skype, and distributes a meeting invitation to all participants |
| 2. | The DRP Director calls the Program Director (or Deputy Project Manager) |
| 3. | The Program Director calls the ISO, the Cybersecurity Engineer, the Deputy Project Manager, the Project Coordinator, and the DRP Coordinator |
| 4. | The DRP Coordinator notifies all infrastructure team members as necessary |
| 5. | The ISO notifies the VA security team as necessary |
| 6. | The Program Director notifies VA program management in accordance with notification protocols and requirements |
Enter your Organization call tree in the table below:
| Name |
| Role |
| Telephone |
| Priority |
Figure 4: DRP Call Tree [Organization 2 name/acronym]
| 1. | The help desk team calls the DRP Director (or alternate), opens a communications bridge over Skype, and distributes a meeting invitation to all participants |
| 2. | The DRP Director calls the Program Director (or Deputy Project Manager) |
| 3. | The Program Director calls the ISO, the Cybersecurity Engineer, the Deputy Project Manager, the Project Coordinator, and the DRP Coordinator |
| 4. | The DRP Coordinator notifies all infrastructure team members as necessary |
| 5. | The ISO notifies the VA security team as necessary |
| 6. | The Program Director notifies VA program management in accordance with notification protocols and requirements |
: Personnel Contact Data – Vendors
| Vendor Contact Data |
| Details |
| Vendor Name |
| Microsoft Azure Government |
| Vendor Type |
| Cloud Service Provider |
| Address |
| One Microsoft Way |
| City, State, and ZIP Code |
| Redmond, Washington 98052 |
| Primary Contact Name |
| Scott Goad |
| Office Phone Number |
| (469) 775-2001 |
Emergency Phone Number
| Email Address |
| sgoad@microsoft.com |
| Secondary Contact Name |
| Ryan Holt |
| Office Phone Number |
| (240) 620-6585 |
| Email Address |
| Ryan.Holt@microsoft.com |
SLA/MOU
Special Instructions
Table 21: Vendor Contact Data
Enter your organization personnel contact data in the table below:
| Vendor Contact Data |
| Details |
Vendor Name
Vendor Type
Address
City, State, and ZIP Code
Primary Contact Name
Office Phone Number
Emergency Phone Number
Email Address
Secondary Contact Name
Office Phone Number
Email Address
SLA/MOU
Special Instructions
Table 22: [Organization 2 name/acronym] Contact Data
: Alternate Storage Site VAEC is a highly available system that automatically fails over to MAG Region 2. Addresses for these sites are not published.
| Alternate Storage Site |
| Details |
| City |
| Not published |
| State |
| Not published |
| Distance from Primary Facility |
| Not published |
| Is VA Owned? |
| No; however, the data is located in a VA-controlled environment within the FedRAMP System High-approved MAG |
| POC Name |
| Windows Azure Live Site (WALS) Team |
wals@microsoft.com https://azure.microsoft.com/en-us/overview/clouds/government/contact/
This is the start of the file's text. The full file is on GovTribe.
File details come from the government source that posted it. Updated .