EPLC_Disaster Recovery Plan_v . .doc

Contingency Plan --- Custom Document Property (SysName) Document Version 2.0

Contingency Plan (CP)

--- Custom Document Property (SysName)

--- Custom Document Property (SysAcronym)

--- Custom Document Property (DocDate)

Food and Drug Administration

(FDA)

Record of Changes

Tip: This page tracks changes to template. Remove these template entries when developing this document for a specific system and enter changes for this specific document. (Remove this tip prior to submission.)

Tip: Items that begin with the text “Custom Document Property” and NOT to be typed over. Go to File>Properties> and modify the values of these properties on the Custom tab. After the changes are made, press Ctrl A to select the entire document and then press F9 to update the fields throughout the document. Custom Document Properties in the Header can only be updated when the Header is open. Select View>Header and Footer, select the Custom Document Properties and press F9. (Remove this tip prior to submission.)

Table of Contents Introduction

Purpose

62.1 Applicability

62.2 Planning Principles

72.3 Assumptions

72.4 References/Requirements

System Identification

93.1 System Name/Title

93.2 Responsible Organization

93.3 Information Contacts and Line of Succession

103.4 Contingency Team Roles and Responsibilities

113.5 Information System Categorization

113.6 Mission Criticality

123.7 System Description and Architecture

123.8 System Interconnection/Information Sharing

133.9 Risk Assessment Results

Notification and Activation Phase

154.1 Damage Assessment Procedures

154.2 The Contingency Plan Activation

164.3 Alternate Location Procedures

Recovery Operations Phase

175.1 Recovery Procedures and Objectives

Reconstitution Phase

196.1 Concurrent Processing

196.2 Plan Deactivation

Training, Testing and Exercises

207.1 IT Testing and Exercise

207.2 Documenting Test Plan Results

207.3 Plan Maintenance

A-1Appendix A Contingency Plan Organization Chart

B-1Appendix B Emergency/Disaster Team Formulation

B-1Contingency Plan Work Group

B-1Procurement Team

B-1Restoration Management Team

B-2Contingency Coordination Team

B-2Disaster/Emergency Response Team

B-3Recovery/Restoration Team

B-3Acceptance Team

C-1Appendix C Additional Notification Contact List

D-2Appendix D Vendor Contact List

E-1Appendix E Equipment and Specifications

F-1Appendix F Business/Mission Impact Analysis

G-1Appendix G IT Standard Operation Procedures

H-1Appendix H System Backup Procedures

I-1Appendix I Damage Assessment Report

J-1Appendix J Contingency Log and Action Item Checklist

K-1Appendix K IT Contingency Plan Test Plan

L-1Appendix L IT Contingency Plan Test Results

M-1Appendix M Acceptance Test Plan

N-1Appendix N Service Level Agreements

O-1Appendix O Additional Contingency Plans

P-1Appendix P Emergency Management Plan

Q-1Appendix Q Continuity of Operations Plan

Table of Tables

10Table 1: Information Contacts

11Table 2: Information System Categorization

13Table 3: Interconnections Service Agreements

14Table 4: Risk Assessment Results

16Table 5: Alternate Facilities

21Table 6: System Plan Maintenance

B-1Table 7: Contingency Plan Work Group Membership

B-1Table 8: Procurement Team Membership

B-1Table 9: Restoration Management Team Membership

B-2Table 10: Contingency Coordination Team Membership

B-2Table 11: Disaster/Emergency Response Team Membership

B-3Table 12: Contingency Coordination Team Membership

B-3Table 13: Recovery Process

B-4Table 14: Acceptance Team Membership

C-1Table 15: Additional Notification Contact List

D-2Table 16: Vendor Contact List

E-1Table 17: System Hardware

E-1Table 18: System/Application Software

F-1Table 19: Business/Mission Impact Analysis Summary

F-1Table 20: Business/Mission Impact Analysis Detail

G-1Table 21: Recovery and Restoration Team Procedures

G-1Table 22: Acceptance Team Procedures

K-1Table 23: Contingency Plan Testing Requirements

K-1Table 24: Contingency Plan Test Schedule for --- Custom Document Property (SysAcronym)

1 Introduction This document contains the --- Custom Document Property (SysName) (--- Custom Document Property (SysAcronym)) Information Technology (IT) Contingency Plan (CP). It is intended to serve as the centralized repository for the information, tasks, and procedures that would be necessary to facilitate the [System Name] management’s decision-making process and its timely response to any disruptive or extended interruption of the FDA's normal business operations and services. This is especially important if the cause of the interruption is such that a prompt resumption of operations cannot be accomplished by employing only normal daily operating procedures.

In terms of personnel and financial resources, the information tasks and procedures detailed in this plan represent the --- Custom Document Property (SysAcronym) management’s demonstrated commitment to response, resumption, recovery, and restoration planning. Therefore, it is essential that the information and action plans in this plan remain viable and be maintained in a state of currency in order to ensure the accuracy of its contents. To that end, this introduction is intended to introduce and familiarize its readers with the organization of the plan.

Since the information contained in this document describes --- Custom Document Property (SysAcronym) management’s planning assumptions and objectives, the plan should be considered a sensitive document. All of the information and material contents of this document should be labeled, “For Official Use Only”.

The --- Custom Document Property (SysAcronym) management has recognized the potential financial and operational losses associated with service interruptions and the importance of maintaining viable emergency response, resumption, recovery and restoration strategies. This ITCP is intended to provide a framework for constructing plans to ensure the safety of employees and the resumption of time-sensitive operations and services in the event of an emergency and/or disaster (fire, power or communications blackout, tornado, hurricane, flood, earthquake, civil disturbance, etc.).

Although the --- Custom Document Property (SysAcronym) ITCP provides guidance and documentation upon which to base emergency response, resumption, and recovery planning efforts, it is not intended as a substitute for informed decision-making. Business process managers and executives must identify services for which disruption will result in significant financial and/or operational losses.

Tip: Plans should include detailed responsibilities and specific tasks for emergency response activities and business resumption operations based upon pre-defined time frames. Constructing a plan and presenting it to senior management may satisfy the immediate need of having a documented plan. However, this is not enough if the goal is to have a viable response, resumption, recovery, and restoration capability. (Remove this tip prior to submission.)

An ITCP is not a one-time commitment and is not a project with an established start and end date. Instead, ITCP’s are an on-going, funded business activity budgeted to provide resources required to:

· Perform activities required to update and create plans.

· Train employees on a continuous basis.

· Develop and revise policies and standards as the department changes.

· Exercise strategies, procedures, and team and resource requirements.

· Re-exercise unattained exercise objectives.

· Report on-going continuity planning to senior management.

· Research processes and technologies to improve resumption and recovery efficiency.

· Perform plan maintenance activities.

The ITCP elements necessary to create a viable, repeatable and verifiable continuity capability include:

· Implement accurate and continuous vital records, data backup, and off-site storage.

· Implement capability for rapid switching of voice and data communication circuits to alternate site(s).

· Provide an alternate site for operations.

· Construct contingency response teams.

2 Purpose

Tip: If this is the first time the ITCP has been written, the Contingency Planning Officer should consult with system owners, developers and/or staff concerning information on current Contingency / Disaster Recovery Planning (DRP). Indicate here if this is the first time the ITCP has been written. If testing has been conducted in the past, input those results here. (Remove this tip prior to submission.)

This ITCP establishes procedures to recover and reconstitute --- Custom Document Property (SysAcronym) following a disruption. The following objectives have been established for this plan:

· Maximize the effectiveness of contingency operations through an established plan that consists of the following phases:

· Notification/Activation phase

· Recovery phase

· Reconstitution phase

· Identify the activities, resources, and procedures needed to carry out --- Custom Document Property (SysAcronym) processing requirements during prolonged interruptions to normal operations.

· Assign responsibilities to designated personnel and provide guidance for recovering --- Custom Document Property (SysAcronym) during prolonged periods of interruption to normal operations.

· Ensure coordination with other staff that will participate in the contingency planning strategies. Ensure coordination with external points of contact and vendors who will participate in the contingency planning strategies.

2.1 Applicability

The --- Custom Document Property (SysAcronym) ITCP applies to the functions, operations, and resources necessary to restore and resume --- Custom Document Property (SysAcronym) operations at [Primary Location Name], [City], [State]. The --- Custom Document Property (SysAcronym) ITCP applies to all personnel associated with --- Custom Document Property (SysAcronym) as identified under Section 2.3, Responsibilities.

The --- Custom Document Property (SysAcronym) ITCP is supported by [Plan Names, i.e., FDA COOP, System DRP, Configuration Documentation, Design Documentation], which provides the [Purpose of the plans]. Procedures outline in this plan are coordinated with and support the aforementioned documents.

2.2 Planning Principles

This plan does not identify contingencies for every possible scenario. Rather, it focuses on two scenario types: (1) Minor System Failure and (2) Major System Failure. Full ITCP implementation may not be required for all disruptions. In some cases, partial implementation may be appropriate. The --- Custom Document Property (SysAcronym) ITCP is based on the following scenarios: ITCP will not be activated for a Minor System Failure; however, should be considered for full implementation for most Major System Failures. The scenarios are described in more detail below.

2.2.1 Scenario 1 – Minor System Failure

A localized threat has caused a disruption of --- Custom Document Property (SysAcronym) services in one or both of the following:

· The application has become inoperable for a period of less than [24/12/6/3] hours and major components of the system are still maintained and operational.

· The facility that houses the failed component(s) has not been physically damaged to an extent that requires personnel evacuation or major repair to the environment.

2.2.2 Scenario 2 – Major System Failure

A localized threat has caused a disruption of --- Custom Document Property (SysAcronym) services in one or both of the following:

· The application has become inoperable for a time period (present or expected) to exceed more than [24/12/6/3] hours.

· The facility that houses the failed component(s) has been physically damaged to an extent that requires temporary personnel evacuation or major repair to the environment.

2.3 Assumptions

Based on these principles, the following assumptions were used when developing the ITCP:

· The GSS Local Area Network/Wide Area Network (LAN/WAN) is operational.

· Key --- Custom Document Property (SysAcronym) personnel have been identified and trained in their emergency response and recovery roles and are available to activate the ITCP.

· Access to backup data, configuration files, hardware, and software is accessible by key ITCP personnel.

· Service level agreements (SLA) are maintained with system hardware and software vendors to support emergency system recovery.

· The --- Custom Document Property (SysAcronym) ITCP is up to date and accessible to key personnel responsible for executing the plan.

· The procedures within the plan have been tested to ensure that the strategies are viable.

Tip: Add additional assumptions as necessary. (Remove this tip prior to submission.)

The --- Custom Document Property (SysAcronym) ITCP does not apply to the following situations:

· Overall recovery and continuity of business operations;

· Emergency evacuation of personnel;

· Detailed procedures for the relocation of IT functions to an alternate site if the original site is damaged.

Tip: Add additional situations as necessary. (Remove this tip prior to submission.)

2.4 References/Requirements

ITCP’s are constructed using five components. These are depicted in Figure 1.

--- Custom Document Property (SysAcronym) ITCP complies with the FDA Policies as follows:

· System Owners will develop CP’s for information systems under their purview in accordance with National Institute of Standards and Technology (NIST) Special Publication (SP) 800-34.

· System Owners will review the --- Custom Document Property (SysAcronym) ITCP once a year, update them as necessary, and communicate any changes to the Program Office responsible for COOP, if applicable.

· System Owners will ensure that plans for systems rated Low, Moderate, or High are tested/exercised at least annually. Testing will be coordinated with elements responsible for COOP and incident response. For systems rated Moderate and High, the level of rigor for testing must increase every year. For systems rated High, the System Owners will ensure testing at the alternate processing site.

The --- Custom Document Property (SysAcronym) ITCP also complies with the following federal and departmental policies:

· The Computer Security Act of 1987;

· Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources, Appendix III, November 2000;

· Federal Information Processing Standards Publication (FIPS PUB) 87, Guidelines for ADP Contingency Planning, March 1981 (superseded by this publication);

· Federal Preparedness Circular (FPC) 65, Federal Emergency Management Agency (FEMA) June 15, 2004;

· Homeland Security Presidential Directive (HSPD) 8, National Preparedness, December 17, 2003.

· National Institute of Standards and Technology (NIST) Special Publication (SP) 800-34, Contingency Planning Guide for Information Technology Systems, June 2002;

· NIST SP 800-100, Information Security Handbook: A Guide for Managers, October 2006.

Tip: Add in additional departmental policies if needed. (Remove this tip prior to submission.)

3 System Identification

3.1 System Name/Title

Tip: Do NOT type in these fields. They are provided automatically after you update the Custom Document Properties and refresh all fields. Go to File > Properties > Custom. Modify the properties. Close the dialog box. Press Ctrl A to select the entire document. Press F9 to update all fields. You must open the Header and Footer, select the fields, and press F9 to update them. (Remove this tip prior to submission).

--- Custom Document Property (SysName)/--- Custom Document Property (SysAcronym)

3.2 Responsible Organization

Tip: List the organization that owns and is responsible for the data in the application or the organization that is responsible for supporting the GSS. The responsible organization owns the data, and controls the use of the data. List the federal organizational sub-component responsible for the system such as a contractor, what role that contractor performs, and describe the relationship with the System owner organization. Be specific about the organization and do not abbreviate. Include phone numbers, physical locations, and addresses. (Remove this tip prior to submission).

The responsible organization owns the --- Custom Document Property (SysAcronym) system, the data it contains, and controls the use of the data.

Example: (Remove this tip prior to submission).

System Owner:

Center for Devices and Radiological Health (CDRH)

Office of the Director

U.S. Food and Drug Administration

202-555-1234 x1234

555 99th Street S.W., Washington, DC 20410 Example: (Remove this tip prior to submission).

The System is maintained by:

Appropriate Contractor Firm

123-456-7890 x0000

1234 Main St

Anywhere, USA, 12345

3.3 Information Contacts and Line of Succession

Tip: Specify the program owner, program manager, Information System Security Officer (ISSO), and the system manager for the system. Include their address, telephone numbers, and e-mail. List the contact information for one or more persons designated to be the point(s) of contact for this system. The designated persons should have sufficient knowledge of the system to be able to provide additional information or points of contact, as needed. (Remove this tip prior to submission).

The designated person(s) have sufficient knowledge of the --- Custom Document Property (SysAcronym) system to be able to provide additional information or points of contact regarding the Contingency Plan and the system, as needed. The insert position is responsible for execution of the contingency procedures documented within the --- Custom Document Property (SysAcronym) ITCP, and for ensuring the safety of personnel. If the [insert position] is unable to function as the overall authority or chooses to delegate this responsibility to a successor, the [insert position] will assume this responsibility.

Table 1: Information Contacts

Business Owner

Project Manager

System Administrator (if applicable)

Information System Security Officer (ISSO)

Designated Representative

3.4 Contingency Team Roles and Responsibilities

Tip: Describe each team, their responsibilities, leadership, and coordination with other applicable teams during a recovery operation. Insert hierarchical diagram of recovery teams. Show team names and leaders; do not include actual names of personnel. (Remove this tip prior to submission).

Names of personnel can change and for ease of document changes, will only be identified in Appendix A.

Tip: Describe each team separately, highlighting overall recovery goals and specific responsibilities. Do not detail the procedures that will be used to execute these responsibilities, as these procedures will be itemized in the appropriate phase sections. (Remove this tip prior to submission).

Example:

· Management Teams:

· Program Office/System Owner

· Information System Security Officer

· Chief Information Officer

· Individuals with Key Contingency Roles

· Chief Information Security Officer

· Office of Security and Emergency Planning

· Functional Teams:

· Data Center Server Support

· Application Support Team

· Damage Assessment Team

· User Support

· Contingency Plan Development Teams

· Procurement Team

· Recovery Team

· Contingency Coordination Team

· Disaster/Emergency Response Team

· Acceptance Team

3.5 Information System Categorization

Tip: Identify the Security Objective categorization as identified in the System Security categorization and the System Security Plan document. Enter the high water mark for Confidentiality, Integrity and Availability. The Security Category is the high water mark for the CIA categories. (Remove this tip prior to submission).

Table 2: Information System Categorization

3.6 Mission Criticality

Tip: The System’s Mission Criticality must be documented here. To determine whether the system is Mission Critical, review the System Security Categorization and the FDA System Inventory Guide process to determine if this system has been identified as Mission Critical. (Remove this tip prior to submission.)

Mission criticality is an assessment of how integral the GSS or application is to the performance of the FDA mission and must be determined using the Homeland Security Presidential Directive #7 (HSPD 7) and current Administration definitions as shown below.

The term mission critical system means any telecommunications or information system (GSS and/or MA) used or operated by FDA or by a contractor of FDA, or other organization on behalf of FDA, that:

· is defined as a national security system under section 5142 of the Clinger-Cohen Act of 1996 (40 U.S.C. 1452); or

· is protected at all times by procedures established for information which has been specifically authorized under criteria established by an Executive order or an Act of Congress to be classified in the interest of national defense or foreign policy; or

· processes any information, the loss, misuse, disclosure or unauthorized access to or modification of would have a debilitating impact on the FDA mission; or

· supports a FDA high-impact program or required to support a national security emergency event.

The FDA System Inventory Guide document provides the guidance on determining if a system is mission critical and the results documented in the Information System Inventory.

--- Custom Document Property (SysAcronym) has been determined [to be/not to be] a FDA Mission Critical system. This classification is based on the completion of the system’s Security Categorization and FDA System Inventory Process.

3.7 System Description and Architecture

Tip: Provide a general description of system architecture and functionality. Indicate the operating environment physical location general location of users and partnerships with external organizations/systems. Include information regarding any other technical considerations that are important for recovery purposes such as backup procedures (brief synopsis – System Backup Procedures in detail will be included in Appendix H). Provide a diagram of the architecture. Ensure to include identified known issues regarding the system. Replace the diagram below. (Remove this tip prior to submission.)

3.8 System Interconnection/Information Sharing

System interconnection is the direct connection of systems for the purpose of sharing information resources as defined by NIST SP800-47. System interconnection, if not appropriately protected, may result in a compromise of all connected systems and the data they store, process, or transmit. It is important that system operators, information owners, and management obtain as much information as possible about the vulnerabilities associated with system interconnection and information sharing and the increased controls required to mitigate those vulnerabilities. OMB Circular A-130 requires that written management authorization (often in the form of a Memorandum of Understanding or Agreement,) be obtained prior to connecting with other systems and/or sharing sensitive data/information. The written authorization shall detail the rules of behavior and controls that must be maintained by the interconnecting systems.

Tip: .A description of the rules for interconnecting systems and for protecting shared data must be included with this security plan. In this section, provide the following information concerning the authorization for the connection to other systems (including the internet) or the sharing of information:

List of interconnected systems (including Internet);

Unique system identifiers, if appropriate;

Name of system(s);

Organization owning the other system(s);

Users of the interconnection;

Type of interconnection (TCP/IP, Dial, SNA, etc.);

Name and title of authorizing management official(s);

Date of authorization;

System of Record, if applicable (Privacy Act data);

Sensitivity level of each system;

Security concerns and Rules of Behavior of the other systems that need to be considered in the protection of this system.

(Remove this tip prior to submission).

Table 3: Interconnections Service Agreements

3.9 Risk Assessment Results

The --- Custom Document Property (SysAcronym) Risk Assessment dated [mm/yyyy] documents the risks to system resources. These results are presented as a prioritized list of observations. Observations are stated based on the technical risk description and the Risk Severity is same as the severity rating.

Tip: Populate the following table with the risks from the Risk Assessment process. (Remove this tip prior to submission).

Table 4: Risk Assessment Results

4 Notification and Activation Phase This section addresses the initial actions taken to detect and assess damage inflicted by a disruption to --- Custom Document Property (SysAcronym). Based on the assessment of the event, the plan may be activated by the IT Contingency Plan Coordinator (CPC).

In an emergency or disaster, the FDA’s top priority is to ensure the health and safety of its staff before proceeding to the Notification and Activation procedures.

The --- Custom Document Property (SysAcronym) IT CPC will be responsible for initiating notifications using the Emergency Team List that is Error! Reference source not found.. Notifications should be made beginning at the top of the contact list and proceeding downward until positive contact is made. The --- Custom Document Property (SysAcronym) IT CPC is responsible for updating the emergency notification contact periodically or as needed as personnel assignments change.

4.1 Damage Assessment Procedures

The IT CPC, following an emergency event based on the damage assessment, will implement this plan if needed. It is imperative that the nature of the emergency and the extent of the damage be assessed as quickly as conditions allow. Details of damage assessment procedures are located in Appendix I.

Tip: The CPC is expected to conduct a damage assessment of IT components (application) and supporting components in accordance with business and operational requirements. This analysis should be detailed enough to provide adequate information for plan activation. (Remove this tip prior to submission).

If damage assessment cannot be performed locally because of unsafe conditions, the Disaster/Emergency Response Team is to follow the outline below of any additional requirements that may be necessary in addition to completing Appendix I.

· <start typing here> Upon notification from the CPC, the Disaster/Emergency Response Team Leader is to contact employees and advise of alternate plans. These procedures are to include the following:

· <start typing here>

4.2 The Contingency Plan Activation

The ITCP should be activated only when the damage assessment indicates that one or more of the activation criteria for that system are met. If an activation criterion is met, the CPC in conjunction with the System Owner should activate the plan.

Tip: Activation criteria for events are unique for each organization and should be stated in the contingency planning policy statement. (Remove this tip prior to submission).

Criteria may be based on:

· Safety of personnel an/or extent of damage to the facility;

· Extent of damage to system (e.g., physical, operational, or cost);

· Criticality of the system to the organization’s mission (e.g., critical infrastructure protection asset); and

· Anticipated duration of disruption.

The System Owner should identify the requirements for activation. However, due to the cost of activating a contingency plan, system owner should consider criteria for non-activation as exceptions.

CP Activation Criteria

· The damage assessment reveals critical components supporting the system will be unavailable for more than [24/12/6/3] hours.

· <list additional criteria here>

CP Activation Exclusions

· Activation of FDA COOP/DRP

· Agency wide outage beyond system control

· Planned outages or service disruptions

Notification and Activation Roles and Responsibilities

· When the plan is activated, the IT CPC is to notify all Team Leaders and inform them of the details of the event and if relocation is required.

· Upon notification from the IT CPC, Team Leaders are to notify their respective teams. Team members are to be informed of all applicable information and prepared to respond and relocate if necessary.

· The IT CPC is to notify the off-site storage facility that a contingency event has been declared and to ship the necessary materials (as determined by damage assessment) to the alternate site.

· The IT CPC is to notify the Alternate site, if applicable, that a contingency event has been declared and to prepare the facility for the Organization’s arrival.

4.3 Alternate Location Procedures

The ITCP Coordinator will be responsible for contacting appropriate personnel and all other appropriate agencies immediately of the agency’s alternate location, operational and communications status, and anticipated duration of relocation, if known. Alternate facility information and details are outlined in the FDA COOP.

Tip: Enter appropriate information in the box below, if applicable. Not all systems will have an alternate facility. (Remove this tip prior to submission).

Table 5: Alternate Facilities

5 Recovery Operations Phase Tip: This section focuses on procedures for recovering the application at the alternate site, whereas other efforts are directed to repair damage to the original system and capabilities. (Remove this tip prior to submission).

When a disruption occurs, despite the preventive measures implemented, a recovery strategy must be in place to recover and restore data and system operations within the recovery time objective period. The recovery strategy is designed from a combination of methods, which together address the full spectrum of information system risks.

The following procedures are for recovering the --- Custom Document Property (SysAcronym) at the alternate data center. Procedures are outlined per team required. Each procedure should be executed in the sequence it is presented to maintain efficient operations.

5.1 Recovery Procedures and Objectives

Tip: State recovery procedures and objectives as determined by the Business Impact Analysis (BIA). For each team responsible for executing a function to meet this objective, state the team names and list their respective procedures. Include the following at a minimum. (Remove this tip prior to submission).

What personnel have the access privileges to the offsite storage facility?

Who is the offsite Point of Contact (POC)?

What type of restoration can be performed from the alternate site - full, partial?

As a part of restoration/recovery, do transaction logs automatically update master files?

How do non-captured transactions get re-entered?

Depending on the nature of the system outage, a combination of recovery procedures should be implemented by the Recovery Team. Recovery procedures listed in this section provide the CPC with high-level guidance regarding the types of activities to be performed during contingency operations. Additionally, Error! Reference source not found. contains an action item checklist to assist the CPC track these operations.

While monitoring recovery operations, the CPC should notify the Contingency Plan Director (CPD) to escalate the situation to higher management if it is likely that the system will not be recovered within 48 hours.

If the building that houses the failing system component(s) is damaged or has been evacuated, the CPC contacts the datacenter management for assistance in obtaining an estimated time that recovery personnel may safely reenter the building to begin supporting the CP operations.

The CPC should coordinate with the datacenter management to obtain information on the physical environment of the computer room that houses the application and get an estimated time that the building will be cleared for reentry by support personnel.

The CPC should provide periodic updates regarding the reopening of the datacenter and the restoration of facilities services.

The FDA Recovery Teams are responsible for the recovery of the IT Infrastructure (e.g., LAN/wide area network (WAN), server, database) and the reinstallation of the application code.

Support personnel should refer to their standard operating procedures for recovery of all IT components. This includes reinstalling all hardware and software (operating system, etc.) that serves as the platform of the application. ISG personnel should reference the application’s maintenance manual for step-by-step procedures and supporting documentation. Support personnel should also contact all vendors to provide additional support as needed.

The Application Recovery Team works in conjunction with the FDA Recovery Teams personnel to ensure a seamless restoration of the application’s software and data. All recovery team personnel should reference the application’s Code Deployment Procedures in the PVCS for additional information.

6 Reconstitution Phase

This section discusses activities necessary for restoring --- Custom Document Property (SysAcronym) operations at the system’s original or new site. When the computer center at the original or new site has been restored, --- Custom Document Property (SysAcronym) operations at the alternate site must be transitioned back.

Once the original site is restored or new site is operational to the level that it can support the IT system processes, the system may be transitioned back to the respective data center. Until the primary location is restored and operational and the system has been tested at the original or new site and deemed operational, the contingency data center should be maintained and utilized. The goal is to provide a seamless transition of operations from the contingency (alternate) site to the original or new data center.

6.1 Concurrent Processing

Tip: Outline in this section, the procedures, to operate the system back at the original or at the new site. These procedures should include testing the original or new system until it is functioning properly and the contingency system is shut down gracefully. (Remove this tip prior to submission).

<enter Team name>

· < enter Team Resumption Procedures <enter Team name>

· < enter Team Resumption Procedures

6.2 Plan Deactivation

Tip: Procedures must be outlined, per team, to clean the alternate site of any equipment or other materials belonging to the organization, with a focus on handling sensitive information. Materials, equipment, and backup media must be properly packaged, labeled, and shipped to the appropriate location(s) according to the FDA Security Policies. Team members must be instructed to return to the original or new site. (Remove this tip prior to submission).

<start typing here> 7 Training, Testing and Exercises

To ensure an effective and viable --- Custom Document Property (SysAcronym) CP capability, periodic training, testing, and exercises should be conducted on the --- Custom Document Property (SysAcronym) ITCP. This process will range from exercising parts of the plan (e.g., Notification phase) to full-scale tests of the entire [System Name] recovery and resumption process. This section presents procedures and standards for developing a training and testing program. It also discusses methods for maintaining, updating, and distributing the ITCP to all responsible personnel to ensure an active state of readiness.

A successful execution of a Contingency Plan depends on the ability of --- Custom Document Property (SysAcronym) personnel to perform their responsibilities efficiently and correctly. Training provides an effective means of enhancing individuals’ familiarity with the ITCP and increases their ability to implement the plan properly. --- Custom Document Property (SysAcronym) personnel must receive training on the ITCP and participate in hands-on contingency planning exercises. New personnel with ITCP responsibilities will receive a copy of the current ITCP and an orientation on its use. Personnel will periodically review the ITCP to refresh their knowledge of the procedures. Thus, when responding to a disaster or contingency, they can respond accordingly.

7.1 IT Testing and Exercise

Since the ITCP is a living document, regular testing and evaluation are an essential part of plan maintenance. Testing familiarizes personnel with the contingency planning process by simulating notification/activation, recovery, and resumption processes, and identifies key areas in which Recovery Team members may require additional training or orientation concerning the --- Custom Document Property (SysAcronym) ITCP. ITCP testing should be coordinated by the CPC and the Information System Security Officer (ISSO) to ensure that all major elements and components of --- Custom Document Property (SysAcronym) are tested. A testing schedule has been developed that includes the time frames and participants for the main testing steps: developing the test plan, executing the test plan, and documenting the test plan results and recorded in Appendix K and Appendix L. Testing must be performed at least annually to ensure that the ITCP could be relied on in the event of a system failure.

7.2 Documenting Test Plan Results

The test plans and the results of the tests are documented and maintained as part of the overall ITCP documentation. The latest test results are documented in Appendix L. Documentation helps in evaluating liabilities and identifying achievements. Once the test plans and their results have been documented, the CPC should forward the documentation to the ISSO and System Owner for review. The ISSO and System Owner must sign off on the documents to indicate that the plan was successfully tested. This signed documentation must then be provided to the CIO’s office for FISMA recording and tracking. This documentation will serve as a record of “observations” and will assist in determining which procedures worked well and which need to be modified.

7.3 Plan Maintenance

As a living document, the ITCP must be maintained and updated over time. The --- Custom Document Property (SysAcronym) System Owner is responsible for maintaining, testing, and updating the plan. However, the System Owner may delegate all or parts of this responsibility to the CPC, ISSO, or others, as appropriate. The System Owner, or designated delegate, should coordinate with key individuals in planning testing and training schedules and objectives, analyzing testing and training documentation (e.g., observations and results), and monitoring major changes to [System Name]. To ensure that the plan maintains a current state of readiness, it is important that the System Owner, or designated delegate, keep the following maintenance schedule:

The CP should be review and updated at least annually based on significant changes to the --- Custom Document Property (SysAcronym) application architecture.

The contact sheets and vendor information should be reviewed and updated at least semi-annually based on significant changes to --- Custom Document Property (SysAcronym) personnel, vendors, and supporting contractors

The CP should be updated whenever test results indicate procedural changes.

Table 6: System Plan Maintenance

· Incorporate ‘lessons learned’ and change in policy and philosophy

· Manage distribution of plan updates

· Annual (at least) general update

· Semiannual updates of personnel contact information

· With significant changes to components, configurations, organization structure, or controls

· Following tests and exercises

· Schedule participation in all training and exercise events

· Within 3 months of hiring personnel with ITCP responsibilities

· Brief all key personnel on their responsibilities under the contingency plan

· Annually

· When redistributing the plan

· When adding new key personnel

· Review and update Activation procedures

· Review and update Recovery procedures

· Review and update Resumption procedures

· Annually, or as needed

· Conduct walkthrough exercises

· Conduct joint exercises with FDA personnel and supporting contractors

· Document and review lessons learned

· Annually (at least), or as needed

The System Owner, or designated delegate, is responsible for distributing the ITCP. Two copies of the ITCP should be distributed to those responsible for coordinating the plan and those who participate in recovery and resumption operations. Each person is required to maintain one copy of the plan holder’s office, and the other copy should be kept at the plan holder’s home for after-hours emergency situations. The original document is to be stored at a secure location that is easily accessible by both the System Owner and the CPC. If there is a change to the plan, the entire obsolete ITCP or obsolete page(s) must be retrieved or confirmed to have been destroyed by all personnel before the new or updated plan or pages are distributed. This practice will ensure that all personnel maintain the most current ITCP and that contingency operations can be completed efficiently. All key personnel are required to fully examine the ITCP to better understand their roles and responsibilities.

Appendix A Contingency Plan Organization Chart

When notified by the CPC that the ITCP has been activated, the Coordinator will ensure the appropriate Recovery Teams are notified and Recovery Team Leaders apprised of the situation (not all teams may be required for all events).

Appendix B Emergency/Disaster Team Formulation

Tip: Fill in the following tables as appropriate. Develop a flow diagram to show how these teams will interact. (Remove this tip prior to submission).

Contingency Plan Work Group

The ITCP Work Group supports the ITCP Coordinator in the development of strategies to restore critical business function in the event of an emergency or disaster. The work group completes all preliminary activities, including selecting team members and reaching agreements on offsite storage and alternate facilities.

Table 7: Contingency Plan Work Group Membership

IT Contingency Plan Coordinator

Alternate Cont. Plan Coordinator

Team Member

Team Member

Procurement Team The Procurement Team acquires any required hardware/software and related documentation and/or other items that must be placed at the alternate facility.

Table 8: Procurement Team Membership

Team Lead

Alternate Team Lead

Team Member

Team Member

Restoration Management Team

The Restoration Management Team is responsible for determining the best course of action for restoring critical business functions in the event of an emergency or disaster.

Table 9: Restoration Management Team Membership

Restoration Manager

Alternate Restoration Manager

Team Member

Team Member

Contingency Coordination Team

The IT Contingency Coordinator is responsible for informing FDA management about the status of efforts to restore critical business functions in the event of an emergency or disaster and ensures the coordination of restoration activities at the alternate facility. The IT Contingency Coordination Team conveys management instructions and disseminates status information to leaders of other teams.

Table 10: Contingency Coordination Team Membership

IT Contingency Coordinator

Alternate IT Contingency Coordinator

Team Member

Team Member

Disaster/Emergency Response Team The Disaster/Emergency Response Team assesses the functionality of systems, conducts and initial damage assessment, and provides ongoing status reports in the event of an emergency or disaster. After the Disaster/Emergency Response Team assesses an incident, members may disperse to lead or participate in other teams.

Table 11: Disaster/Emergency Response Team Membership

Team Lead

Alternate Team Lead

Infrastructure Representative

Network Operations Representative

Server Administration Representative

Team Member

Team Member

Recovery/Restoration Team

The Recovery/Restoration Team has overall responsibility for restoring a system, either partially or fully, as instructed by the Restoration Management Team.

Table 12: Contingency Coordination Team Membership

Recovery/Restoration Coordinator

Alternate IT Recovery/Restoration Coordinator

Team Member

Team Member

Recovery Process for the Recovery Team

These procedures are used for recovering a file from backup tapes. The Recovery Team is responsible for reloading all critical files necessary to continue production.

Table 13: Recovery Process

Identify file and date from which file is to be recovered

Identify tape number using tape log book

If tape is not in tape library, request tape from recovery facility; fill out with appropriate authorizing signature

When tape is received, log date and time

Place tape into drive and begin recovery process

When file is recovered, notify Recovery Team Leader

Acceptance Team The Acceptance Team is responsible for conducting functional testing on and for accepting or rejecting a restored system after an emergency or disaster. A case for accepting a system with limited capabilities must be justified to management, which is responsible for declaring the system operational.

Table 14: Acceptance Team Membership

Acceptance Coordinator

Alternate Acceptance Coordinator

Team Member

Team Member

Appendix C Additional Notification Contact List Table 15: Additional Notification Contact List Primary

Alternates

Appendix D Vendor Contact List Table 16: Vendor Contact List

Appendix E Equipment and Specifications Table 17: System Hardware

Table 18: System/Application Software

Appendix F Business/Mission Impact Analysis Table 19: Business/Mission Impact Analysis Summary

--- Custom Document Property (SysName) (--- Custom Document Property (SysAcronym))

Tip: Fill out the tables using the following criteria:

Recovery Priority – List the priority associated with recovering a specific resource, based on the outage impacts and allowable outage times.

Critical Resource/Function – Identify the specific hardware, software, and other resources (include system dependencies and interdependencies) that comprise the system; include quantity and type

Critical Role and Department – Identify the individuals, positions, offices, agencies or organizations that depend on or support the system; also specify their relationship to the system

Maximum Allowable Outage – Identify the maximum acceptable period that the resource could be unavailable before unacceptable impacts resulted

Business Impact – Impact (High, Moderate, Low); Comments (Describe the business impact on mission if the critical resource is unavailable.)

(Remove this tip prior to submission).

Table 20: Business/Mission Impact Analysis Detail

Appendix G IT Standard Operation Procedures Tip: Fill in the tables or provide any SOP’s that may currently exist for the system. (Remove this tip prior to submission).

Table 21: Recovery and Restoration Team Procedures

Receive notice that the operating environment has been recovered and is ready for system to be recovered.

Test IT equipment to see if it is operating correctly

Request offsite backup

Receive offsite backup

Verify applications software version

Load the most recent version of the software

Validate that the system was successfully recovered

Obtain the most recent backup of the system specific data

Load the system specific data

Validate that the system specific data has been recovered (verify data counts, verify date information)

Validate security controls

Notify Acceptance Team

Table 22: Acceptance Team Procedures

Validate system specific security.

Validate all functions are operational

Read system specific data

Read interconnected system data

System update function for (XX)

System update function online

System update function for (XX)

System update function online

Verify all updates are successful

Notify user community

Prepare report

Appendix H System Backup Procedures

Tip: Describe additional application/system backup procedures. If additional application code/programming or unique conditions require additional archive procedures, document the process below. If not additional requirements are necessary leave this section blank.

Include Backup media and frequency (systems software, application software); Type of backup (weekly, daily - total, incremental); On-site location of backup and protective measures; Off-site location of backup and frequency; Backup logs, audit trails, etc.; Password accessibility.

Include the restoration methods used and needed to be available for recovery.