EPLC_Contingency and Disaster Recovery Plan_v . .doc

Food and Drug Administration <Project Name & Release Version #>

ENTERPRISE PERFORMANCE LIFECYCLE

Contingency and Disaster Recovery Plan Document Version Number: <#.#> Document Version Date: <mm/dd/yyyy>

TABLE OF CONTENTS

Introduction

Purpose

32.1 Applicability

32.2 Planning Principles

42.3 Assumptions

52.4 References/Requirements

System Identification

73.1 System Name/Title

73.2 Responsible Organization

83.3 Information Contact(s) and Line of Succession

83.4 [System Name] Contingency Team Roles and Responsibilities

93.5 Information System Categorization

103.6 Mission Criticality

113.7 System Description and Architecture

113.8 System Interconnection/Information Sharing

133.9 Risk Assessment Results

NOTIFICATION AND ACTIVATION PHASE

144.1 Damage Assessment Procedures

144.2 The Contingency Plan Activation

154.3 Alternate Location Procedures

RECOVERY OPERATIONS PHASE

165.1 Recovery Procedures and Objectives

RECONSTITUTION PHASE

186.1 Concurrent Processing

186.2 Plan Deactivation

TRAINING, TESTING AND EXERCISES

197.1 IT Testing and Exercise

197.2 Documenting Test Plan Results

207.3 Plan Maintenance

Appendices

IT CONTINGENCY PLAN REVIEW/APPROVAL SHEET

We have reviewed the [System Name]’s Contingency Plan (CP) and accept the information and conclusions contained within.

[First Name Last Name]

Date

System Owner

[First Name Last Name]

Date

[System Name] Information System Security Officer

[System Name] CHANGE INFORMATION PAGE

INSTRUCTIONS ON COMPLETING THIS TEMPLATE

Instructions for each section are included in each Discussion Box on how to complete and fulfill the requirements for that section. Discussion Boxes can be deleted after the section has been completed or may be left there for future reference.

· Texts in Blue are either instruction texts or text that needs to be deleted and filled in by the writer of the plan.

· Texts in Black are standard template text that are required to be included in the Contingency Plan and should not be deleted unless necessary.

· Texts in Red are examples given within each section. It is advised that the examples are deleted after the section has been completed to prevent any reader confusion.

1 Introduction This document contains the [System Name] Information Technology (IT) Contingency Plan (CP). It is intended to serve as the centralized repository for the information, tasks, and procedures that would be necessary to facilitate the [System Name] management’s decision-making process and its timely response to any disruptive or extended interruption of the FDA's normal business operations and services. This is especially important if the cause of the interruption is such that a prompt resumption of operations cannot be accomplished by employing only normal daily operating procedures.

In terms of personnel and financial resources, the information tasks and procedures detailed in this plan represent the [System Name] management’s demonstrated commitment to response, resumption, recovery, and restoration planning. Therefore, it is essential that the information and action plans in this plan remain viable and be maintained in a state of currency in order to ensure the accuracy of its contents. To that end, this introduction is intended to introduce and familiarize its readers with the organization of the plan.

Since the information contained in this document describes [System Name] management’s planning assumptions and objectives, the plan should be considered a sensitive document. All of the information and material contents of this document should be labeled, “For Official Use Only”.

The [System Name] management has recognized the potential financial and operational losses associated with service interruptions and the importance of maintaining viable emergency response, resumption, recovery and restoration strategies. This ITCP is intended to provide a framework for constructing plans to ensure the safety of employees and the resumption of time-sensitive operations and services in the event of an emergency and/or disaster (fire, power or communications blackout, tornado, hurricane, flood, earthquake, civil disturbance, etc.).

Although the [System Name] ITCP provides guidance and documentation upon which to base emergency response, resumption, and recovery planning efforts, it is not intended as a substitute for informed decision-making. Business process managers and executives must identify services for which disruption will result in significant financial and/or operational losses.

Discussion: Plans should include detailed responsibilities and specific tasks for emergency response activities and business resumption operations based upon pre-defined time frames. Constructing a plan and presenting it to senior management may satisfy the immediate need of having a documented plan. However, this is not enough if the goal is to have a viable response, resumption, recovery, and restoration capability.

An ITCP is not a one-time commitment and is not a project with an established start and end date. Instead, ITCPs are an on-going, funded business activity budgeted to provide resources required to:

· Perform activities required to update and create plans

· Train employees on a continuous basis.

· Develop and revise policies and standards as the department changes

· Exercise strategies, procedures, team and resources requirements

· Re-exercise unattained exercise objectives

· Report on-going continuity planning to senior management

· Research processes and technologies to improve resumption and recovery efficiency

· Perform plan maintenance activities

The ITCP elements necessary to create a viable, repeatable and verifiable continuity capability include:

· Implement accurate and continuous vital records, data backup, and off-site storage

· Implement capability for rapid switching of voice and data communication circuits to alternate site(s)

· Provide a alternate site for operations

· Construct contingency response teams

· Implement contingency strategies

2 Purpose Discussion: If this is the first time the ITCP has been written, the Contingency Planning Officer should consult with system owners, developers and/or staff concerning information on current Contingency / Disaster Recovery Planning (DRP). Indicate here if this is the first time the ITCP has been written. If testing has been conducted in the past, input those results here.

This ITCP establishes procedures to recover and reconstitute the [System Name] following a disruption. The following objectives have been established for this plan:

1) Maximize the effectiveness of contingency operations through an established plan that consists of the following phases:

· Notification/Activation phase

· Recovery phase

· Reconstitution phase

2) Identify the activities, resources, and procedures needed to carry out [System Name] processing requirements during prolonged interruptions to normal operations.

3) Assign responsibilities to designated personnel and provide guidance for recovering [System Name] during prolonged periods of interruption to normal operations.

4) Ensure coordination with other staff that will participate in the contingency planning strategies. Ensure coordination with external points of contact and vendors who will participate in the contingency planning strategies.

2.1 Applicability

The [System Name] ITCP applies to the functions, operations, and resources necessary to restore and resume [System Name] operations at [Primary Location Name], [City], [State]. The [System Name] ITCP applies to all personnel associated with [System Acronmy] as identified under Section 2.3, Responsibilities.

The [System Name] ITCP is supported by [Plan Names, i.e., FDA COOP, System DRP, Configuration Documentation, Design Documentation], which provides the [Purpose of the plans]. Procedures outline in this plan are coordinated with and support the aforementioned documents.

2.2 Planning Principles

This plan does not identify contingencies for every possible scenario. Rather, it focuses on two scenario types: (1) Minor System Failure and (2) Major System Failure. Full ITCP implementation may not be required for all disruptions. In some cases, partial implementation may be appropriate. The [System Name] ITCP is based on the following scenarios: ITCP will not be activated for a Minor System Failure; however, should be considered for full implementation for most Major System Failures. The scenarios are described in more detail below.

Scenario 1 – Minor System Failure

A localized threat has caused a disruption of [System Name] services in one or both of the following:

1) The application has become inoperable for a period of less than [24/12/6/3] hours and major components of the system are still maintained and operational.

2) The facility that houses the failed component(s) has not been physically damaged to an extent that requires personnel evacuation or major repair to the environment.

Scenario 2 – Major System Failure

A localized threat has caused a disruption of [System Name] services in one or both of the following:

1) The application has become inoperable for a time period (present or expected) to exceed more than [24/12/6/3] hours.

2) The facility that houses the failed component(s) has been physically damaged to an extent that requires temporary personnel evacuation or major repair to the environment.

2.3 Assumptions

Based on these principles, the following assumptions were used when developing the ITCP:

· The GSS Local Area Network/Wide Area Network (LAN/WAN) is operational.

· Key [System Name] personnel have been identified and trained in their emergency response and recovery roles and are available to activate the ITCP.

· Access to backup data, configuration files, hardware, and software is accessible by key ITCP personnel.

· Service level agreements (SLA) are maintained with system hardware and software vendors to support emergency system recovery.

· The [System Name] ITCP is up to date and accessible to key personnel responsible for executing the plan.

· The procedures within the plan have been tested to ensure that the strategies are viable.

Discussion: Add additional assumptions as necessary.

The [System Name] ITCP does not apply to the following situations:

· Overall recovery and continuity of business operations

· Emergency evacuation of personnel

· Detailed procedures for the relocation of IT functions to an alternate site if the original site is damaged.

Discussion: Add additional situations as necessary.

2.4 References/Requirements

ITCP’s are constructed using five components. These are depicted in Figure 1-1.

Figure 1-1, IT Contingency Plan Development Strategy

[System Name] ITCP complies with the FDA Policies as follows:

· System Owners will develop CPs for information systems under their purview in accordance with National Institute of Standards and Technology (NIST) Special Publication (SP) 800-34.

· System Owners will review the [System Name] ITCP once a year, update them as necessary, and communicate any changes to the Program Office responsible for COOP, if applicable.

· System Owners will ensure that plans for systems rated Low, Moderate, or High are tested/exercised at least annually. Testing will be coordinated with elements responsible for COOP and incident response. For systems rated Moderate and High, the level of rigor for testing must increase every year. For systems rated High, the System Owners will ensure testing at the alternate processing site.

The [System Name] ITCP also complies with the following federal and departmental policies: Add in additional departmental policies if needed.

· The Computer Security Act of 1987.

· Office of Management and Budget (OMB) Circular A-130, Management of Federal Information Resources, Appendix III, November 2000.

· Federal Information Processing Standards Publication (FIPS PUB) 87, Guidelines for ADP Contingency Planning, March 1981 (superseded by this publication).

· Federal Preparedness Circular (FPC) 65, Federal Emergency Management Agency (FEMA) June 15, 2004.

· Homeland Security Presidential Directive (HSPD) 8, National Preparedness, December 17, 2003.

· National Institute of Standards and Technology (NIST) Special Publication (SP) 800-34, Contingency Planning Guide for Information Technology Systems, June 2002.

· NIST SP 800-100, Information Security Handbook: A Guide for Managers, October 2006

3 System Identification

3.1 System Name/Title

Discussion: Enter the system name and acronym given to the general support system or application. Also identify the General Support System that this system relies on.

3.2 Responsible Organization

Discussion: In this section, list the organization that owns and is responsible for the data in the application. Identify the responsible organization for the system, the data it contains, and who controls the use of the data. List the federal organizational sub-component responsible for the system. If a state or local government or contractor performs the function, identify both the federal and other organization and describe the relationship. Be specific about the organization and do not abbreviate. Include phone numbers, physical locations, and addresses.

The responsible organization for the system, the data it contains, and who controls the use of the data.

Example:

System Owner:

Center for Devices and Radiological Health (CDRH)

Office of the Director John Smith U.S. Food and Drug Administration

202-555-1234 x1234

555 99th Street S.W., Washington, DC 20410

The System is maintained by:

Appropriate Contractor Firm

123-456-7890 x0000

1234 Main St

Anywhere, USA, 12345

3.3 Information Contact(s) and Line of Succession

Discussion: Specify the program owner, program manager and the system manager to contact for further information regarding the contingency plan and the system. Include their address, telephone number(s), and e-mail. List the name, title, organization, and telephone number of one or more persons designated to be the point(s) of contact for this system. The contacts given should be identified as the system owner, program manager, system manager, and Information System Security Officer (ISSO). The designated persons should have sufficient knowledge of the system to be able to provide additional information or points of contact, as needed.

The designated person(s) have sufficient knowledge of the system to be able to provide additional information or points of contact regarding the system, as needed. The [System Owner]/ [insert position], [Center Name] is responsible for execution of the contingency procedures documented within the [System Name] ITCP, and for ensuring the safety of personnel. If the [insert name or position] is unable to function as the overall authority or chooses to delegate this responsibility to a successor, the [insert name or position] will assume this responsibility.

Program Manager

System Manager

Information System Security Officer (ISSO)

Designated Representative

3.4 [System Name] Contingency Team Roles and Responsibilities Discussion: Describe each team, their responsibilities, leadership, and coordination with other applicable teams during a recovery operation. Insert hierarchical diagram of recovery teams. Show team names and leaders; do not include actual names of personnel. Names of personnel can change and for ease of document changes, will only be identified in Appendix A.

Describe each team separately, highlighting overall recovery goals and specific responsibilities. Do not detail the procedures that will be used to execute these responsibilities, as these procedures will be itemized in the appropriate phase sections.

Example:

Management Teams:

Program Office/System Owner

Information System Security Officer

Chief Information Officer

Individuals with Key Contingency Roles

Chief Information Security Officer

Office of Security and Emergency Planning

Functional Teams:

Data Center Server Support

Application Support Team

Damage Assessment Team

User Support

Other teams include:

Contingency Plan Development Teams

Procurement Team

Recovery Team

Contingency Coordination Team

Disaster/Emergency Response Team

Acceptance Team

3.5 Information System Categorization

Discussion: Identify the Security Objective categorization as identified in the System Security categorization and the System Security Plan document. Enter the high water mark for Confidentiality, Integrity and Availability. The Security Category is the high water mark for the CIA categories.

(Low-Mod-High)

Confidentiality

Integrity

Availability

3.6 Mission Criticality

Discussion: The System’s Mission Criticality must be documented here. To determine whether the system is [System Name] Mission Critical, review the System Security Categorization and the FDA System Inventory Guide process to determine if this system has been identified as Mission Critical.

Mission criticality is an assessment of how integral the GSS or application is to the performance of the FDA mission, and must also be considered in the inventory process. Using the Homeland Security Presidential Directive #7 (HSPD 7) and current Administration definitions as shown below.

The term mission critical system means any telecommunications or information system (GSS and/or MA) used or operated by FDA or by a contractor of FDA, or other organization on behalf of FDA, that:

· is defined as a national security system under section 5142 of the Clinger-Cohen Act of 1996 (40 U.S.C. 1452); or

· is protected at all times by procedures established for information which has been specifically authorized under criteria established by an Executive order or an Act of Congress to be classified in the interest of national defense or foreign policy; or

· processes any information, the loss, misuse, disclosure or unauthorized access to or modification of would have a debilitating impact on the FDA mission.

· supports a FDA high-impact program or required to support a national security emergency event The FDA System Inventory Guide document provides the guidance on determining if a system is mission critical and the results documented in the Information System Inventory.

[System Name] has been determined [to be/not to be] a FDA Mission Critical system. This classification is based on the completion of the [System Name] Security Categorization and FDA System Inventory Process.

3.7 System Description and Architecture

Discussion: Provide a general description of system architecture and functionality. Indicate the operating environment physical location general location of users and partnerships with external organizations/systems. Include information regarding any other technical considerations that are important for recovery purposes such as backup procedures (brief synopsis – System Backup Procedures in detail will be included in Appendix M). Provide a diagram of the architecture. Ensure to include identified known issues regarding the system.

[Insert Diagram Here]

Figure 3-1: Accreditation boundary diagram

3.8 System Interconnection/Information Sharing

Discussion: System interconnection is the direct connection of systems for the purpose of sharing information resources. System interconnection, if not appropriately protected, may result in a compromise of all connected systems and the data they store, process, or transmit. It is important that system operators, information owners, and management obtain as much information as possible about the vulnerabilities associated with system interconnection and information sharing and the increased controls required mitigating those vulnerabilities.

OMB Circular A-130 requires that written management authorization (often in the form of a Memorandum of Understanding or Agreement,) be obtained prior to connecting with other systems and/or sharing sensitive data/information. The written authorization shall detail the rules of behavior and controls that must be maintained by the interconnecting systems. System owners are expected to document all interconnections, data dependencies and interdependencies in all system documentation. A description of the rules for interconnecting systems and for protecting shared data must be included with this plan.

In this section, provide the following information concerning the authorization for the connection to other systems (including the internet) or the sharing of information. List of interconnected systems and data dependencies and interdependencies:

· Unique system identifiers, if appropriate;

· Name of system(s);

· Organization owning the other system(s);

· Type of interconnection (TCP/IP, Dial, SNA, etc.);

· Name and title of authorizing management official(s);

· Date of authorization;

· System of Record, if applicable (Privacy Act data);

· Sensitivity level of each system

System Name

(ISA/MOU/MOA)

Status Auth.

Official

Table 3-2: MOU/ISA/MOA Identification

3.9 Risk Assessment Results

[System Name] Risk Assessment dated [mm/yyyy] document the risks to system resources. These results are presented as a prioritized list of observations. Observations are stated based on the technical risk description and the Risk Severity is same as the severity rating.

Discussion: Populate the following table with the risks from the Risk Assessment process.

(Low-Mod-High)

Table 3-3: Risk Assessment Results

4 NOTIFICATION AND ACTIVATION PHASE

This section addresses the initial actions taken to detect and assess damage inflicted by a disruption to [System Name]. Based on the assessment of the event, the plan may be activated by the IT Contingency Plan Coordinator (CPC).

In an emergency / disaster, the FDA’s top priority is to ensure the health and safety of its staff before proceeding to the Notification and Activation procedures.

The [System Name] IT CPC will be responsible for initiating notifications using the Emergency Team List that is Appendix A. When making notifications begin at the top of the contact list and proceed downward until positive contact is made. The [System Name]’s IT CPC is responsible for updating the emergency notification contact periodically or as needed as personnel assignments change.

4.1 Damage Assessment Procedures

The IT CPC, following an emergency event based on the damage assessment, will implement this plan if needed. It is imperative that the nature of the emergency and the extent of the damage be assessed as quickly as conditions allow. Details of damage assessment procedures are located in Appendix I.

Discussion: CPC is expected to conduct a damage assessment of IT components (application) and supporting components in accordance with business and operational requirements. This analysis should be detailed enough to provide adequate information for plan activation.

If damage assessment cannot be performed locally because of unsafe conditions, the Disaster/Emergency Response Team is to follow the outline below of any additional requirements that may be necessary in addition to completing Appendix I. Additional procedures for this system should be identified in this section.

Damage Assessment Procedures:

Detailed procedures should be outlined and completed refer to Appendix I.

Upon notification from the CPC, the Disaster/Emergency Response Team Leader is to contact employees and advise of alternate plans. These procedures are to include the following….

4.2 The Contingency Plan Activation

Discussion: This section will document the criteria use for activating the CP. The ITCP should be activated only when the damage assessment indicates that one or more of the activation criteria for that system are met. If an activation criterion is met, the CPC in conjunction with the System Owner should activate the plan. Activation criteria for events are unique for each organization and should be stated in the contingency planning policy statement.

Criteria may be based on:

· Safety of personnel an/or extent of damage to the facility

· Extent of damage to system (e.g., physical, operational, or cost)

· Criticality of the system to the organization’s mission (e.g., critical infrastructure protection asset)

· Anticipated duration of disruption The System Owner should identify the requirements for activation. However, due to the cost of activating a contingency plan, system owner should consider criteria for non-activation as exceptions.

[System Name] CP Activation Criteria

· The damage assessment reveals critical components supporting the system will be unavailable for more than [24/12/6/3] hours.

· System owner should document other criteria as they apply to the system.

[System Name] Plan Activation Exclusions

· Activation of FDA COOP/DRP

· Agency wide outage beyond system control

· Planned outages or service disruptions

· System owner should document other exclusions as they apply to the system.

Notification and Activation Roles and Responsibilities

· When the plan is activated, the IT CPC is to notify all Team Leaders and inform them of the details of the event and if relocation is required.

· Upon notification from the IT CPC, Team Leaders are to notify their respective teams. Team members are to be informed of all applicable information and prepared to respond and relocate if necessary.

· The IT CPC is to notify the off-site storage facility that a contingency event has been declared and to ship the necessary materials (as determined by damage assessment) to the alternate site.

· The IT CPC is to notify the Alternate site, if applicable, that a contingency event has been declared and to prepare the facility for the Organization’s arrival.

· The IT CPC is to notify remaining personnel (via notification procedures) on the general status of the incident.

4.3 Alternate Location Procedures

The ITCP Coordinator will be responsible for contacting appropriate personnel and all other appropriate agencies immediately of the agency’s alternate location, operational and communications status, and anticipated duration of relocation, if known. Alternate facility information and details are outlined in the FDA COOP.

Discussion: Enter appropriate information in the box below, if applicable. Not all systems will have an alternate facility.

Example:

5 RECOVERY OPERATIONS PHASE

Discussion: This section focuses on procedures for recovering the application at the alternate site, whereas other efforts are directed to repair damage to the original system and capabilities.

When a disruption occurs, despite the preventive measures implemented, a recovery strategy must be in place to recover and restore data and system operations within the recovery time objective period. The recovery strategy is designed from a combination of methods, which together address the full spectrum of information system risks.

The following procedures are for recovering the [System Name] at the alternate data center. Procedures are outlined per team required. Each procedure should be executed in the sequence it is presented to maintain efficient operations.

5.1 Recovery Procedures and Objectives

Discussion: State recovery procedures and objectives as determined by the Business Impact Analysis (BIA). For each team responsible for executing a function to meet this objective, state the team names and list their respective procedures. Include the following at a minimum:

What personnel have the access privileges to the offsite storage facility?

Who is the offsite Point of Contact (POC)?

What type of restoration can be performed from the alternate site - full, partial?

As a part of restoration/recovery, do transaction logs automatically update master files?

How do non-captured transactions get re-entered?

Example:

Depending on the nature of the system outage, a combination of recovery procedures should be implemented by the Recovery Team. Recovery procedures listed in this section provide the CPC with high-level guidance regarding the types of activities to be performed during contingency operations. Additionally, Appendix J contains an action item checklist to assist the CPC track these operations.

While monitoring recovery operations, the CPC should notify the Contingency Plan Director (CPD) to escalate the situation to higher management if it is likely that the system will not be recovered within 48 hours.

If the building that houses the failing system component(s) is damaged or has been evacuated, the CPC contacts the datacenter management for assistance in obtaining an estimated time that recovery personnel may safely reenter the building to begin supporting the CP operations.

The CPC should coordinate with the datacenter management to obtain information on the physical environment of the computer room that houses the application and get an estimated time that the building will be cleared for reentry by support personnel.

The CPC should provide periodic updates regarding the reopening of the datacenter and the restoration of facilities services.

The FDA Recovery Teams are responsible for the recovery of the IT Infrastructure (e.g., LAN/wide area network (WAN), server, database) and the reinstallation of the application code.

Support personnel should refer to their standard operating procedures for recovery of all IT components. This includes reinstalling all hardware and software (operating system, etc.) that serves as the platform of the application. ISG personnel should reference the application’s maintenance manual for step-by-step procedures and supporting documentation. Support personnel should also contact all vendors to provide additional support as needed.

The Application Recovery Team works in conjunction with the FDA Recovery Teams personnel to ensure a seamless restoration of the application’s software and data. All recovery team personnel should reference the application’s Code Deployment Procedures in the PVCS for additional information

6 RECONSTITUTION PHASE

This section discusses activities necessary for restoring [System Name] operations at the system’s original or new site. When the computer center at the original or new site has been restored, [System Name] operations at the alternate site must be transitioned back.

Once the original site is restored or new site is operational to the level that it can support the IT system processes, the system may be transitioned back to the respective data center. Until the primary location is restored and operational and the system has been tested at the original or new site and deemed operational, the contingency data center should be maintained and utilized. The goal is to provide a seamless transition of operations from the contingency (alternate) site to the original or new data center.

6.1 Concurrent Processing

Discussion: Outline in this section, the procedures, to operate the system back at the original or at the new site. These procedures should include testing the original or new system until it is functioning properly and the contingency system is shut down gracefully.

{Team Name}

− Team Resumption Procedures {Team Name}

− Team Resumption Procedures

6.2 Plan Deactivation

Discussion: Procedures must be outlined, per team, to clean the alternate site of any equipment or other materials belonging to the organization, with a focus on handling sensitive information. Materials, equipment, and backup media must be properly packaged, labeled, and shipped to the appropriate location(s) according to the FDA Security Policies. Team members must be instructed to return to the original or new site.

7 TRAINING, TESTING AND EXERCISES

To ensure an effective and viable [System Name] CP capability, periodic training, testing, and exercises should be conducted on the [System Name] ITCP. This process will range from exercising parts of the plan (e.g., Notification phase) to full-scale tests of the entire [System Name] recovery and resumption process. This section presents procedures and standards for developing a training and testing program. It also discusses methods for maintaining, updating, and distributing the ITCP to all responsible personnel to ensure an active state of readiness.

A successful execution of a Contingency Plan depends on the ability of [System Name] personnel to perform their responsibilities efficiently and correctly. Training provides an effective means of enhancing individuals’ familiarity with the ITCP and increases their ability to implement the plan properly. [System Name] personnel must receive training on the ITCP and participate in hands-on contingency planning exercises. New personnel with ITCP responsibilities will receive a copy of the current ITCP and an orientation on its use. Personnel will periodically review the ITCP to refresh their knowledge of the procedures. Thus, when responding to a disaster or contingency, they can respond accordingly.

7.1 IT Testing and Exercise

Since the ITCP is a living document, regular testing and evaluation are an essential part of plan maintenance. Testing familiarizes personnel with the contingency planning process by simulating notification/activation, recovery, and resumption processes, and identifies key areas in which Recovery Team members may require additional training or orientation concerning the [System Name] ITCP. ITCP testing should be coordinated by the CPC and the Information System Security Officer (ISSO) to ensure that all major elements and components of [System Name] are tested. A testing schedule has been developed that includes the time frames and participants for the main testing steps: developing the test plan, executing the test plan, and documenting the test plan results and recorded in Appendix K and L. Testing must be performed at least annually to ensure that the ITCP could be relied on in the event of a system failure.

7.2 Documenting Test Plan Results

The test plans and the results of the tests is documented and maintained as part of the overall ITCP documentation. The latest test results are documented in Appendix L. Documentation helps in evaluating liabilities and identifying achievements. Once the test plans and their results have been documented, the CPC should forward the documentation to the ISSO and System Owner for review. The ISSO and System Owner must sign off on the documents to indicate that the plan was successfully tested. This signed documentation must then be provided to the CIO’s office for FISMA recording and tracking. This documentation will serve as a record of “observations” and will assist in determining which procedures worked well and which need to be modified.

7.3 Plan Maintenance

As a living document, the ITCP must be maintained and updated over time. The [System Name] System Owner is responsible for maintaining, testing, and updating the plan. However, the System Owner may delegate all or parts of this responsibility to the CPC, ISSO, or others, as appropriate. The System Owner, or designated delegate, should coordinate with key individuals in planning testing and training schedules and objectives, analyzing testing and training documentation (e.g., observations and results), and monitoring major changes to [System Name]. To ensure that the plan maintains a current state of readiness, it is important that the System Owner, or designated delegate, keep the following maintenance schedule:

· Review and update the plan annually or more frequently based on significant changes to the [System Name] application architecture

· Review and update the contact sheets and vendor information semi-annually or more frequently based on significant changes to [System Name] personnel, vendors, and supporting contractors

· Update the plan when test results indicate procedural changes.

Table 7-1: System Plan Maintenance

· Incorporate ‘lessons learned’ and change in policy and philosophy

· Manage distribution of plan updates

· Annual (at least) general update

· Semiannual updates of personnel contact information

· With significant changes to components, configurations, organization structure, or controls

· Following tests and exercises

· Schedule participation in all training and exercise events

· Within 3 months of hiring personnel with ITCP responsibilities

· Brief all key personnel on their responsibilities under the contingency plan

· Annually

· When redistributing the plan

· When adding new key personnel

· Review and update Activation procedures

· Review and update Recovery procedures

· Review and update Resumption procedures

· Annually, or as needed

· Conduct walkthrough exercises

· Conduct joint exercises with FDA personnel and supporting contractors

· Document and review lessons learned

· Annually (at least), or as needed

The System Owner, or designated delegate, is responsible for distributing the ITCP. Two copies of the ITCP should be distributed to those responsible for coordinating the plan and those who participate in recovery and resumption operations. Each person is required to maintain one copy of the plan holder’s office, and the other copy should be kept at the plan holder’s home for after-hours emergency situations. The original document is to be stored at a secure location that is easily accessible by both the System Owner and the CPC. If there is a change to the plan, the entire obsolete ITCP or obsolete page(s) must be retrieved or confirmed to have been destroyed by all personnel before the new or updated plan or pages are distributed. This practice will ensure that all personnel maintain the most current ITCP and that contingency operations can be completed efficiently. All key personnel are required to fully examine the ITCP to better understand their roles and responsibilities.

8 Appendices

· Organization Chart

· Emergency / Disaster Team Formulation

· Notification Contact List

· Vendor Contact List

· Equipment and Specifications

· Business Impact Analysis

· IT Standard Operating Procedures

· System Backup Procedures

· Damage Assessment Report

· Action Items Checklist

· ITCP Test Plan

· ITCP Results

· Acceptance Test Plan

· Service Level Agreements

· Additional CP (where applicable)

· Emergency Management Plan

· Continuity of Operations Plan (COOP) Contingency Plan Organization Chart

When notified by the CPC that the ITCP has been activated, the Coordinator will ensure the appropriate Recovery Teams are notified and Recovery Team Leaders apprised of the situation (not all teams may be required for all events).

Emergency / Disaster Team Formulation

Fill in the following tables as appropriate. Develop a flow diagram to show how these teams will interact.

Contingency Plan Development Team(s) Contingency Plan Work Group

The ITCP Work Group supports the ITCP Coordinator in the development of strategies to restore critical business function in the event of an emergency or disaster. The work group completes all preliminary activities, including selecting team members and reaching agreements on offsite storage and alternate facilities.

IT Contingency Plan Coordinator

Alternate Cont. Plan Coordinator

Team Member

Team Member

Procurement Team The Procurement Team acquires any required hardware/software and related documentation and/or other items that must be placed at the alternate facility.

Team Lead

Alternate Team Lead

Team Member

Team Member

Restoration Management Team The Restoration Management Team is responsible for determining the best course of action for restoring critical business functions in the event of an emergency or disaster.

Restoration Manager

Alternate Restoration Manager

Team Member

Team Member

Contingency Coordination Team The IT Contingency Coordinator is responsible for informing FDA management about the status of efforts to restore critical business functions in the event of an emergency or disaster and ensures the coordination of restoration activities at the alternate facility. The IT Contingency Coordination Team conveys management instructions and disseminates status information to leaders of other teams.

IT Contingency Coordinator

Alternate IT Contingency Manager

Team Member

Team Member

Disaster/Emergency Response Team The Disaster / Emergency Response Team assesses the functionality of systems, conducts and initial damage assessment, and provide ongoing status reports in the event of an emergency or disaster. After the Disaster / Emergency Response Team assesses an incident, members may disperse to lead or participate in other teams.

Alternate Team Lead

Infrastructure Representative

Network Operations Representative

Server Administration Representative

Recovery/Restoration Team The Recovery/Restoration Team has overall responsibility for restoring a system, either partially or fully, as instructed by the Restoration Management Team.

Alternate Recovery/Restoration Coordination

Team Member

Team Member

Team Member

Recovery Process for the Recovery Team:

These procedures are used for recovering a file from backup tapes. The Recovery Team is responsible for reloading all critical files necessary to continue production.

· Identify file and date from which file is to be recovered Time: __:__

· Identify tape number using tape log book Time: __:__

· If tape is not in tape library, request tape from recovery facility;

fill out with appropriate authorizing signature Time: __:__

· When tape is received, log date and time Time: __:__

· Place tape into drive and begin recovery process Time: __:__

· When file is recovered, notify Recovery Team Leader Time: __:__

Acceptance Team The Acceptance Team is responsible for conducting functional testing on and for accepting or rejecting a restored system after an emergency or disaster. A case for accepting a system with limited capabilities must be justified to management, which is responsible for declaring the system operational.

Acceptance Coordinator

Alternate Acceptance Coordination

Team Member

Team Member

Team Member

Additional Notification Contact List

PRIMARY

aLTERNATE

Vendor Contact List

Hardware/

Software

Equipment and Specifications

System Hardware

System/Application Software

Business/Mission Impact Analysis

[System Name]

Organization

Date BIA Completed

BIA POC

System Description: include architectural diagram and process flow.

Based on the outage impacts and allowable outage times provided. Use quantitative or qualitative scale (e.g., high/moderate/low).

High Mod Low Comments

Describe the business impact on mission if the critical resource is unavailable.

1.

2.

3.

4.

5.

6.

7.

8.

IT Standard Operating Procedures

Fill in the below and/or provide any SOPs that may currently exist for the system.

I – Recovery and Restoration Team Procedures

1. Receive notice that the operating environment has been recovered and is ready for system to be recovered

2. IT Equipment should be tested to see if they are operating

3. Request or receive offsite backup

4. Verify applications software version

5. Load the most recent version of the software

6. Validate the system was successfully recovered

7. Validate the required specific data and databases are available

8. Obtain the most recent backup of the system specific data

9. Load the system specific data

10. Validate system specific data has been recovered

· Verify data counts

· Verify date information

11. Validate security controls

12. Notify Acceptance Team

II – Acceptance Team Procedures

1. Validate system specific security

2. Validate all functions are operational

3. Read System Specific Data

4. Read Interconnected System Data

5. System Update Function for (XX)

6. System Update Function On Line

7. System Update Function for (XX)

8. Verify all updates are successful

9. Notify User Community

10. Prepare Report

System Backup Procedures

Describe additional application/system backup procedures. If additional application code/programming or unique conditions require additional archive procedures, document the process below. If not additional requirements are necessary leave this section blank.

Include Backup media and frequency (systems software, application software); Type of backup (weekly, daily - total, incremental); On-site location of backup and protective measures; Off-site location of backup and frequency; Backup logs, audit trails, etc.; Password accessibility.

Include the restoration methods used and needed to be available for recovery. The backup and restoration methods selected should be based on system and data availability and integrity requirements (as defined in the BIA).

Please list all (Organization Name [System Name]) vendors (third party) that archive or store backup media that are included in the procedure process.

Damage Assessment Report

The following report is to be used by the recovery teams in developing an assessment of the system failure.

DAMAGE ASSESSMENT REPORT

Name:

Event Information

Personnel Injuries:

System Information

Impact on Components:

Component Resources Affected:

Type of Damage to Resource:

Estimated Equipment Needs:

Recovery Information

Suggested Recovery Strategy:

Activation of CP Recommended:

FORMCHECKBOX

Y FORMCHECKBOX

N

ITCP Manager Signature:

Date/Time

Date/Time

ITCP Director Signature

Date/Time

Contingency Log and Action Item Checklist Example Template:

1. The Contingency Plan Coordinator (CPC) contacts the XXXX Group and instructs them to perform a damage assessment.

2. Complete Damage Assessment Report (Appendix G).

a. Check the cause of the system disruption, including type, scope, location, and time of incident.

b. Check whether the outage is localized (this system only) or widespread.

c. Check the location of failing components and those users without service.

d. Check the impact of the disruption or components damaged.

e. Check the functional status of all system components (e.g., fully functional, partially functional, nonfunctional).

f. Check the potential for additional disruption or system damage.

g. Check the identification of a single point of failure (if possible).

h. Check items to be replaced (e.g., hardware, software, firmware, supporting materials).

i. Check anticipated downtime of the application (e.g., longer than 2 days).

j. Classify disruption as Minor System Failure or Major System Failure.

3. The Recovery Teams contact the CPC to provide an estimated recovery time and begin repair of the components (i.e., the databases, servers, infrastructure or the application software).

4. The CPC notifies all system users that the Minor System Failure is being recovered and will be functioning under normal conditions within the estimated recovery period. Users are also notified via the Hot News bulletin.

5. Minor System Failure is recovered and incident is closed.

6. The CPC reviews the damage assessment report and contacts the Contingency Plan Director (CPD) to activate the contingency plan formally.

7. The CPC contacts all system user groups, alerting them of a major system outage and expected recovery time. Additionally, the CPC contacts the IT Management, the system Help Desk, and Program Office, which will notify additional user groups.

8. The CPC contacts all required Recovery Team personnel to initiate system or application recovery.

9. The CPC coordinates with the Data Center to obtain an estimated downtime…