DRAFT_SOW_-_OCIO_Disaster_Recovery_Services_-_For_Industry_Comments_1.docx

Department of the Interior (DOI) Office of the Chief Information Officer (OCIO) Disaster Recovery Services Statement of Work (SOW)

Overview:

The DOI OCIO has identified a requirement for a new disaster recovery task order.

Scope:

The requirement is to provide information technology services as identified in the SOW and it must include the following information technology resources:

1. Provide (IBM) mainframe for information technology disaster recovery testing, annual exercises and disaster declaration.

2. Provide hot site facility host various DOI network devices such as routers, switches and network circuits that connects various agencies and locations. The network devices and the circuits will be online, utilized and monitored for operational use.

3. Provide hot site facility for the mainframe backup operation. The backup is performed at the DOI production site and then replicated via a dedicated circuit to the contractor site into a DLM - Disk library for Mainframe by EMC. The DLM is owned and managed by DOI but it must be located at the contractor site and must be ready to connect to the contractor mainframe during a disaster declaration and disaster recovery exercises. The replication is a continuous 24 x 7 operational function.

4. Ability to either provide or host DOI network circuits.

5. Ability to host DOI network devices, DOI servers to keep up with the DOI OCIO business demands.

Organization:

DOI OCIO

7301 West Mansfield Avenue Denver, Colorado 802352230

Background and Objectives:

The Interior Business Center (IBC) is a DOI designated administrative support facility that provides common services to DOI and bureaus, as well as other Federal agencies, through negotiated cross servicing agreements supporting functions such as finance and accounting, procurement and contracts, personnel, property, and other general administrative areas. The OCIO obtains, configures, implements, and manages its own large scale mainframe computer and supporting systems to host IBC systems. The data center hosts a variety of agency/bureau administrative and mission-oriented systems.

Purpose:

The increasing dependence of the OCIO and its clients on automated information applications make planning for their protection, availability, and reliability under all conditions a matter of prime importance. In the event of a major disaster, the OCIO and IBC maintains contingency plans which management uses to ensure that losses are minimized to an acceptable level and business activities are resumed within a specific service level agreement (SLA). Disaster recovery is a key component to the business recovery plan. The purpose of this SOW is to provide disaster recovery services to accommodate a catastrophic event of such a magnitude that it requires the movement/relocation of the data center services and equipment to another site, provides for the relocation of key staff, and permits the OCIO the ability to verify the adequacy of the procedures and services by yearly testing of critical applications.

Disaster Recovery Background:

The OCIO’s current environment includes an information system infrastructure that resides in a multiple data centers. The environment includes an IBM mainframe.

OCIO hosts a large number of servers in multiple platforms including IBM mainframe.

OCIO has strategically defined certain systems and applications that require disaster recovery via Contractor services while other systems are recovered internally into a disaster recovery environment located at various OCIO data centers.

The OCIO has multiple data centers which include:

· The Albuquerque Data Center (ADC) located in New Mexico.

The scope of this requirement is mainly focused on the IBM mainframe which is currently hosted at ADC and the supporting infrastructure.

The mainframe hosts major Federal Governmental administrative applications (e.g. FPPS,) and agency/bureau specific administrative systems.

The business recovery site provides a location for key ADC staff located in Lakewood, Colorado, to relocate to continue operations while the applications are being hosted at the disaster recovery site.

Period of Performance:

The offered price is all inclusive of the final configuration and any proposed interim solution.

Please note that the Transition Period 03/15/2024 - 11/14/2024 will be utilized to order and establish network circuits and infrastructure. The transition period will not include any recovery or any other services from the contractor.

1. Recovery Facility Requirements

The Contractor shall provide and maintain physical facilities, equipment, network communications; electrical power; heating, ventilation, air conditioning; security; office and storage space; technical support, operational support, and consulting support; and other supplies and services required under this requirement.

The Contractor shall provide annual test time and technical assistance before, during, and after testing to ensure that the organization’s recovery plans will be implemented successfully at the Contractor’s recovery facilities at time of disaster declaration.

The Contractor shall ensure that all supplies and services provided under this requirement, including technical and operational staff support, are provided prior to testing (to plan and coordinate test activities), at time of testing, after testing to assess test results, at time of disaster alert, and, in the event of a disaster, within twenty-four (24) hours of disaster declaration, or at time of recovery facility occupancy, whichever is sooner.

The Contractor shall have two or more recovery facilities that provide redundant support for the Government requirements. The Government shall be assigned to a primary recovery facility. A secondary facility shall also be available for use during disaster recovery testing and at time of disaster declaration.

The Contractor shall provide the same specified resources listed in Attachment A or better at one of the sites.

The Contractor shall provide a fully operational hot site for scheduled tests, and within twenty-four (24) hours of a disaster declaration, or when a disaster declarer is ready to occupy a hot site facility, whichever is sooner. The hot site shall be provided for a period of up to six (6) weeks occupancy for each disaster declaration. The site shall also be available after the 6 weeks for each disaster declaration for an additional cost.

Each recovery facility shall be a fully operational information technology installation for mainframe, midrange, server and workstation systems that are dedicated to testing the Government's’ disaster recovery plans and recovering operations at time of disaster.

The Contractor shall guarantee to provide the same specified resources or better at one of the sites.

Facilities to be utilized by the Contractor must be able assist in providing SAS-70 audit and or C&A – Certification and Accreditation when needed.

2. Security Requirements

The following security requirements apply to the Contractor’s recovery facilities.

The Contractor shall provide each facility and its component parts (e.g., hot site, warm site, cold site, conference room space, storage space, network communications, and specialized equipment space) an environment to accommodate a sensitive/unclassified system. Facility security shall include, but not be limited to, the following.

Access to each facility and its component parts shall be restricted by a manned and/or automated access control system which identifies individuals, authenticates identity, validates access authorization, and maintains an audit trail. At time of occupancy for testing or disaster, the Government will provide a list of individuals authorized to access the facility and its component parts.

The Contractor shall provide twenty-four (24) hour/day security coverage when the Government uses its facility. Security coverage shall be provided for all internal and external facility components

Written security and emergency procedures shall be in place and posted conspicuously. Sign in logs shall be used whenever entering the recovery facility, at which time a unique access device shall be issued to each authorized person.

Each recovery facility shall contain fire detection and suppression systems in accordance with applicable Local, State, and Federal regulations.

Each recovery facility shall contain adequate environmental control systems to sustain continuous operations within the hot, warm, and cold sites. The environmental control systems shall satisfy the original equipment manufacturer’s specifications for its equipment, and shall include: heating, ventilation, air conditioning, chillers, water detection and protection, dual power feeds from separate power stations, and uninterruptible power supply. Each cold site shall contain adequate environmental systems to accommodate the systems under subscription. The Contractor shall provide a fully operational backup generator at each recovery facility within twelve (12) hours of disaster declaration. The backup generator shall have sufficient capacity to keep the computer systems, air conditioning, lighting, security system, fire detection and suppression system, and private branch exchanges (PBXs) operational in the event of a power disruption.

Each recovery facility shall contain adequate environmental control systems to sustain continuous operations. The environmental control systems shall satisfy the original equipment manufacturer’s specifications for its equipment. The Contractor shall provide environmental control systems, including heating, ventilation, air conditioning; uninterruptible power supply for each server and PBX, and surge protectors for each server and workstation.

Contractor personnel assigned to support the OCIO’s recovery exercises and recovery events shall be subjected to suitability screening and background investigation by the Contractor. The Government reserves the right to perform the screening as well, in accordance with Federal regulations. Background investigations shall include, at a minimum, checking criminal court records and verifying social security number, credit, employment and education records and reports to determine suitability for employment.

Contractor personnel assigned to support the OCIO’s recovery exercises and recovery events shall receive computer security awareness training in accordance with the Federal Information Security Management Act (FISMA).

The Contractor shall reformat and virus scan/virus eradicate all fixed disks on all servers and workstations immediately before a server or workstation is provided to the Government for a test and/or at time of disaster, when applicable.

The Contractor shall confirm that it has completed a Security Plan, Disaster Recovery Plan, and Security Review for the facilities, systems, and networks proposed to satisfy the requirements of this task order not more than one (1) year prior to the date of quote submission.

The Contractor’s facility shall undergo monthly security patching as needed, as performed by the Contractor, when applicable. The Contractor will patch or correct systems in accordance with the scan findings and provide a copy of the test and correction results to the Government. The Contractor shall implement and maintain appropriate counter measures to protect the Government test and recovery operations and safeguard sensitive Government information resources. When (SANS) testing occurs, the Government may (at its discretion) witness as a non-participatory member of this testing.

The Contractor shall comply with the Security Plan, and maintain the specified countermeasures to protect the Government test and recovery operations, and safeguard sensitive Government information resources.

The Contractor shall comply with the Disaster Recovery Plan and maintain the specified recovery capabilities to support the Government test and recovery operations in the event that a disaster affects one or more of the Contractor’s recovery facilities, systems, and/or networks.

The Contractor shall comply with the Security Review’s findings and recommendations and implement and maintain appropriate countermeasures to protect the Government test and recovery operations and safeguard sensitive Government information resources.

3. Technical Support Services

The Contractor shall provide technical support before (to plan and coordinate test activities), during, and after all tests (to assess test results), during a disaster alert, and during a declared disaster. Systems Engineers, Software Specialists, Communication Analysts, operations support personnel, facilities maintenance personnel, and all other necessary personnel shall be provided. Contractor technical support personnel shall perform functions such as preparing the hot site for the Government occupancy, configuring and maintaining equipment, analyzing and correcting equipment operation, compatibility, communication problems, and maintaining environmental support systems.

Additionally, a dedicated Contract Administrator and Project Manager shall be provided to handle scheduling exercises, contract modifications, scheduling meetings to plan exercises and drills, etc.

4. Equipment Maintenance

The Contractor shall provide twenty-four (24) hour/day, seven (7) day/week facility maintenance on all hot and cold site equipment (e.g., computer systems, HVAC). Preventive and corrective maintenance shall be provided in accordance with the original equipment manufacturer’s specifications.

5. Network Services

The Contractor shall provide a dedicated, high-capacity wide area network (WAN) to connect the Contractor’s recovery facilities. The WAN shall provide access to multiple local and long-distance carriers. Typically, the Government will install and maintain one or more dedicated and/or switched communication lines to access the Contractor’s WAN. The cost of installing and maintaining those communication lines is the Government’s responsibility. The cost of utilizing the Contractor’s WAN shall be included in the subscription cost. There shall be no additional cost to the Government to utilize the WAN to access one Contractor recovery facility from another Contractor recovery facility, or to utilize the Contractor’s WAN from point of access (e.g., the node nearest to the Government point of access) to point of destination (e.g., a hot site), during testing and/or at time of disaster.

Most Government agencies will obtain long-distance telecommunications circuits and/or services from the Federal Telecommunications Service (FTS) 2001 (and/or successor contract) network providers.

The Contractor shall be responsible for local service and interfacing with the long-distance network circuits and/or services. The Contractor shall provide channel service units (CSU)/digital service units (DSU)/integrated service units (ISU), modems, routers, and cables, etc., to connect the telecommunications circuits and/or services to the Contractor’s WAN, recovery facilities, and computer systems (the Government is responsible for installing the same equipment at its facilities).

The Contractor shall make available the communication accesses specified for each OCIO (e.g., [1] DS-1 access) connection.

The Contractor shall provide network diagnostic and test equipment.

The contract shall provide network forwarding in case the primary recovery site is not available. OCIO and its clients will have active dedicated circuits at the primary disaster recovery sites and the Contractor shall forward the network traffic to the secondary site in case the primary site is not available.

Full and complete access for Government connectivity to the contractor provided DR environment shall be available immediately after the award of the task order. Contractors should note that although network connectivity is the Government responsibility, We are expecting a solution to this problem from the Contractor – possibly a preferred relationship with a Telecommunications Provider, a contract vehicle that we could take advantage of, a technical or engineering solution that would provide an acceptably secure workaround, etc. The proposals will be evaluated specifically on the vendor’s efficacy and efficiency of the solution to this problem.

6. Testing

The Contractor shall make its hot sites available for recovery capability testing for the number of hours per year specified below. There shall be one (1) or more tests per year, per configuration, divided into one (1) or more eight (8) hour increments. The Contractor shall satisfy the Government requests to schedule test time within six (6) months of the date of request, and as far ahead as two (2) years into the future. Tests shall be on dates and times mutually agreeable to the Contractor and the Government.

The Contractor shall send a confirmation e-mail to the Government within seven (7) days of a test being scheduled. The confirmation shall include the date and time of the test, a copy of the test planning and assessment documentation from the Government’s last test, and a test planning form, to be completed by the Government and returned to the Contractor at least thirty (30) days prior to the test. As follow-up to the letter and the completed test planning form, the Contractor shall contact the Government, at least fourteen (14) days prior to the test, to coordinate all technical and logistical aspects of the test. The Contractor shall describe, and the Government will validate, who is responsible for what activities, and when they will be completed, before, during, and after the test. After the test, the Contractor and the Government shall provide a written assessment of each other's performance and the test results.

If a Government’s test is canceled by the Contractor prior to the test’s completion, due to another client declaring a disaster, or if a Government’s test cannot be completed due to the unavailability or malfunctioning of the Contractor’s system or network resources, the scheduled test time shall be credited in full to the Government. Credited test time shall be rescheduled, at the Government’s request, either during the current or the following contract year, and/or a mutually acceptable cost reduction shall be negotiated.

The Government may cancel or reschedule reserved test time sixty (60) or more days before a scheduled test without penalty.

7. Contractor Equipment, Software, Facility, and Network Modifications

The Contractor shall provide to the OCIO, and to the Contracting Officer’s Representative, written notification before making a change that will adversely impact the OCIO. Notification of a planned change to the Contractor’s hardware (including address scheme changes or CPU identification number change), software, firmware, facility, and/or network shall be provided at least sixty (60) days prior to implementation. No changes shall be made that prevent the Government from successfully testing or recovering its operations. The Government shall not be charged if the Contractor elects to make to upgrade or improve its capabilities. The changes shall be made at the Contractor’s expense since it was not requested by the government.

8. Service Level Agreement

1. The Contractor shall provide disaster recovery test dates within two weeks from request.

2. The OCIO is a hosting facility and is under strict deadlines submitting proposals and competing for services. OCIO will frequently require an estimate of cost for disaster recovery services associated with new business for which OCIO is competing. Therefore, the OCIO requests that the Contractor submit an estimated cost within two business days from the date of requesting an estimate. Estimates should contain the cost associated with hardware, network, storage, etc, so that the OCIO can generate its cost estimate faster. The new business obtained for the disaster recovery services will then be incorporated into this task order.

9. OCIO Recovery Requirements

The Contractor shall provide disaster recovery sites, computer systems, network resources, and technical professional services specified below to support disaster recovery test exercises and disaster recoveries within twenty-four (24) hours of a disaster declaration, or when the Government personnel occupy the Contractor’s recovery facility, whichever is sooner.

10. Additional Recovery Requirements

The Contractor shall provide additional disaster recovery services (e.g., disk replication, mirroring, and shadowing), electronic vaulting, remote journaling, storage area networks (SAN), network-attached storage (NAS), long-distance telecommunication circuits and/or services (e.g., DS1, DS3, VPN), Internet recovery services, tape handling services, system and network restoration services, disaster recovery planning consulting services, disaster recovery planning software, and mobile recovery facilities) as specified in Attachment A.

The specific IT disaster recovery requirements of the DOI are listed below.

Existing Disaster Recovery Facilities

The existing disaster recovery site for the ADC sites is provided by Kyndryl. It includes:

Systems and network hardware provided by the vendor, Government owned hardware stored on-site, and telecommunication circuits to accommodate access during testing and at time of disaster.

The costs to remove and relocate the Government-owned hardware, and the cost to terminate and install telecommunications circuits, will be taken into account as part of the Government’s price analysis.

Specific Requirements:

Hardware and network are defined in Attachment A.

The detailed testing requirements are defined in Attachment A.

Ability to conduct test exercises per task order year as indicated in the SOW.

Ability to change the requirements as the hosting capability of the center changes.

Full and complete access for the Government connectivity to the Contractor provided disaster recovery environment shall be available immediately after the award of the task order. Contractors should note that although network |connectivity is DOI responsibility, DOI is still expecting a solution to this problem from the Contractor – possibly a preferred relationship with a Telecommunications Provider, a contract vehicle that we could take advantage of, a technical or engineering solution that would provide an acceptably secure workaround, etc. The quotes will be evaluated specifically on the Contractor’s efficacy and efficiency of the solution to this problem.

ATTACHMENT A

Recovery exercise requirements:

The Department of Interior – (DOI-OCIO) needs two recovery exercises per contract year as indicated below.

Exercise One

The contractor or and DOI will mutually agree to combine the two exercises into one or not.

This exercise runs from 0800 thru the end of the test.

The exercise will be scheduled during regular work days unless DOI requests otherwise.

Optional: extra 24 hours for a total of 104 hours exercise.

Optional: on demand separate testing for 24 hours for a total of 24 hours exercise for the purpose testing certain recovery methods.

During the exercises various OCIO owned equipment, included in the table below, will also be exercised. This includes communications equipment as well as servers.

Support provisions include a conference room and/or office for OCIO personnel.

DOI equipment must be located in dedicated locked racks close in proximity to each other.

Recovery Exercises Schedule:

Exercises Schedule

1 Exercise / 16 hrs

1 Exercise / 16 hrs

Part 1: Network Circuits

Network Circuits

1. DOI Info:

300Mbps (500Mbps) PIP / VPNS

VPNS-54668

Lumen Info:

Order N26074327 29/KXGS/651967//MS PON A2006964430N01 500 meg

2. DOI Info:

100Mbps (200Mbps Port) PIP / VPNS

VPNS-54316

Lumen Info:

3. DOI Info:

50Mbps (100Mbps Port) PIP / VPNS

VPNS-53218

Part 2: Mainframe zOS Configuration

Same configuration for disaster recovery testing and for the disaster recovery declaration.

2. 1. SYSTEM RESOURCES FOR A DISASTER DECLARATION

Memory and DASD

Memory - 64GB FICON attached 6 cables

Connectivity

ICC Terminals

OSA-Express6S 1000Base-T Ethernet

PC's with 3270 emulation software

Off Site Continues Backup replication to DLM at the contractor site (Customer Owned)

Please provide two configuration / cost options regarding DASD as follows:

Option one:

the contractor provides the 38 TB DASD and the

Option two:

DOI will place DOI customer owned DASD and DOI would continuously replicate production data into to the disaster recovery site DASD owned by DOI.

The contractor will need to attach the customer owned DASD to contractor Mainframe at the time of an exercise or an actual disaster declaration.

This is not an option – this is DOI owned equipment and must be hosted at the contractor site:

Virtual Tape Library provided by DOI

Component

Serial number

Data Domain 6900

APM00220915829

VTE1

APM00220704653

VTE2

APM00220501535

2. 2. MAINFRAME CONFIGURATION FOR DISASTER RECOVERY EXERCISE

Same configuration listed in section 2.1

Part 3: DOI owned equipment This is not an option – this is DOI owned equipment and must be hosted at the contractor site:

3.1 DOI Network Equipment

Total Actual Power Load is 6.99 KW

3.2 Mainframe backup replication equipment

DLM (Customer Owned)

Virtual Tape Library

3.3 Servers (Customer owned)