2014-07-02_ECP_RFP_CURRENT_OPERATIONS_OVERVIEW.docx

CURRENT OPERATIONS OVERVIEW

1. PURPOSE

The purpose of this document is to provide Offerors with an overview of the current state or “as is” state of operations with the Department of State, Bureau of Information Resource Management’s Enterprise Server Operations Center (ESOC) and the Global Information Technology Modernization (GITM) programs. The intent is that this document will provide some context or background on each of the two programs.

2. Introduction and Background to the Domestic Data Center Operating Environment The Department of State, Information Resource Management (IRM) Office of the Deputy CIO for Operations (OPS), Directorate of Systems Integration Office (SIO), Enterprise Server Operations Center (ESOC) provides data center services to the Department.

ESOC’s mission is to provide enterprise-level domestic data center services, including Infrastructure as a Service (IaaS) via a private cloud, and overseas support for virtual infrastructure. The ESOC directly supports Department of State objectives by consolidating and supporting Department (1) data centers, (2) servers, and (3) data center services. The ESOC program benefits are higher system availability, data center scalability, enhanced security, and improved cost containment for the Department’s worldwide IT infrastructure.

ESOC operates multiple Tier 3 Domestic Data Centers for the Department. The current customer base predominately extends to 25,000 domestic IT users; however, centralized services for approximately the same number, or more, of overseas users are rapidly moving into the ESOC.

All of the ESOC Domestic Data Centers function as primary providers for enterprise applications, though in many cases a single data center is primary with one or more locations functioning as secondary backup data centers. Within the data centers, the ESOC hosts virtual infrastructure, organizational/user shared data, and disk-based file-level backups residing on managed storage that support the operations of the Department.

The virtual infrastructure consists of storage, compute, and networking to support approximately 4000 virtual machines. These machines are further supported on approximately 5.2 petabytes (PB) of storage leveraging virtual port channels (vPC) over 10+ GB networking.

Compute, storage, and networking for the 4000 virtual machines is duplicated and replicated to the secondary site(s). Organizational shares use approximately 722 terabytes at the read-write site with an exact read-only copy of the data available offsite. Disk-based file recovery and disaster recovery storage systems use approximately 2.7 PB of storage. See Figure 1 (ESOC Service Stack) below.

Figure 1 –ESOC Service Stack

The ESOC Domestic Data Center provides five levels of service: co-located, co-managed, hosted, managed, and engineering and project services. The services are progressive with each providing additional support that builds upon the preceding service levels as pictured in Figure 1 (ESOC Service Stack) above. These services are provided on OpenNet, ClassNet, and the Demilitarized Zones’ (DMZ) networks.

The Domestic Data Centers maintains two Tier 3 productions, enterprise data centers to provide these services. The two data centers are geographically separated and are on separate power grids:

· Domestic Data Center East is an 11,000 square foot data center co-located within a commercially maintained facility in the outskirts of the metro DC area; and,

· Domestic Data Center West is a 15,000 square foot data center in a federally-owned data center facility on a Federal Government campus in Colorado.

· The two production Domestic Data Centers are connected via two 10 gigabit connections to each; the connections each follow a geographically diverse route.

· Each site is fed by redundant power feeds from diverse power suppliers: up to 2 MW uninterrupted power to Domestic Data Center West; in excess of 30 MW to Domestic Data Center East.Figure 2 - Typical Server Row

· The infrastructure within the Domestic Data Centers is designed with complete redundancy for power and network from the external access points, through the Domestic Data Centers’ core, to the server racks. A typical server row configuration—as shown in Figure 2 (Typical Server Row)—is as follows:

· All network cabling is overhead;

· All power is under the floor or within the rack;

· Redundant network and power feeds are run to each row of racks via diverse routes;

· Redundant Communications Racks are located at each end of each row; and

· Standard rack is 42U 600mm wide X 1200mm deep Part #AR3300.

· The Domestic Data Centers provide redundant power to each rack. The redundant power feeds are fed by separate floor Power Distribution Units (PDUs) that are fed by separate UPS, each of which has its own generator backup.

· Domestic Data Center services are supported by a number of core infrastructure services that are provided by other IRM organizations, though most of these services are hosted within the Domestic Data Centers, including Active Directory, Antivirus, and Dynamic Host Configuration Protocol (DHCP).

· Network services are currently provided by Cisco switches and routers. A number of other Offeror technologies are used to provide DMZ firewall services, including BlueCoat, StoneGate, Juniper, McAfee, and IBM. The Department uses 10GB network connectivity for user traffic and backend storage traffic.

· The Domestic Data Centers are a tapeless environment and all backup data is written to NetApp replication devices. The Department currently has two primary backup and Disaster Recovery (DR) solutions in place.

· The virtual infrastructure (VI) components are deployed on the following networks:

· OpenNet

· Class Net

· DMZ

· The VI uses VMware vSphere as the Virtualization Technology (VT) with Network Appliance (NetApp) as the back-end storage. Currently, at Domestic Data Center East, Fibre Channel (FC) and Network File System (NFS) are the storage protocol utilized, while Domestic Data Center West utilizes NFS protocol. However, future Virtual Machines (VM) at Domestic Data Centers will be deployed on NFS/Internet Small Computer Systems Interface (iSCSI) as they transition away from FC. Additionally, the VI is leveraging Domestic Data Centers provided telecommunications infrastructure for network connectivity.

· Currently, the VI is deployed at two sites, Domestic Data Centers East and West. Both are not only state-of-the-art Data Centers that provide identical services, but they are also “hot” sites. Either site, therefore, can be leveraged for application deployments depending on customer requirements. From a VI management standpoint, these sites are managed by independent VMware vSphere implementations and, therefore, a failure of one site has no impact on the services provided by the other site.

· The architecture is described by a logical design, which is independent of hardware-specific details. This design includes the following:

· Two physical sites

· Clusters of hosts for load balancing through VMotion and VMware High Availability (HA) for host and Guest Operating System (OS) VM failure

· VMware vCenter integrated with Microsoft Active Directory (AD) – vCenter leverages the extensive inventory of existing AD users and groups to secure access to vSphere

· Redundancy in network and shared storage infrastructure

· System component monitoring, with Simple Network Management Protocol (SNMP) traps and email alerts

· VMware vCenter Update Manager for automating patching of all hosts and VMware tools

· In VMware vSphere, a Data Center is the highest-level logical boundary and is typically used to delineate separate physical sites/locations or vSphere infrastructures with completely independent purposes. Domestic Data Centers East and West have two independent virtual Data Centers.

· The Domestic Data Centers Storage Area Network (SAN) provides an IaaS to its customers. The Domestic Data Centers provides both SAN and Network Attached Storage (NAS) for the Department of State with the flexibility to meet diverse customer requirements – Logical Unit Numbers (LUNs) for databases, user and organizational shares, and storage for virtual machines. The storage infrastructure allows the Domestic Data Centers to provide monitoring, backup/recovery and fault tolerance to the Enterprise. The Domestic Data Centers are in the process of migrating from traditional fibre attached storage to an IP-based storage infrastructure using NFS/iSCSI for shared storage hosting in a virtualized environment.

3. Introduction to the Remote Office Operating Environment The Department of State, Information Resource Management (IRM), Office of the Deputy CIO for Operations (OPS), Directorate of Information Technology Infrastructure (ITI), Global Information Technology Modernization (GITM) Division serves as the IT modernization arm for the Department of State by providing centralized OpenNet and ClassNet desktop and core LAN IT equipment modernizations using a standardized life-cycle management process. GITM IT modernizations ensure that Department of State employees possess the latest hardware technology necessary to carry out the Department’s foreign policy mission worldwide and enable collaboration with U.S. Government agencies.

3.1 Existing Technical Environment

The standard GITM core suite hardware components currently include:

· Dual core switches

· Redundant layer 3 devices

· Distribution switch(es)

· Layer 2 devices which provide connectivity between the core and client switches

· Three physical servers

· Two virtual hosts (ESXi)

· One management server (used for system management and backup/data recovery (DR))

· Keyboard, Video, Mouse (KVM) device

· Primary and secondary storage (single array can be used if it meets redundancy requirements)

· Primary – Contains virtual machines

· Secondary – Contains backup data

· LTO-4/5 tape device (to meet offsite backup requirements)

· Three 110VAC, 20 Amp or 220VAC, 10 AMP Uninterrupted Power Supplies (UPS) (required runtime of 10 minutes)

· 1 rack (pre-existing racks are currently utilized) The graphic below (Figure 3) represents the general virtualized configuration that is currently being deployed by the GITM program:

Figure 3 - Current GITM Configuration The current GITM IT environment may vary from site to site, but in general, it consists of a combination of the following:

· DC11, DC12 – Microsoft domain controllers

· MB – Microsoft Exchange 2003 or 2010

· FP – Microsoft File/Print server for file shares

· AP – Application servers that normally consist of Microsoft SQL 2000/2005/2008 server

· IIS – Microsoft Internet Information Services (IIS) SMS – Microsoft Systems Management Server (SMS) or System Center Configuration Manager (SCCM)

· VH01/VH02 – VMware ESXi version 4.1-5.0 (clustered, managed by vCenter)

· MGT – Microsoft Windows server currently used to manage systems and backups.

Backups are transferred from the primary storage to the secondary storage. The current backup solution is TSM FastBack; however, GITM is transitioning to NetApp SnapProtect, which uses only one storage array.

· Tape System – The tape drive displayed above the MGT01 server consists of a SAS Linear Tape-Open (LTO)-4/5 single tape solution.

· RS1/RS2 – Core switches used to provide redundant communications to all core equipment and communications to distribution and client switches. The core switches are layer 3 devices.

· Primary Storage – A NetApp 2040A (12 X 450GB to 12 X 900GB) is the legacy system. No storage-based or VMware snapshots are taking place. FastBack currently provides all backup operations for VMs. Currently a NetApp 2240-2HA (12 X 600GB to 24 X 900GB) is being used to replace the legacy system, and to store the virtual machines (VM) listed on the graphic above. Expansion shelves may be used to expand primary storage requirements.

· Secondary Storage – A NetApp 2040A (12 X 450GB to 12 X 900GB) is the legacy system that is used to store backups from TSM FastBack. This system is being replaced with a NetApp 2240-2HA (12 X 600GB to 24 X 900GB. Expansion shelves may be used to expand secondary storage requirements. The secondary storage arrays are being removed as part of the NetApp 2240-2HA (12 X 600GB to 24 X 900GB fielding.

· The following backup solutions are currently considered part of the GITM baseline:

· IBM TSM FastBack (with UltraBac for “tape out”)

· NetApp SnapProtect

3.2 Existing Infrastructure

3.2.1 Virtual Host Servers: Physical servers are configured with VMware ESXi hypervisor. Licensing for the hypervisor is already in place in accordance with the current environment described in this document. In most cases, virtual host servers are currently configured in cluster to provide redundancy. A single virtual host (or two in larger sites) is able to resource all virtual machines at a single site in the event of a server failure.

3.2.2 Management Server: The current management server is a physical server with a Windows Server 2008R2 operating system. The management server currently provides site administrators with a system to manage their virtual environment. It also provides the backup/recovery solution processing requirement. The management server technical requirements are the same as a virtual host so that in a worst case scenario it can be repurposed into a virtual host.

3.2.3 Virtual Machine Configuration: The current VM configuration is a set of VMware VMs, networking infrastructure (virtual switches and ports) for the VMs, VMotion, IP storage, and service console elements that are preconfigured. In cases where the software is not a common application or the application has been created internally, the type of application and general characteristics of the application are provided below. The sum of the VMs provides a reasonable understanding of the overseas workload. The sections below provide some indication of the expected workload for each defined VM. There is the usual virtualization overhead with different mixes of workloads. VMware best practices recommend complimentary workloads be put on the same host server. The mapping includes design decisions and justifications as well. VM resources are defined by the application and the number of users the systems support. The user standard capacity distribution is described below, in section 3.2.4. All other capacities defined within this document are used to support other enterprise systems.

3.2.4 VM Minimum Configuration: The table below represents the virtual machine minimum resource and storage requirements for the current environment. Limited capacity is an issue for the Department of State. The Department would like to maximize capacity in order to provide overseas locations with more storage.

Users Users Users Users Users Users Users Users

3.2.5 Domain Controller: The Department of State AD domain controller authenticates and authorizes all users and computers in a Windows domain-type network, assigning and enforcing security policies for all computers and installing or updating software.

3.2.6 SCCM Distribution Point VMs: Microsoft SCCM is the software distribution backbone of the Department of State.

3.2.7 Exchange Server VMs: Microsoft Exchange 2010 server is being deployed at all overseas locations.

3.2.8 File/Print Server VMs: Each overseas user requires a minimum of five (5) GBs of usable shared storage space. Industry reports suggest growth of total data is about 10% per year. However, Department of State has realized that growth of total data is closer to at least 25% per year.

In addition, many of the locations’ printers are shared. On average, there is a shared printer for every 10 local workstations. Larger locations may require multiple print servers. Technologies like Common Internet Files Systems (CIFS) can take the place of File/Printer (FP) data stores however FPs will still be provided to support network printer requirements.

3.2.9 Network Management Server: The network management server provides environment monitoring tools. This virtual machine may have several applications installed on it and perform monitoring of the local site.

3.2.10 Lync Server VMs: The integration of voice, video and data services is one of the initiatives called for by the IRM Innovation Council. Microsoft Lync 2010 enables part of this initiative. Lync is deployed in two stages: Today, Microsoft Lync 2010 enables only Instant Messaging and Presence Services; over the next four years, the Department will begin deploying integrated voice and data. Lync features pertaining to conference call and enterprise voice features will also be enabled.

3.2.11 BlackBerry Enterprise Server VMs: BlackBerry Enterprise Server (BES) provides the BlackBerry interface to the Department’s current mobile solution platform. The middleware connects to Microsoft Exchange and redirects emails and synchronizes contacts and calendar information between servers and mobile devices.

3.2.12 Local Application Server VMs: Locations will have individual requirements for additional virtual machines. This can consist of a single VM, or up to 6 VMs.

3.2.13 Keyboard, Video, Mouse (KVM): The KVM is used to access the virtual hosts and management server directly.

3.2.14 UPS: Currently each overseas location is responsible for providing three 110VAC/20AMP or 220VAC/10AMP circuits for each rack. Not all Department of State locations have building level UPS (3 phase and 100amps). Environmental ranges are as follows:

Ambient Environmentals

4. Enterprise Software Environment The Government's current virtual infrastructure architecture uses VMware vSphere. The security infrastructure is based on Microsoft AD. The virtual infrastructure is comprised primarily of virtual machines built with the Microsoft 2008, R2 operating system. The Government has licensing vehicles in place to support the current environment. Licensing is based on existing numbers and forecasted changes.

image2.emf image3.png image1.jpeg