CMS TRA Virtualization Supp v 1 1_07282010.pdf

Restricted Distribution – For Official Use Only

Department of Health and Human Services

Centers for Medicare & Medicaid Services IT Modernization Program

CMS TRA –Virtualization Supplement

Version 1.1

July 28, 2010

CMS TRA –Virtualization Supplement i Version 1.1 July 28, 2010

Restricted Distribution – For Official Use Only

Foreword

The Technical Reference Architecture, Version 2.0 (TRA), provides the technical architecture approach and technical reference standards of the Centers for Medicare & Medicaid Services (CMS).

The TRA has been reviewed and accepted as a foundational component of CMS’ Enterprise

Architecture in accordance with CMS’ Information Technology (IT) governance process. TRA

Supplements will provide additional engineering detail allowing CMS contractors to build environments that adhere to both the approved CMS architecture as well as other CMS standards.

This CMS Technical Reference Architecture –Virtualization Supplement, Version 1.1, provides the approved guidelines and standards for server and network virtualization in all CMS Production

Environments in accordance with CMS’ technical architecture approach and technical reference standards.

CMS’ Chief Technology Officer leads the development of this Architecture with the support of all components of the Office of Information Services (OIS) and input from CMS’ IT contractors. The

Agency maintains the Architecture according to its established Business Rules, as described in CMS

TRA – Business Rules Supplement, Version 1.0. The CMS TRA document and the CMS TRA

Supplement documents, as well as their development and maintenance schedules, are managed by the

CMS Enterprise Architecture and Strategy Group (EASG)/Division of IT Governance.

Any changes to the Technical Reference Architecture must be approved by the CMS Chief

Technology Officer. Any request for grant of special considerations should go to the CMS Technical

Review Board (TRB). The CMS TRA – Architecture Change Request Process Supplement, Version

1.0 describes the process for handling all requests for changes.

/s/ July 28, 2010

Henry Chao Date Chief Technology Officer (CTO) Office of Information Services

CMS TRA –Virtualization Supplement ii

Restricted Distribution – For Official Use Only

Record of Changes

Number Date Reference A=Add, M=Modify, D=Delete

Description of Change CR #

1 May 24, 2010 All Publication of version 1.0. NA

2 July 28, 2010 Sections1, 2, 6, and Acronym List

A, M Incorporation of mainframe portions of the Virtualization Supplement as Version 1.1 for publication.

NA

CR: Change Request

CMS TRA –Virtualization Supplement iii

Restricted Distribution – For Official Use Only

Table of Contents

1. Introduction

1.1 Background

1.2 Purpose

1.3 Scope

1.4 Audience

1.5 CMS Standard Terminology

1.6 Document Organization

2. Server Virtualization

2.1 Overview of Server Virtualization

2.2 Server Virtualization-Specific Terminology

2.3 Business Rules for Server Virtualization (SV)

2.3.1 Business Rule SV-1

2.3.2 Business Rule SV-2

2.3.3 Business Rule SV-3

2.3.4 Business Rule SV-4

2.3.5 Business Rule SV-5

2.3.6 Business Rule SV-6

2.3.7 Business Rule SV-7

2.3.8 Business Rule SV-8

2.3.9 Business Rule SV-9

2.3.10 Business Rule SV-10

2.3.11 Business Rule SV-11

2.3.12 Business Rule SV-12

2.3.13 Business Rule SV-13

2.3.14 Business Rule SV-14

2.3.15 Business Rule SV-15

3. UNIX Virtualization

3.1 Business Rules for UNIX Virtualization

3.1.1 Business Rule UV-1

3.1.2 Business Rule UV-2

3.1.3 Business Rule UV-3

3.1.4 Business Rule UV-4

3.1.5 Business Rule UV-5

3.1.6 Business Rule UV-6

3.1.7 Business Rule UV-7

3.1.8 Business Rule UV-8

3.2 UNIX Virtualization-Specific Terminology

3.3 UNIX Virtualization Implementation and Security

3.3.1 UNIX Production Zones

CMS TRA –Virtualization Supplement iv

Restricted Distribution – For Official Use Only

3.3.2 UNIX Virtualization Implementation Requirements

3.3.3 UNIX Virtualization Security Rules

4. Windows Virtualization

4.1 Windows Virtualization Business Rules (WV)

4.1.1 Business Rule WV-1

4.1.2 Business Rule WV-2

4.1.3 Business Rule WV-3

4.1.4 Business Rule WV-4

4.1.5 Business Rule WV-5

4.1.6 Business Rule WV-6

4.1.7 Business Rule WV-7

4.1.8 Business Rule WV-8

4.2 Windows Virtualization-Specific Terminology

4.3 Windows Server Implementation and Security

4.3.1 Windows Virtualization Implementation Rules

5. Network Virtualization

5.1 Overview of Network Virtualization

5.2 Network Virtualization-Specific Terminology

5.3 Business Rules for Network Virtualization (NV)

5.3.1 Business Rule NV-1

5.3.2 Business Rule NV-2

5.3.3 Business Rule NV-3

5.3.4 Business Rule NV-4

5.3.5 Business Rule NV-5

5.3.6 Business Rule NV-6

5.4 Network Virtualization Scenarios

5.4.1 WAN Virtualization

5.4.2 LAN Virtualization

5.4.3 Virtual Network Inside a Physical Server

5.4.4 Private Virtual Local Area Network

5.5 Network Virtualization Implementation and Security Considerations

5.5.1 WAN

5.5.2 LAN and Core Network

5.5.3 VLAN Tagging (802.1Q)

5.5.4 Production Zone Network Connectivity

6. Mainframe Virtualization

6.1 Business Rules for Mainframe Virtualization (MV)

6.1.1 Business Rule MV-1

6.1.2 Business Rule MV-2

6.1.3 Business Rule MV-3

6.1.4 Business Rule MV-4

CMS TRA –Virtualization Supplement v

Restricted Distribution – For Official Use Only

6.1.5 Business Rule MV-5

6.1.6 Business Rule MV-6

6.1.7 Business Rule MV-7

6.1.8 Business Rule MV-8

6.1.9 Business Rule MV-9

6.1.10 Business Rule MV-10

6.1.11 Business Rule MV-11

6.1.12 Business Rule MV-12

6.1.13 Business Rule MV-13

6.1.14 Business Rule MV-14

6.1.15 Business Rule MV-15

6.1.16 Business Rule MV-16

6.1.17 Business Rule MV-17

6.1.18 Business Rule MV-18

6.1.19 Business Rule MV-19

6.1.20 Business Rule MV-20

6.1.21 Business Rule MV-21

6.1.22 Business Rule MV-22

6.1.23 Business Rule MV-23

6.2 Mainframe Virtualization-Specific Terminology

6.3 Mainframe Server Implementation and Security

6.3.1 Mainframe Virtualization Implementation Rules

6.3.2 Mainframe Virtualization Security Rules

Acronyms

List of References

CMS TRA –Virtualization Supplement vi

Restricted Distribution – For Official Use Only

List of Figures

Figure 1. UNIX Server Virtualization

Figure 2. Windows Server Virtualization

Figure 3. Application Zone VLAN before Server Virtualization

Figure 4. Application Zone VLAN After Server Virtualization

Figure 5. Virtual Network

Figure 6. Private VLAN

Figure 7. Server Connections With VDC Technology

Figure 8. Physical Server Connection to Management Zone

Figure 9. Virtual Server Connection to Management Zone

Figure 10. Mainframe Virtualization

List of Tables

Table 1. Definitions of Standard CMS Terms

Table 2. Server Virtualization-Specific Terms

Table 3. Virtualization Implementation-Specific Security Boundaries

Table 4. Solaris Virtualization-Specific Terms

Table 5. Windows Virtualization-Specific Terms

Table 6. Network Path Isolation and Virtualization-Specific Terms

Table 7. Mainframe Virtualization-Specific Terms

CMS TRA –Virtualization Supplement 1

Restricted Distribution – For Official Use Only

1. Introduction

1.1 Background

The Centers for Medicare & Medicaid Services (CMS) are undertaking the development of a large-scale Information Technology (IT) Modernization initiative to improve the quality and delivery of health care services to beneficiaries, providers, and business partners. This initiative spans the CMS enterprise and includes development and implementation of mid-tier and mainframe infrastructure known as the CMS Technical Architecture, a multi-zone architecture that is crucial to the effective operations of all CMS Production Environments. As CMS implements the components of the IT Modernization, it needs new policies, processes, and procedures for managing and controlling the enterprise.

This CMS Technical Reference Architecture –Virtualization Supplement (hereafter simply ―CMS

TRA – Virtualization Supplement‖) defines CMS’ Server and Network Virtualization standards.

The CMS Technical Reference Architecture document contains CMS’ architectural standards that are employed in all of the Agency’s production environments (hereafter simply ―CMS

Production Environments‖).

1.2 Purpose

This document defines the Virtualization guidelines and standards for the CMS Production

Environments. Once the Agency promulgates these standards, this document should assist all

Agency contractors in sustaining the three critical technical objectives in support of CMS’ health care mission: (1) move CMS Production workloads efficiently, (2) provide appropriate and sufficient disaster recovery capability, and (3) facilitate the migration and transition of CMS

Business Owner applications into new production environments. Variations to the TRA

Virtualization standards require consensus from all specified contractor stakeholders and explicit approval from CMS’ technical and executive-level management.

1.3 Scope

This document facilitates the development of CMS Virtualization guidelines and standards used by CMS and CMS/Contractor partners for the CMS Production Environments. The guidance and standards stated in this CMS TRA – Virtualization Supplement reflect the agreed-upon industry and government best practices to support the most viable approach for CMS that meets legislatively mandated security and privacy requirements and current technical standards.

1.4 Audience

CMS has expressly restricted the distribution of this document to CMS staff; The MITRE

Corporation, the Agency’s Federally Funded Research and Development Center (FFRDC) advisor; and any entity that has received explicit access to this document from CMS executive management.

Centers for Medicare & Medicaid Services Introduction

CMS TRA –Virtualization Supplement 2

Restricted Distribution – For Official Use Only

1.5 CMS Standard Terminology

This document relies on specific definitions of terms to explain the concepts that support CMS’ foundation for and preparation of this Supplement. To ensure the audience fully understands the conceptual foundation for this guidance, the presentation of the following terms, definitions, and explanations progressively builds the foundation for the guidance. The definition of specific terminology appears in the sections describing each type of virtualization (UNIX, Windows, Mainframe, or Network) and implementation. In order to facilitate the alignment between existing terminology and virtualization or implementation-specific terms, Table 1 provides the definitions of standard CMS terminology first.

Table 1. Definitions of Standard CMS Terms

Term Definition

Enterprise Services Network services such as WebSphere Application Server (WAS)/Message Queue (MQ), Lightweight Directory Access Protocol (LDAP), and Firewalls.

Environment One of four types of CMS data center environments supporting business application projects built on distributed systems platforms: Development, Test and Implementation (a.k.a. Pre-Production), and Production.

Zone A security architectural element for CMS Production Environments. Firewalls separate each zone to support application systems’ security. The CMS TRA

– Security Services Supplement defines four zones: the “Presentation Zone” supporting web servers, the “Application Zone” supporting business logic for Enterprise Applications, the “Data Zone” supporting database services for Enterprise Applications, and the “Management Zone” supporting systems management functions performed by privileged users, such as system backup, patch management, and security monitoring functions.

1.6 Document Organization

This document is organized as follows:

Section Purpose

Section 2: Server Virtualization Provides initial discussion points to help facilitate the development of CMS Server Virtualization standards to ensure consistency across all CMS Production Environments.

Section3: UNIX Virtualization Provides initial discussion points to help facilitate the development of CMS UNIX Virtualization standards to ensure consistency across all CMS Production Environments.

Section 4: Windows Virtualization Provides initial discussion points to help facilitate the development of CMS Windows Virtualization standards to ensure consistency across all CMS Production Environments.

Section 5: Network Virtualization Provides initial discussion points to help facilitate the development of CMS Network Virtualization standards to ensure consistency across all CMS Production Environments.

Centers for Medicare & Medicaid Services Introduction

CMS TRA –Virtualization Supplement 3

Restricted Distribution – For Official Use Only

Section Purpose

Section 6: Mainframe Virtualization Provides initial discussion points to help facilitate development of CMS Mainframe Virtualization standards to ensure consistency across all CMS Production Environments.

Acronym List Defines the acronyms used in this document.

List of References Lists the references used in preparing this document.

CMS TRA –Virtualization Supplement 4

Restricted Distribution – For Official Use Only

2. Server Virtualization

CMS’ overarching goal for virtualization is to facilitate the Agency’s transition from managing individual data center components to managing pools of computing resources. The following goals are common to all server virtualization implementations:

Consolidating applications to run on a single server

Improving the efficiency of data center operations by increasing the utilization of computing resources

Simplifying the management and maintenance of infrastructure

Enhancing business continuity and disaster recovery (DR) capabilities.

Successful server virtualization can lead to improved server utilization and more efficient use of server assets. Server virtualization is also crucial to ensuring the success of server consolidation projects that maintain the isolation of separate systems. This section addresses the principles of server virtualization and applicable business rules. It describes implementation options and identifies the design and implementation requirements for consistency with the CMS Technical

Reference Architecture for UNIX, Windows, and Mainframe virtualization implementations.

Finally, it describes the necessary requirements for these implementations to ensure adherence to the policies described in the CMS TRA – Security Services Supplement and the CMS Information

Security (IS) Acceptable Risk Safeguards (ARS) (hereafter simply ―CMS ARS‖).

2.1 Overview of Server Virtualization

The main driver for virtualization is consolidating multiple hosts and services on a single machine. Virtualization reduces costs by sharing hardware, infrastructure, and administration.

The benefits of server virtualization include:

Increased hardware utilization

Greater flexibility in resource allocation

Reduced power requirements

Reduce server hardware footprint

Fewer management costs

Lower cost of ownership.

In addition to cost savings, other drivers affect and shape server virtualization decisions. These drivers include:

Server sprawl – managing the ever-increasing need for hardware and software (and controlling the cost of these investments), data center floor space, power, and cooling

Centers for Medicare & Medicaid Services Server Virtualization

CMS TRA –Virtualization Supplement 5

Restricted Distribution – For Official Use Only

Tight IT budgets – with continuing pressures to do ―more with less,‖ virtualization can achieve significant savings in costs

Challenges of the mission-critical environment – mission-critical business applications that demand high performance, availability, and security requirements may seem harder to consolidate and ―virtualize‖ (e.g., segregating production environments from lower environments). Nevertheless, virtualization opportunities do exist in the mission-critical environment.

2.2 Server Virtualization-Specific Terminology

Although specific server virtualization implementations employ disparate terminologies and techniques o identify and implement architectural elements and their associated technologies, there is significant commonality in the underlying architectural and operational requirements for the deployment of server virtualization. This commonality stems from the common goals of and architectural elements employed by server virtualization implementations. The following terminology in Table 2 applies to all server virtualization implementations.

Table 2. Server Virtualization-Specific Terms

Term Definition

Application An operating system (OS), e.g., Solaris 10, or a program running on an OS that implements an Enterprise Service (see subsection 1.5).

Virtualization Implementation

A method for providing virtualized computing facilities to an application.

For the purposes of this supplement, it means UNIX, Windows, or Mainframe virtualization.

Virtualization Server A single hardware platform that supports multiple applications running independently.

Blade Server A single hardware platform that supports multiple virtualization servers.

Generally, processor and memory resources reside in separate units called blades and access pooled interfaces and storage.

Blade Server Manager A management application used to configure and monitor resources on blade servers.

Resource Architectural elements required by a virtualization server. Resources can be divided into the following four basic types (Memory, Interfaces, Processors, and Storage):

Memory: Volatile data storage that can be read, written, and erased by applications. Memory usage quotas can be enacted to limit memory usage on a per-application basis.

Interfaces

– Network Interface: An interface to an Internet Protocol (IP)-based data network. This is as opposed to an interface to a Storage Area Network (SAN) or directly connected peripheral. Network interfaces can be divided into virtual and physical interfaces.

– Virtual network interfaces share not only network interface cards

(NIC) but the IP stack, which communicates with an IP data network external to the virtualization server. This single stack supports network address translation (NAT) of virtual applications and the shared NIC and IP stack. The virtual interface may be associated with a NIC or a Virtual Local Area Network (VLAN) on a NIC.

CMS TRA –Virtualization Supplement 6

Restricted Distribution – For Official Use Only

Term Definition

– Physical network interfaces share only the NICs. Each application has its own IP stack and data link layer instance. The data link layer instance may be associated with a NIC or a VLAN on a NIC.

Processors: This resource type includes shared Central Processing

Units (CPU) and special use processors, Storage: Permanent data storage that can be read, written, or erased via an OS either directly through an OS interface or by other applications using an application-programming interface (API) to the OS.

Storage is divided into file systems and further sub-divided into directory structures and their individual files. Access to these file systems, directories, or files can be controlled based on user or group. In addition, these file systems can have per user or group usage quotas enacted to limit file system usage. Storage can be backed up and restored either via networked or storage or backup servers, e.g., tape backup devices.

Instance An individual physical instantiation of a resource, e.g., a single CPU on a virtualization server or a single NIC.

Pool A set of resource instances of similar type on a single virtualization server, e.g., a set of CPUs that can be used by an application.

OS Variant A specific make and version of an operating system, e.g., Solaris 10.

Oversubscribed Pooled Resource

A pooled resource is oversubscribed when the sum of all minimum subscriptions is greater than 100 percent of the total available resources in the pool. For example, if five (5) virtual machines (VM) share a processor pool and their minimum processor utilization is set to 25 percent, then the processor pool is 25 percent oversubscribed.

Virtualization Controller (a.k.a hypervisor)

An OS variant that can be used to configure virtual instances of other OS variants and their global resource quotas, as well as to enforce these quotas. Global resource quotas pertain to all instances of all resources on the virtualization server. As a result, only one instance of the virtualization controller may be running on a single virtualization server.

Hypervisors can be categorized as type 1 or type 2:

Type 1 Hypervisor: The virtualization layer is installed on a clean system. Access to hardware resources from the virtualization layer is direct, providing great efficiency, performance, and scalability. One guest OS instance is provided with special service controls to provide access to the hypervisor.

Type 1 Embedded Hypervisor: Similar to a Type 1 Hypervisor, the embedded hypervisor provides hardware-like security and reliability because it does not use a privileged Guest OS to monitor and manage.

Type 2 (Host) Hypervisor: The virtualization software is run as an application on the pre-existing OS. This host OS is relied upon for device support and physical resource management. This method is flexible, supporting the broadest range of hardware configurations, but performance is lower than with a Type 1 Hypervisor. Another drawback is the increased security vulnerabilities of the host OS that is now a source of control for the VMs.

CMS TRA –Virtualization Supplement 7

Restricted Distribution – For Official Use Only

Term Definition

Supervised OS [a.k.a.

Virtual Machine (VM)]

An instance of an OS variant, e.g., Windows 2003 Server, running under the control of a hypervisor.

Management Access Access to a VM or hypervisor using an account with management privileges

User Access Access to a VM using an account with user privileges

Management Application An application that can only be run with management access. This includes management programs and OS.

User Application An application that can be run with either user or management access.

These applications include business and office automation applications.

Project Work done to develop a single application. This may be done by multiple contractors; however, the project’s scope should be covered by a single statement of work.

Despite this commonality, there are significant differences between Solaris, Windows, and

Mainframe virtualization implementations. Specifically, there are substantial differences in the hardware, software, and security architectures employed by these implementations. These architectural differences require identification of the hypervisor and network security boundaries with implementation-specific architectural elements. As a result, server virtualization will be further subdivided along these lines. Table 3 defines the terms for implementation-specific security boundaries.

Table 3. Virtualization Implementation-Specific Security Boundaries

Virtualization Implementation

Hypervisor Security

Boundary Network Security Boundary

Hypervisor Architecture

Solaris Physical Sun Server

User Data: Physical NIC

Not a hypervisor Management Segment: Physical NIC

Security Segment: Physical NIC

VMware on Egenera Egenera pBlade

User Data: Virtual NIC on User VLAN mapped to physical cBlade NIC for inter-zone communications through firewalls

Type 1

Management Segment: Virtual NIC on internal Management VLAN mapped to physical cBlade NIC in external Management

VLAN

Security Segment: Virtual NIC on Security

VLAN mapped to physical cBlade NIC in external Security VLAN

IBM System z10 LPAR

User Data: Virtual NIC on User VLAN mapped to physical NIC on CHIPID dedicated to the LPAR

Type 1

Based on discussions with Sun Solaris Container subject matter experts.

CMS TRA –Virtualization Supplement 8

Restricted Distribution – For Official Use Only

2.3 Business Rules for Server Virtualization (SV)

Although server virtualization technologies are evolving at a rapid pace, CMS has established the following business rules to help the implementation of these technologies meet CMS’ needs.

The following server-specific virtualization business rules support the consistent implementations of these technologies in accordance with the CMS TRA guidance.

2.3.1 Business Rule SV-1

CMS will leverage virtualization technologies to gain operational efficiencies and agility as long as security requirements are maintained or exceeded.

2.3.2 Business Rule SV-2

Applications that have conflicting maintenance windows and uptime requirements must not be hosted on the same virtualization server and/or VM.

2.3.3 Business Rule SV-3

Applications with disparate security assurance levels must not be hosted on the same virtualization server and/or VM.

2.3.4 Business Rule SV-4

Applications are not permitted to run n-zones (web front end/business logic/database) virtualized on a single virtualization server in Production environments.

2.3.5 Business Rule SV-5

All virtualization servers and VMs must configure cluster high-availability and disaster recovery services to mitigate failure.

2.3.6 Business Rule SV-6

Production and Non-production (Test, Development, and Implementation) environments virtualization servers, and their associated VMs, must reside on separate physical server hardware or LPARs (see subsection 6.2).

2.3.7 Business Rule SV-7

All VMs must have adequate resource constraints (upper and lower bounds) to ensure that they do not adversely impact the performance of other VMs on the same virtualization server.

2.3.8 Business Rule SV-8

CMS shall gather virtualization server and VM performance metrics on a consistent basis to ensure maintenance of proper resource allocation.

CMS TRA –Virtualization Supplement 9

Restricted Distribution – For Official Use Only

2.3.9 Business Rule SV-9

All virtualization servers and VMs must adhere to security-hardening requirements established by the Chief Information Security Officer (CISO). These requirements include:

Restrict power on/power off of guests

Restrict copying of guests

Restrict host-to-guest commands, and guest-to-guest commands

Restrict creation or deletion of virtual machines and virtual disks.

2.3.10 Business Rule SV-10

CMS must maintain separation of duties between the hypervisor and VMs.

2.3.11 Business Rule SV-11

CMS shall continuously monitor the hypervisor for unauthorized changes and other anomalies.

2.3.12 Business Rule SV-12

The CMS Change Control Board (CCB) process shall govern all changes to the hypervisor and

VMs.

2.3.13 Business Rule SV-13

Development and Test business applications, and their associated VMs, may reside on the same virtualization server when physical separation is not required to meet performance requirements and/or security requirements.

2.3.14 Business Rule SV-14

Management access for virtualization server must be accomplished via dedicated physical network ports on the server that are accessible only by management stations in the CMS

Management Zone.

2.3.15 Business Rule SV-15

All virtual instances resources shall be tracked in the CMS asset management database and provide a cross-reference to the host server hardware and host operating system.

CMS TRA –Virtualization Supplement 10

Restricted Distribution – For Official Use Only

3. UNIX Virtualization

This section provides the UNIX Virtualization (UV) business rules to help the implementation of these technologies meet CMS’ needs. The following server-specific virtualization business rules support the consistent implementations of these technologies in accordance with the CMS TRA guidance.

3.1 Business Rules for UNIX Virtualization

In addition to the Server Virtualization Business rules listed in subsection 2.3, CMS derived the following UNIX-specific business rules from the CMS TRA Business Rules and TRA Distributed

Systems Platform Supplement.

3.1.1 Business Rule UV-1

CMS will not grant root access to the servers to the developers or users.

3.1.2 Business Rule UV-2

All users must adhere to CMS Security policies.

3.1.3 Business Rule UV-3

The specific environment’s role-based access criteria will control all access rights.

3.1.4 Business Rule UV-4

All access is restricted to CMS-approved authentication methods.

3.1.5 Business Rule UV-5

The business application must adhere to the CMS TRA.

3.1.6 Business Rule UV-6

The business application must have the capability for scheduling using an automated tool, e.g., Tivoli Workload Scheduler (TWS).

3.1.7 Business Rule UV-7

CMS-standard interfaces shall be used between the Application and Data Zone.

3.1.8 Business Rule UV-8

UNIX virtualization only applies to the use of resources controlled by UNIX virtualization servers, e.g., not routers.

Centers for Medicare & Medicaid Services UNIX Virtualization

CMS TRA –Virtualization Supplement 11

Restricted Distribution – For Official Use Only

3.2 UNIX Virtualization-Specific Terminology

Table 4 defines the terms specific to virtualization of CMS UNIX server implementations.

Table 4. Solaris Virtualization-Specific Terms

Term Definition

Solaris Container (a.k.a. Solaris Zone)

A VM instance on a Solaris 10 virtualization server. The OS variant may be Solaris 8, 9 or 10, or Linux.

Global Container An instance of a Solaris 10 OS that assigns resources to and controls the use of these resources by VMs. Only one Global Container may run on a single virtualization server.

Local Container An instance of a VM running on a Solaris 10 virtualization server. Multiple Local Containers may run on a single virtualization server.

Public Interface A physical or virtual interface that serves traffic in zones other than Management Zones.

3.3 UNIX Virtualization Implementation and Security

The following subsections provide a description of the design and implementation requirements to ensure consistency with the CMS TRA vision.

3.3.1 UNIX Production Zones

Figure 1depicts the UNIX server virtualization in a three-zone environment. In this three-zone environment, intranet and extranet applications use separate virtualization servers. The TRA mandates that there must always be physical separation between the security interface and other management interfaces. Although Figure 1 shows a single management interface for such functions as backup, in practice, multiple management interfaces are in use.

CMS TRA –Virtualization Supplement 12

Figure 1. UNIX Server Virtualization

3.3.2 UNIX Virtualization Implementation Requirements

CMS derived the following UNIX-specific implementation requirements from the business rules listed in subsections 2.3 and 2.4.1 and industry best practices. The general requirements apply to all environments and zones within that environment. In the current CMS implementation, all environments use the same configurations as the Production environment. Specifically, all virtualization servers use Solaris 10 on Sun servers. The virtualization supervisor is implemented in the Global Container and VMs are implemented in local containers.

3.3.2.1 General UNIX Virtualization Implementation Requirements

The following implementation requirements apply to UNIX virtualization servers and VMs in all environments and zones.

1. All management access must be limited to a unique management interface.

2. Management interfaces must not be pooled.

3. Management interfaces serving the Security segment must be physically separate from those serving all other Management segments, e.g., Backup segments.

4. Virtualization servers in the Management Zone must not oversubscribe pooled resources, e.g., memory.

5. No business applications may run in the virtualization controller.

Management

Security

Prod Zone-1

Prod Zone-2

Virtual

Machine

Virtualization Server

Virtual

Machine

Virtual

Machine

UNIX Server

CMS TRA –Virtualization Supplement 13

Restricted Distribution – For Official Use Only

6. Disk quotas must be enacted on a per-user and group basis to avoid filling file systems.

7. Processor resources should be allocated at the ―core‖ level to prevent contention for special use processors.

8. VMs must not oversubscribe any pooled resources.

9. Multiple projects may not share the same virtualization server, except when CMS approves a multiple-project configuration for the Production environment.

10. VMs must be configured to save crash dump information, and should not overwrite crash dump information after rebooting.

11. Virtualization servers must not be shared between zones.

12. Public interfaces may use pooled interfaces.

3.3.2.2 Zone-Specific UNIX Virtualization Implementation Rules

The following implementation rules apply to UNIX virtualization servers and VMs in all environments and their zones. The general rules in subsection 3.3.2.1 apply to all zones within the environment.

3.3.2.2.1 Data Zone Rules

The following implementation rules apply to UNIX virtualization servers and VMs in all environments and their Data Zones.

1. Data Zone VMs must user physical interfaces.

3.3.2.2.2 Application Zone Rules

The following implementation rules apply to UNIX virtualization servers and VMs in Production environments and their Application Zones.

1. Production Application Zone local VMs may use VLAN virtual interfaces.

2. Production Application Zone virtualization servers may host only business applications that process Protected Health Information (PHI)/Personally Identifiable Information (PII) or those that do not process PHI/PII, but not both.

3.3.2.2.3 Presentation Zone Rules

The following implementation rules apply to UNIX virtualization servers and VMs in Production environments and their Presentation Zones.

1. Production Presentation Zone local VMs may use VLAN virtual interfaces.

2. Production Presentation Zone VMs may be configured to reboot, mount their file systems, and start web applications.

3. Production Presentation Zone virtualization servers may host only business applications that process PHI/PII or those that do not process PHI/PII, but not both.

CMS TRA –Virtualization Supplement 14

Restricted Distribution – For Official Use Only

3.3.2.2.4 Management Zone Rules

The following implementation rules apply to UNIX virtualization servers and in Production environments and their Management Zones.

1. Production Management Zone virtualization servers must not oversubscribe pooled resources.

2. Production Management Zone VMs must use physical network interfaces.

3. Production Management Zone VMs may not use IP multipathing (IPMP).

4. Production Management Zone VMs must not have direct access to PHI or PII.

3.3.3 UNIX Virtualization Security Rules

The following UNIX-specific security rules were derived from security rules listed in the CMS

TRA and CMS TRA – Security Services Supplement. The general rules in subsection 3.3.3.1 apply to all environments and zones within that environment. In addition, zone-specific rules are provided for each environment and each zone within that environment as applicable.

3.3.3.1 General UNIX Virtualization Security Rules

The following security rules apply to UNIX virtualization servers and VMs in all environments and zones.

1. Management interfaces must be IP exclusive, i.e., use a physical network interface or

VLAN virtual interfaces.

2. Management traffic must only use management interfaces and VLANs.

3. Management traffic for VMs on the same virtualization server may share the same

VLAN.

4. Management traffic for virtualization supervisors must use a separate, dedicated VLAN.

5. The management interface is the sole access for Global Containers.

6. The management interface is the sole management access to VMs.

7. All management traffic must be encrypted and must only use Secure Shell (SSH) version 2 or Secure Sockets Layer (SSL)/Transport Layer Security (TLS), not telnet or

HyperText Transport Protocol.

8. All management traffic must originate or terminate in the Management Zone.

9. Virtualization controllers and VMs must have individual access rules and roles for authentication through a Role Based Security System (RBS).

10. All accesses and configurations must be in accordance with the CMS ARS guidance.

11. Virtualization controllers must not have direct root access VMs.

12. Management accounts with root privileges in VMs must not have root privileges in the virtualization controllers.

CMS TRA –Virtualization Supplement 15

Restricted Distribution – For Official Use Only

13. Production environment virtualization server NICs must be Media Access Control

(MAC) address locked to prevent spoofing.

14. Production environment containers:

a. Must use RBS Authentication, Authorization, and Accounting (AAA) functions provided by a common AAA server to ensure that management and user roles are common throughout a zone and to prevent accidental or unauthorized access.

b. User groups and user names must be unique across all file systems to prevent inter-container access.

15. Crash dumps residing on a common log server must be cryptographically protected to prevent accidental destruction or unauthorized access.

16. Production environment virtualization servers and VMs must use a common, authenticated time source to ensure the correlation of crash dumps and audit logs between containers, virtualization servers, and zones.

17. When mandated by CMS ARS requirements on the resident application, each virtualization server or VM will implement a Host-Based Security System (HBSS).

18. All virtualization servers and management interfaces to VMs will implement an HBSS configured to allow communication only with authorized management servers.

19. Virtualization servers, when required, and management interfaces to VMs must encrypt all traffic using Federal Information Processing Standards (FIPS) Publication (PUB)

140-2 certified cryptographic algorithms.

3.3.3.2 Zone-Specific UNIX Virtualization Security Rules

The following security rules apply to UNIX virtualization servers and VMs in all environments and their zones. The general rules in subsection 3.3.3.1 apply to all zones within the environment.

3.3.3.2.1 Data Zone Rules

The following security rules apply to UNIX virtualization servers and VMs in all environments and their Data Zones.

1. Production Data Zone virtualization servers must encrypt all traffic to and from networked file systems using Kerberos or LDAP with FIPS PUB 140-2 certified cryptographic algorithms.

2. User groups and names must be unique to each virtualization server and VM in a

Data Zone.

3. When required by CMS ARS on the resident application, each VM will implement an

HBSS configured to allow communication only with authorized Application Zone VMs over approved ports, protocols, and services.

CMS TRA –Virtualization Supplement 16

Restricted Distribution – For Official Use Only

3.3.3.2.2 Application Zone Rules

The following security rules apply to UNIX virtualization servers and VMs in all environments and their Application Zones.

1. User groups and names must be unique to each virtualization server and VM in an

Application Zone.

2. When mandated by CMS ARS requirements on the resident application, each VM will implement an HBSS configured to allow communication only with authorized

Presentation and Data Zone containers over approved ports, protocols, and services.

3.3.3.2.3 Presentation Zone Rules

The following security rules apply to UNIX virtualization servers and VMs in all environments and their Presentation Zones.

1. User groups and names must be unique to each virtualization server and VM in a

Presentation Zone.

2. When mandated by CMS ARS requirements on the resident application, each VM will implement an HBSS configured to allow communication only with authorized

Application-to-Zone containers over approved ports, protocols, and services.

3. Production Presentation Zone virtualization servers must authenticate file systems before remount after VM crash.

4. When Production environment Presentation Zone VMs reside on the same virtualization server:

a. A common AAA server must provide RBS AAA functions to ensure that management and user roles are unique for each container.

b. User groups and user names must be unique across all file systems to prevent inter-container access.

3.3.3.2.4 Management Zone Rules

The following security rules apply to UNIX virtualization servers and VMs in all environments and their Management Zones.

1. Production Management Zone virtualization servers must encrypt all traffic using FIPS

PUB 140-2 certified cryptographic algorithms.

2. User groups and names must be unique to each virtualization server and VM in a

Management Zone.

3. Each VM will implement an HBSS configured to allow communication only with authorized Data, Application, or Presentation Zone containers over approved ports, protocols, and services.

CMS TRA –Virtualization Supplement 17

Restricted Distribution – For Official Use Only

4. Windows Virtualization

The strategy for Windows Server Consolidation and Virtualization is a highly centralized and highly consolidated environment that uses server virtualization technology. This strategy will achieve a large reduction in infrastructure and support costs. Figure 2 provides an overview of the basic components that comprise Windows server virtualization solutions.

Figure 2. Windows Server Virtualization

4.1 Windows Virtualization Business Rules (WV)

In addition to the Server Virtualization Business rules listed in subsection 2.4, CMS derived the following Windows-specific business rules from the CMS TRA Business Rules and the

CMS TRA – Distributed Systems Platform Supplement.

4.1.1 Business Rule WV-1

Root access to the servers will not be granted to the developers or users.

4.1.2 Business Rule WV-2

All users must adhere to CMS Security policies.

Management

Security

Prod Zone-1

Prod Zone-2

P ro c e s s o r B la d e V irtu a liz a tio n

S e rv e r

P ro c e s s o r B la d e V irtu a liz a rv e r

P ro c e s s o r B la d e

V irtu a liz a rv e r

Controller Blade

Virtual Switch

Centers for Medicare & Medicaid Services Windows Virtualization

CMS TRA –Virtualization Supplement 18

Restricted Distribution – For Official Use Only

4.1.3 Business Rule WV-3

The specific environment’s role-based access criteria will control all access rights.

4.1.4 Business Rule WV-4

All access will be through CMS-approved authentication methods.

4.1.5 Business Rule WV-5

The business application adheres to the CMS TRA guidance.

4.1.6 Business Rule WV-6

Legacy 2 zone architectures will implement collapsed Application and Presentation Zones. The combined Implementation and Security Rules in subsection 4.3 will apply to this collapsed zone architecture.

4.1.7 Business Rule WV-7

CMS standard interfaces shall be employed between Application and Data Zones.

4.1.8 Business Rule WV-8

Microsoft virtualization only applies to the use of resources controlled by Windows virtualization servers (i.e., not routers).

4.2 Windows Virtualization-Specific Terminology

Table 5 defines the following terms that are specific to virtualization of CMS Windows server implementations.

Table 5. Windows Virtualization-Specific Terms

Term Definition cBlade An Egenera blade responsible for PAN (Processor Area Network) management and external communications’ pBlade An Egenera blade that provides processor and memory to VMs.

Egenera Blade A diskless CPU and memory on a board, managed by a PAN manager.

Egenera BladeFrame A rack, under the control of a PAN manager, supporting Egenera Blades interconnected by a high-speed switching fabric, access to Storage Area Networks (SAN) and Network Area Storage (NAS) storage, and network interfaces.

VMWare Datastores Virtual representations of combinations of underlying physical storage resources.

CMS TRA –Virtualization Supplement 19

Restricted Distribution – For Official Use Only

Term Definition

Network Attached Storage (NAS)

Network Attached Storage describes a complete storage system that is attached to a traditional data network. NAS consists of one or more file servers and provides data access to heterogeneous network clients (e.g., Windows, UNIX/Linux).

Storage Area Network

(SAN)

A Storage Area Network is a network specifically dedicated to the task of transporting data for storage and retrieval. The SAN provides an architecture for attaching remote computer storage devices to servers. In this architecture the devices appear as locally attached to the operating system. The remote system accesses storage as though it were a local disk. A SAN can provide better performance and a larger range of configuration options.

Transparent Page Sharing

A technique for sharing memory resources. Memory pages that are identical in two or more virtual machines are stored once in the host system’s RAM. Each of the virtual machines has read-only access.

VMWare Host The virtual representation of the computing and memory resources of a physical machine running ESX Server.

VMWare Cluster The grouping of two or more physical machines (e.g., Egenera blades) to work and be managed together as a whole.

VMware Virtual Machine File System

(VMFS)

A cluster file system for VMWare virtual machines.

Processor Area Network (PAN)

Used by Egenera, a PAN is a set of resource pools representing the four types of resources running on a blade server (see subsection 2.2).

PAN Manager Egenera blade server manager.

4.3 Windows Server Implementation and Security

This subsection describes the design and implementation rules to ensure implementation and security consistency with CMS TRA guidance.

4.3.1 Windows Virtualization Implementation Rules

CMS derived the following Windows-specific implementation rules from the business rules listed in subsections 2.3 and 2.5.1. The general rules in subsection 4.3.1.1 apply to all environments and zones within an environment. In addition, this subsection provides rules for each environment and each zone within an environment when required.

4.3.1.1 General Windows Virtualization Implementation Rules

The following implementation rules apply to Windows virtualization servers and VMs running on blade servers in all environments and zones.

1. All management access to VMs (e.g., administrator login) must be limited to a single physical management interface, or management interface pool, that is not accessible by user processes in a VM.

2. All management access to virtualization servers must be from the Management Zone via a CMS-approved secure access method.

CMS TRA –Virtualization Supplement 20

Restricted Distribution – For Official Use Only

3. All management access to the Blade server manager must be from the Management Zone via a CMS-approved secure access method.

4. Management accounts for VM management access must be different from those used to access VMs and Blade server managers.

5. Management accounts for Blade server manager access must be different from those used to access VI Clients and VMs.

6. IP addresses of management interfaces on cBlades must be in a subnet on the

Management VLAN.

7. IP addresses of management interfaces for virtualization servers must be in a subnet on the Management VLAN.

8. Management interfaces serving the Security segment must be physically separate from those serving all other Management segments (e.g., Backup segments).

9. Virtualization servers in the Management Zone must not oversubscribe pooled resources

(e.g., memory).

10. Disk quotas must be enacted on a per-user and group basis to avoid filling file systems.

11. Source code within a VMFS must not reside on the same file systems as test data.

4.3.1.1.1 Non-Production Environment Windows Virtualization Implementation Rules

The following implementation requirements apply to Windows virtualization servers and VMs using virtualization server running on Blade servers in Non-Production environments, i.e., Test, Development, and Implementation environments, and their zones. The general rules in subsection 4.3.1.1 apply to all zones within the environments. In addition, this subsection provides rules for each zone within the environment when required.

1. When multiple development projects share the same virtualization server, each project must reside in a different VM.

2. When multiple development projects share the same virtualization server, VMs may not oversubscribe pooled resources.

4.3.1.1.1.1 Data Zone Rules

The following implementation rules apply to Windows virtualization servers and VMs using virtualization server running on Blade servers in Non-Production environments and their Data

Zones.

1. Non-Production Data Zone application servers must reside in unique VMs.

2. Non-Production Data Zone VMs must not oversubscribe pooled resources.

3. Non-Production Data Zone VMs must use VLAN virtual network interfaces mapped to individual or pooled NICs.

CMS TRA –Virtualization Supplement 21

Restricted Distribution – For Official Use Only

4.3.1.1.1.2 Application and Presentation Zone Rules

The following implementation rules apply to Windows virtualization servers and VMs using

Virtualization server running on Blade servers in Non-Production environments and their

Application and Presentation Zones.

1. Non-Production Application and Presentation Zone application servers may share VMs.

2. Non-Production Application and Presentation Zone VMs may use shared VLAN virtual network interfaces.

3. Non-Production Application and Presentation Zone VMs may oversubscribe pooled resources.

4.3.1.1.1.3 Management Zone Rules

The following implementation rules apply to Windows virtualization servers and VMs using virtualization server running on Blade servers in Non-Production environments and their

Management Zones.

1. Non-Production Management Zone application servers must reside in unique VMs.

2. Non-Production Management Zone VMs residing on the same virtualization server must not oversubscribe pooled resources.

3. Non-Production Management Zone VMs must use VLAN physical network interfaces or interface pools.

4.3.1.2 Production Environment Windows Virtualization Implementation Rules

The following implementation rules apply to Windows virtualization servers and VMs using virtualization server running on Blade servers in Production environments and their VMs. The general rules in subsection 4.3.1.1 apply to all zones within the environment. In addition, this subsection provides rules for each zone within the environment when required.

1. Production environment VMs and resource pools must not be shared between zones.

2. Production environment application servers must reside in unique VMs.

3. All Production environment VMs must use VLAN-dedicated physical or pooled network interfaces.

4. Management interfaces serving the Security segment must be physically separate from those serving all other Management segments, e.g., Backup segments.

5. Production environment VMs running on the same virtualization server must not oversubscribe any pooled resources.

6. Multiple…