Attachment J.6_SAS Recommended Architecture.pdf

Hardware Recommendations

SAS Grid

Prepared for

US Department of the Treasury

Office of the Comptroller of

Currency (OCC)

September 9, 2011

Hardware Recommendations

Copyright Copyright © 2011, SAS Institute Inc., Cary, NC, USA. All rights reserved. Produced in the United States of

America.

Disclaimer No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, or otherwise, without the prior written permission of the publisher, SAS Institute Inc.

U.S. Government

Restricted

Rights Notice

Use, duplication, or disclosure of this software and related documentation by the U.S. government is subject to the Agreement with SAS Institute and the restrictions set forth in FAR 52.227-19, Commercial

Computer Software-Restricted Rights (June 1987).

Trademarks and Patents SAS Institute Inc., SAS Campus Drive, Cary, North Carolina 27513.

SAS® and all other SAS Institute Inc. product or service names are registered trademarks or trademarks of

SAS Institute Inc. in the USA and other countries. ® indicates USA registration.

Other brand and product names are registered trademarks or trademarks of their respective companies.

Confidentiality This document is provided for informational purposes. SAS' contractual commitments are contained solely in the agreement for professional services executed between SAS and the Client. This document is the confidential and proprietary property of SAS Institute Inc. This document may contain approaches, techniques and other information proprietary to SAS. This document shall not be disclosed in whole or in part to third parties without the prior written consent of SAS.

This hardware estimate herein is a no fee service. Office of the Comptroller of the Currency acknowledges that it is responsible for all costs associated with procuring any recommended hardware.

This hardware estimate does not imply nor guarantee any performance measures.

This hardware estimate does not address any existing performance issues and is not intended to correct any existing performance issues.

This SAS hardware estimate is based on workload descriptions provided by Office of the Comptroller of the

Currency. The provided workload descriptions are assumed to be consistent with intended ordinary and typical uses of the software for the types of sessions characterized. At its core, SAS is a fourth generation programming language and allows customers the flexibility and power to use the software in novel ways that exceed the original intended ordinary and typical use of the software. Such use may result in workloads in excess of those described in this recommendation and may require additional resources; Office of the

Comptroller of the Currency acknowledges that changes in hardware may result in SAS software upgrade license fees owed to SAS.

S A S G R I D H A R D W A R E R E C O M M E N D A T I O N S

Contents

1 Introduction

1.1 Background

1.2 Document Purpose

1.3 Proposed Software

2 Hardware Estimates

2.1 Critical Considerations

2.2 Summary of System-Wide Resource Totals

2.3 Hardware Estimate

2.3.1 Production Environment

2.3.2 Test Environment

2.3.3 Storage Environments

2.3.4 Assumptions

3 Infrastructure Recommendations

3.1 Main Grid Architecture Recommendation

3.2 Test Grid Architecture Recommendation

3.3 Grid I/O Configuration Recommendation

4 SAS Product Allocation Model

1 Introduction

1.1 Background

The Office of the Comptroller of the Currency (OCC) has been a SAS customer for over 15 years. SAS supports multiple business units, and departments in the US Treasury and many use SAS extensively.

From September to November of 2010, SAS conducted a Proof of Concept (PoC) for OCC. Some of the OCC’s most challenging and long-running programs were run on SAS Grid architecture against sample data with which the OCC currently works. At the conclusion of the PoC, SAS recommended implementing an architecture that could host multiple users and business units. SAS Grid Computing provides a flexible architecture for sharing and managing computing resources, maximizing their availability and allowing the OCC to move closer to a more commodity-based architecture. This architecture would also allow the OCC to maximize the value of SAS software licensing by ensuring that SAS products are installed and utilized where needed—ensuring optimal return from its SAS investment.

1.2 Document Purpose

This document makes recommendations as to what technologies could be beneficial to the OCC in implementing

SAS Grid architecture. Specifically, it:

Identifies a highly available, redundant, externally hosted solution that ensures high performance even for mobile users and supports all current applications without unwanted data or hardware redundancy

Implements a scalable solution to accommodate large and/or unexpected requirements for additional data retention and analysis capacity, as well as additional user and applications.

Utilizes a security strategy that ensures that sensitive data (including PII) is adequately protected and that all data is backed up.

1.3 Proposed Software

Proposed software:

SAS® Grid Manager

SAS® Analytics Pro (Base SAS®, SAS/STAT®, and SAS/GRAPH®)

SAS/CONNECT®

SAS/ACCESS® engine 1 (PC File Format)

SAS/ACCESS® engine 2 (ODBC)

SAS/ACCESS® engine 3 (TBD depending on whether solution providers propose a proprietary data store)

SAS/ACCESS® engine 4 (TBD depending on whether solution providers propose a proprietary data store)

SAS® Integration Technologies

SAS/SHARE®

SAS/IntrNet® (on 2 grid nodes)

SAS/ETS®

The applications identified below were considered to be part of SAS’ recommended architecture for OCC. While these products are recommended by SAS, they are optional in the opinion of the OCC. The OCC is willing to consider either these SAS products or any other commercially available product that can provide the analogous capabilities requested by the OCC’s Performance and Technical Requirements. Thus if another product can meet the OCC’s needs—as described in its requirements document—the offeror may propose that product in lieu of any of the products listed in this table. As stated in the OCC’s RFP, all departures from the SAS recommended architecture must be explained and justified in the offeror’s proposal.

SAS® Enterprise Data Integration Server (on one grid node)

SAS/SECURE™

EEDI includes two access engines, SAS Integration Technologies, SAS/CONNECT, and Base SAS.

2 Hardware Estimates

2.1 Critical Considerations

SAS is the primary technology used by OCC staff to manipulate and analyze data.

OCC staff will continue to maintain desk-side analytical applications (both SAS and non-SAS) for use when disconnected from the network and for SAS/CONNECT sessions to the solution.

All functions and components of the solution that require high input/output (I/O) support, such as SAS, will be physical to the degree necessary to accommodate a minimum I/O throughput of 75 MB per second/per core on all 60 cores even if part of the computing architecture becomes inoperable.

OCC is seeking a cost-effective increase in the speed of processing analytic and ETL programs.

Critical considerations from the SAS Enterprise Excellence Center (EEC):

The estimated resources below are a combination of Production, Development, Staging, Collaboration, and User requirements. Changes to the requirements of the non-Production environments may affect the resources required.

2.2 Summary of System-Wide Resource Totals

Phase 1 Totals for System-Wide Resources

*Total SAS users registered in metadata? 48

*Total concurrent SAS sessions? 40

All Phases Totals for System-Wide Resources

*Total SAS users registered in metadata? 93

*Total concurrent SAS sessions? 60

Session Type – Phase 1 *Concurrent Sessions *Average Input Data Volume per

Session (specify MB / GB / TB)

Data Preparation / ETL / DI / EDI

Base SAS related to data preparation / ETL 40 Average is 338 GB

30 @ 600 GB

5 @ 100 GB

5 @ 50 GB

SAS Analytics

Basic Statistics

(Display Manager and/or Enterprise Guide)

40 338 GB

Advanced Analytics

(Display Manager and/or Enterprise Guide)

32 338 GB

Session Type – All Phases *Concurrent Sessions *Average Input Data Volume per

Session (specify MB / GB / TB)

Base SAS related to data preparation / ETL

30 @ 600 GB, 15 @ 100 GB, 15 @ 50 GB

60 Average is 338 GB

30 @ 600 GB

15 @ 100 GB

15 @ 50 GB

SAS Analytics

Basic Statistics

(Display Manager and/or Enterprise Guide)

60 338 GB

Advanced Analytics

(Display Manager and/or Enterprise Guide)

48 338 GB

2.3 Hardware Estimate

Based on the above listed estimate of workloads, this solution should be deployed in an environment such as described in the following sub-sections.

2.3.1 Production Environment

In phase 1, there will be 5 grid nodes total for a total of 60 cores, each configured as follows. In phase 3, an additional node will be required.

Tier Cores / RAM

SAS Grid Compute Server

(Foundation, Workspace Server, Stored Process Server, and related server components)

4 * 12 cores

4 * 48 GB RAM

Tier Cores / RAM

SAS Metadata Server/SAS Grid Control Server/SAS Grid

Compute Server

(Foundation, Workspace Server, Stored Process Server, and related server components)

12 cores with 48GB RAM

This sizing was based on an Intel six-core Xeon X5690 processor with a clock speed of 3.46 GHz running the

Red Hat Enterprise Linux 64-bit operating system.

The estimate is an order of magnitude estimate of the required hardware based upon the workloads as presented above. Data storage and overall I/O throughput of the environment may play an important role in the success of this project and should be carefully considered before beginning implementation.

Phase 1 requires 5 Grid Nodes for a total of 60 cores. This includes failover capability for the Grid Control

Server, SAS Metadata Server and SAS Grid Compute Servers. The SAS Enterprise Data Integration Server software will be dedicated to one of the grid nodes. SAS/IntrNet will be confined to two grid nodes.

The sustained sequential I/O throughput must be at least 75 MB/second per core. Peak rates can be higher. The total sustained sequential I/O requirement for Phase 1 is 4,500 MB/second. The total sustained sequential I/O requirement is expected to grow from 4,500 to 5,400 MB/second by the end of Phase 3.

Centralized storage is mandatory for all Grid Nodes and the Grid Control Server. Examples include Network

Attached Storage (NAS), Storage Attached Network (SAN), and Cluster File Systems (CFS) such as GPFS.

2.3.2 Test Environment

In all phases, there will be 2 grid nodes total for a total of 24 cores, each configured as follows.

Tier Cores / RAM

SAS Grid Compute Server

(Foundation, Workspace Server, Stored Process Server, and related server components)

1 * 12 cores

1 * 48 GB RAM

SAS Metadata Server/SAS Grid Control Server/SAS Grid

Compute Server

(Foundation, Workspace Server, Stored Process Server, and related server components)

12 cores with 48GB RAM

This sizing was based on an Intel six-core Xeon X5690 processor with a clock speed of 3.46 GHz running the

Red Hat Enterprise Linux 64-bit operating system.

The estimate is an order of magnitude estimate of the required hardware based upon the workloads as presented above. Data storage and overall I/O throughput of the environment may play an important role in the success of this project and should be carefully considered before beginning implementation.

Phase 1 requires 2 Grid Nodes for a total of 24 cores. This includes failover capability for the Grid Control

Server, SAS Metadata Server and SAS Grid Compute Servers. The SAS Enterprise Data Integration Server software will be dedicated to one of the grid nodes. SAS/IntrNet will be confined to two grid nodes.

The sustained sequential I/O throughput must be at least 75 MB/second per core. Peak rates can be higher. The total sustained sequential I/O requirement for Phase 1 is 4,500 MB/second. The total sustained sequential I/O requirement is expected to grow from 4,500 to 5,400 MB/second by the end of Phase 3.

Centralized storage is mandatory for all Grid Nodes and the Grid Control Server. Examples include Network

Attached Storage (NAS), Storage Attached Network (SAN), and Cluster File Systems (CFS) such as GPFS.

2.3.3 Storage Environments

The OCC plans to implement five logical environments: Production, Development, Staging, Collaboration, and

User. OCC does not envision a traditional physical architecture with completely segregated environments. The storage arrays of each environment must be logically separate; however, all other architectural components, such as the computing and networking resources, can be shared. Data analysis will be conducted in all five environments. The data storage requirements for these five logical environments are:

At implementation; the solution shall provide a minimum storage capacity for Phase 1 equal to:

60 TB of data in the collaboration environment

27 TB of data in the development environment

12 TB of data in the staging environment

12 TB of data in the production environment

102 TB of data in the user environment

73 TB of data in the archive tiers

The solution shall provide a minimum of 62 TB of sort space for SAS WORK to support the Performance

Requirements in Phase 1. The solution shall provide for a minimum of 20% year over year increase to the

SAS WORK space for a cumulative five-year total of 155 TB.

The solution shall provide additional storage capacity for Phase 3 equal to:

5 TB of data in the collaboration environment

1.7 TB of data in the development environment

1.1 TB of data in the staging environment

1.1 TB of data in the production environment

10 TB of data in the user environment

4 TB of data in the archive tiers

The solution shall provide an additional 3 TB of storage for SAS WORK to support the Performance

Requirements in Phase 3. The solution shall provide for a minimum of 20% year over year increase to the

SAS Work space.

The solution shall provide for the annual growth of Phase 1 data equal to:

10% of the data in the collaboration environment

10% of the data in the user environment

10% of the data in the archive tiers

8 TB of data in the development environment

4 TB of data in the staging environment

4 TB of data in the production environment

The solution shall provide for the annual growth of Phase 3 data equal to:

10% of the data in the collaboration environment

10% of the data in the user environment

10% of the data in the archive tiers

0.5 TB of data in the development environment

0.3 TB of data in the staging environment

0.3 TB of data in the production environment

2.3.4 Assumptions

SAS is a very powerful tool and therefore can require significant resources. The following assumptions were used to arrive at the results and estimation of needed resources:

The estimation of resources herein is a no-fee service. This estimate is not a performance benchmark, and does not imply nor guarantee any performance measures. Any use or reliance on this estimate is at customer’s risk. SAS disclaims any liability with regards to (i) the hardware sizing estimate provided herein;

or (ii) Office of the Comptroller of the Currency’s reliance on the estimate. Office of the Comptroller of the

Currency acknowledges that it is responsible for all costs associated with procuring any hardware.

This estimate does not address any existing performance issues and is not intended to correct any existing performance issues

This SAS hardware estimate is based on the workload descriptions provided by Office of the Comptroller of the Currency. The provided workload descriptions are assumed to be consistent with the intended ordinary and typical uses of the software for the types of sessions characterized. At its core, SAS is a fourth generation programming language and allows customers the flexibility and power to use the software in novel ways that exceed the original intended ordinary and typical use of the software. Such use may result in workloads in excess of those described in the estimate and may require additional resources

This sizing is based on SAS 9.1.3 and SAS 9.2 best practices, guidelines, and benchmarks

This sizing estimate is based on a combination of guidelines provided by SAS R&D, SAS Product

Management, test data, and field experience

The estimates included herein assume that the servers are to be used for the SAS Workloads as described above. Addition of other third-party applications to the same server as SAS is generally discouraged; the estimated resources are for deployment of SAS applications only.

The sizing estimate is based on the SAS Workloads as described above. Changes to the workload (in either number of sessions or data volumes), operating system, or preferred vendor or chipset may render this sizing as void.

Deployments of some SAS solutions include 3rd party products/dependencies. This estimate does not address necessary resources for any 3rd party products. Resource requirement should be obtained from the necessary

3rd party providers.

2.3.4.1 Hardware and Operating System Assumptions

Core counts are guidelines only. Complexity, user loads, and data models can require additional system resources

2.3.4.2 Storage Assumptions

SAS tends to be I/O intensive. SAS may utilize the SAS Work disk location to temporarily house results as it calculates statistics or performs certain functions. To maintain adequate performance it is essential that I/O be as fast as possible between the SAS Compute Server and the SAS Work disk(s). It is also critical that sufficient I/O throughput be allocated so as to maintain the cores of the Compute Server in a ―busy‖ state.

SAS sessions can consume a wide range of I/O bandwidth across its associated file systems depending on the type of processing that is being performed. For example, ETL-related processing can consume up to 100+MB per second per core while processes concerned with analytics may take as little as 15MB per second per core.

For sizing, it is typically assumed that the average SAS session will require approximately 50MB per second per core to maintain the process in a ―busy‖ state. ETL jobs with high I/O will, on average, consume 50MB –

75MB per second per core throughput. To arrive at the total sustained I/O that is required for the SAS solution, it is therefore a matter of calculating the total number of cores configured in the compute tier times the average sustained I/O required per core.

Storage providers should consider the peak I/O throughput requirements identified by the OCC to ensure that their proposed storage environment can provide the level of I/O required. A significant percentage of

―performance problems‖ reported to SAS Technical Support can be directly attributed to insufficient levels of

I/O throughput.

SAS’ Red Hat Linux recommendation is derived from multiple tests that reflect optimized I/O throughput.

2.3.4.3 Grid Assumptions

Shared centralized storage is mandatory for all Grid Nodes and the Grid Control Server. Examples include

Network Attached Storage (NAS), Storage Attached Network (SAN), and Cluster File Systems (CFS). For further details about centralized storage, SAS WORK, and SAS Grid, please see the link at:

http://support.sas.com/rnd/scalability/grid/Shared_FileSystem_GRID.pdf.

SAS WORK (and UTILLOC) must be available to all grid nodes. In some cases, SAS WORK should be localized to an individual node and should not be shared.

Please refer to page 10, section 2.3, of the ―Best Practices for Data Sharing in a Grid Distributed SAS®

Environment‖ http://support.sas.com/rnd/scalability/grid/Shared_FileSystem_GRID.pdf.

Homogenous systems are preferred for all grid nodes due to potential data access inefficiencies. SAS Data

Integration Studio requires homogenous grid environments. Data Integration Studio requires that the grid be either all UNIX or all Windows as it does not use universal names in the library names or file names that it generates.

http://support.sas.com/rnd/scalability/grid/Shared_FileSystem_GRID.pdf

3 Infrastructure Recommendations

Consideration for five environments, massive data and storage requirements, high availability of Grid resources, and optimized I/O throughput are the key elements to SAS’ proposed architecture. The other key element is flexibility. SAS’ recommendations provide a solid starting point for OCC while allowing for adjustments to environment sizes, test capabilities, and growth of computing resources. It should be expected that adjustments will need to be made as usage patterns and live feedback are incorporated. There also could be changes in technology that allow for additional capabilities in the future, and those advancements would also be incorporated on an ongoing basis for OCC.

3.1 Main Grid Architecture Recommendation

The following diagram depicts how OCC’s software can be deployed for Phase 1 in a Grid that meets the sizing recommendation. All the software to schedule and deploy jobs on the Grid (Platform Grid Management Services, Platform Process Manager) will run on the Grid Control Server. SAS Grid Manager, Platform LSF and

SAS/CONNECT—the three components that enable communication between nodes on the Grid—will run on all of the Grid nodes. The SAS components that are used to process and analyze the data will be run as required on the Grid nodes.

Each Grid node will be connected to a shared file system that will store the OCC data, the SAS software binaries, the SAS configuration files and possibly SAS WORK.

Client computers will submit jobs to the Grid using SAS client applications such as SAS Enterprise Guide, SAS

Data Integration Studio, Base SAS, or through the SAS Grid client submission utility (SASGSUB).

Figure 1: Main Grid Architecture Recommendation for Phase 1

3.2 Test Grid Architecture Recommendation

In addition to the main Grid, SAS recommends that another, smaller Grid be used for testing updates/patches to operating system and SAS software before they are made to the main Grid environment. This Test Grid is depicted in the graphic below.

It is composed of two nodes: a Grid controller node and a SAS grid node.

All of the software installed on the Grid should be installed on these machines so patches to the individual products can be tested.

The Test Grid is composed of the same products included in the main compute nodes.

The Test Grid will use the same clients and shared file system that is used for the Main Grid.

Figure 2: Test Grid Architecture Recommendation

3.3 Grid I/O Configuration Recommendation

The performance of your storage subsystem is the most critical component of implementing SAS in a distributed grid environment. Heavy SAS Foundation processing (such as ETL) requires a minimum of 75 MB per second per core of I/O to keep the processors busy.

2 Where EDI is present, only two access engines are needed. (since two are included with the EDI software). If EDI is not proposed as part of the ECM solution, four access engines will be needed.

As explained in ―Best Practices for Data Sharing in a Grid Distributed SAS® Environment‖ at http://support.sas.com/rnd/scalability/grid/Shared_FileSystem_GRID.pdf, the most commonly implemented, and typically the best performing, storage architecture for SAS software in a distributed environment is a Clustered or

Shared File System (CFS). In this architecture, standard storage devices available from various hardware vendors are shared via SAN architecture (for example, connected together with a fiber channel switch). A CFS was used as the storage subsystem for the OCC PoC.

Figure 3: Grid I/O Configuration Recommendation

4 SAS Product Allocation Model

Phase 1 Phase 3

# of total cores 60 72

Production (5 Environments) cores

SAS Grid Manager 60 72

SAS Analytics Pro (Base SAS, Stat, Graph) 60 72

SAS EDI (includes Base SAS, SAS Integration Technologies, SAS/CONNECT, 2 Access engines)

12 12

SAS Integration Technologies 48 60

SAS/CONNECT 48 60

Access engine 1 48 60

Access engine 2 48 60

Access engine 3 60 72

Access engine 4 60 72

SAS/SHARE 60 72

SAS/IntrNet 24 24

SAS/ETS 60 72

SAS/SECURE 60 72

Test cores

SAS Grid Manager 24 24

Analytics Pro (Base SAS, SAS/STAT, SAS/GRAPH) 24 24

SAS EDI (includes Base SAS, SAS Integration Technologies, SAS/CONNECT, 2 Access engines)

12 12

SAS Integration Technologies 12 12

SAS/CONNECT 12 12

Access engine 1 12 12

Access engine 2 12 12

Access engine 3 24 24

Access engine 4 24 24

SAS/SHARE 24 24

SAS/IntrNet 24 24

SAS/ETS 24 24

SAS/SECURE 24 24