SCINet2_-_Statement_of_Objectives_-_DRAFT.pdf

31-May-2019 - 1 -

FINAL REVISION 31-May-2019

USDA Agricultural Research Service Scientific Computing Infrastructure Network 2 (SCINet 2)

STATEMENT OF OBJECTIVES (SOO)

I. Purpose:

The Agricultural Research Service (ARS) of the US Department of Agriculture seeks to evolve the existing Scientific Computing Infrastructure Network (SCINet) into an Agency-wide, spatially distributed (across 86 geographical locations) computing, high-speed network, and storage infrastructure, collectively called SCINet 2, that provides services for its scientists and collaborators to help answer critical research questions.

SCINet 2 shifts from an original SCINet resource-intensive architecture into a more easily deployable, standardized, and cost-efficient solution that improves access to scientific computing and delivers services to ARS user communities involved in genomics, hyperspectral imaging, modeling, high throughput phenotyping, sensory outputs, geographic information system (GIS), artificial intelligence (AI), and legacy data originating in natural resources, agricultural production, land management, and allied domains.

II. Scope:

The proposed solution shall include scientific computing infrastructure, computing platforms, and services and be supportive of the evolutionary requirements of ARS research programs and aligned with the objectives and operating constraints outlined in this document.

III. Background:

ARS is the U.S. Department of Agriculture’s (USDA) chief in-house scientific research agency. More than 2,000 ARS scientists and post docs at 86 geographical locations spatially distributed throughout the US (metropolitan, rural, and several overseas) with a mission to discover and apply solutions to agricultural problems that affect Americans every day. Each research location is locally managed and highly diverse in terms of number of scientists, and their range of software and hardware computational capacities and needs.

ARS research focuses on national and regional agricultural priorities affecting the food we eat, the water we drink, and the air we breathe. ARS organizes its 690 research programs into four broad national portfolios, Nutrition, Food Safety/Quality, Animal Productions and Protection, Crop Production and Protection, and Natural Resources and

31-May-2019 - 2 -

Sustainable Agricultural Systems (see https://www.ars.usda.gov/research/programs for details about each portfolio and types of research that we do.) Projects have diverse computational needs including genomic analyses, image processing, long term series, range management, harmonizing historic data with imagery and soils data for management decisions, and snowmelt model prediction, to name a few.

The current generation SCINet consists of a high-performance computing (HPC) cluster hosted in Ames, Iowa, that connects six (6) ARS locations via a private point-to-point high-speed network architecture. Ames, IA hosts also parallel storage and backup/archive systems. See Appendix #1 for detailed description of the current generation SCINet.

IV. Target Vision for SCINet 2:

SCINet 2 solution is scalable, modular, uses state-of-the-art offerings available from the commercial sector, and leverages to the extent possible scientific computing infrastructure available to ARS in academia and federal agencies. The target solution integrates the current generation SCINet’s HPC cluster, data transport, and associated support skills.

V. Objectives:

1. Design and document system architecture.

a. Scalable, flexible, and standardized design with predictable incremental cost structure for adding and maintaining user and provider nodes https://www.ars.usda.gov/research/programs

31-May-2019 - 3 -

b. Effective HPC management and operations mechanisms

c. HPC usage control by logical order

a) equal user access to standard provider nodes

b) priority user access to priority provider nodes (nodes acquired with funding from specific users)

d. Applications management

e. Effective storage management mechanisms

f. Base and surge capacity planning; build and cost management

g. User and provider nodes (use cases) development, integration, and testing

h. Compliance and security in accordance with federal government standards

i. Interconnection with the existing ARS administrative network, ARSNet

j. Meeting economically a wide range of scientific computing needs, from large data sets requiring multi-hour processing time to small data sets requiring less than an hour of processing time

2. Integrate the existing SCINet infrastructure.

a. Provide ongoing support and management for the HPC cluster and storage at

Ames, IA

b. Provide ongoing support and management for data transfer infrastructure

c. Provide ongoing support and management for the network backbone

3. Design, document, and execute processes for integration with services from new collaborators and commercial partners

4. Provide full range (applications, troubleshooting, training, community participation, etc.) of end-user support services for SCINet 2 users

5. Develop deployment phases, build, and operationalize SCINet 2 components

6. Provide management and monitoring for all services

7. Develop high-level project management plan

8. Develop project management plans for specific phases and components

VI. Operating Constraints:

1. Compliance with the Trusted Internet Connections Initiative, ver. 2.0 and later

2. Compliance with FIPS 200, NIST FISMA standards and guidelines, and DISA Security

Technical Implementation Guides

3. Utilization of compute and storage not to exceed 75% of theoretical capacity

4. Network surge transfer rate at minimum 50% more than base transfer rate under average usage load

5. System-wide uptime of 99.9%

6. Inclusion of a blend of high memory, GPU-accelerated, and standard HPC compute nodes

7. Provisioning of multi-OS

31-May-2019 - 4 -

Appendix A

Current SCINet Environment:

1. Private point-to-point network comprised of:

a. Six locations interconnected by Internet2 Advanced Layer 2 Service (AL2S) The current connectivity circuits have “tail circuits” provided by Internet2 members.

i. Beltsville, MD (connected to Ashburn, VA) (Internet2 global access originating in the Beltsville location, Layer3 AL3S, see vii below)

ii. Ames, IA (connected to Chicago, IL)

1. Commercial internet provided by Iowa State Univ at Ames, IA via VLAN

2. Same VLAN utilized for HPC support services (below)

iii. Stoneville, MS (connected to Jackson, MS)

iv. Clay Center, NE (connected to Univ NE, Lincoln)

v. Fort Collins, CO (connected to Colorado State Univ)

vi. Albany, CA (connected to Sunnyvale, CA)

vii. The private network gains access to Internet2 via a 100Gbps Port at

Ashburn, VA

viii. The private network gains access to commercial internet via a 10Gbps service provided by Iowa State University at Ames, IA

ix. Each location hosts a data transfer staging server (Location Data

Transfer node) and firewall to bridge legacy local-area network for one-way data transfers into SCINet border

Present design utilizes full Internet2 AL2S circuit access for any new location.

31-May-2019 - 5 -

Equipment Brand

Type Model Location

Juniper Router MX480 Var

Juniper Firewall SRX1500 Var Brocade Router MLXe8 Var Silicon

Mechanics Data

Transfer Node

R346.v6

SM110153

Var

Silicon Mechanics

PerfSONAR SM91692 Var

Juniper Switches Various models Var

b. Connected to an ARS-owned high-performance compute cluster store (called Ceres)

i. Computational components: 64 regular compute nodes with 20 physical cores at 2.5GHz; 20 regular compute nodes with 36 physical cores at 2.3GHz and AVX-512 allowing increased instructions per cycle; 8 high-memory compute nodes with 40 physical cores at 2.4GHz; and 1 specialized graphical processing (GPU) compute node

ii. Storage capacity: 120TB Tier 1 (node based) scratch storage; 1.3PB Seagate Luster scratch storage; 1.8PB Tier 2 BeeGFS project data storage (not backed up); and 600TB Tier 3 backup and archive storage

iii. HPC administrative support services provided by the Iowa State Univ under a research support agreement with ARS National Programs

31-May-2019 - 6 -

Office (system administration and certain research application support functions)

c. Connected to Amazon Web Services at the National Agricultural Library (NAL) in Beltsville, MD

i. Access via SCINet credentials

ii. User-initiated services via custom service catalog

iii. Monthly arrears invoicing of services used up to a contract cap

31-May-2019 - 7 -

d. Current security model is based upon Bro Intrusion Detection system that monitors network traffic in parallel (at Beltsville ALS3 interface) and also location firewalls at local ARS LAN interface points to limit traffic into the ARS administrative network. There is also a VPN for secure remote access to SCINet services via the commodity internet access point in Ames, IA.

2. Virtual Research Support Core (VRSC)

a. The current-state VRSC is comprised of ARS personnel working as community resources and supplemental contract personnel. The VRSC has three main components: computational support, advanced research IT support and community participation.

i. Informatics support is excluded from scope of this SOO

ii. Advanced research IT support includes, but is not limited to:

1. Advanced Linux/Unix support

2. Advanced high-performance computing system engineering and administration

3. Manage and operate the high-volume parallel file system and associated data transfers

4. Advanced network engineering and operations

5. Advanced cloud computing and orchestration support

6. Scripting and programming system administration tasks

7. Operating research computing environment

iii. Community participation generally occurs in the Basecamp user forum, scheduled workshops and trainings, and within direct user communications

3. Governance

a. SCINet operates at a Low rating under a Federal Information Processing

Standard (FIPS) 199 Standards for Security Categorization of Federal Information.

b. All system security functions and Risks in compliance with the National Institute of Standards and Technology (NIST) Special Publication 800-53, rev 4, as mandated under FIPS 200, Minimum Security Requirements for Federal Information and Information Systems

c. SCINet is subject to Federal Information Technology Acquisition Reform Act (FITARA) and Program Management Improvement Accountability Act (PMIAA) as a federal investment