J9CXQ_HPC_JA_Final_17Jul15_Redacted.pdf

JUSTIFICATION FOR OTHER THAN FULL AND OPEN COMPETITION

J9CXQ HIGH PERFORMANCE COMPUTING

1. Nature and/or Description of Action

The Defense Threat Reduction Agency (DTRA) contracting activity proposes to procure without using full and open competition for an item peculiar to one manufacturer, in accordance with the policies and procedures prescribed in the Federal Acquisition Regulation (FAR). DTRA requires two (2) Hybrid Graph Analytics High Performance Computing (HPC) Systems, Convey HC2ex, manufactured by Micron Semiconductor Products, Inc.

2. Description of the Supplies/Services Required

DTRA’s Special Programs Office, J9CXQ, is conducting computer intensive Research and Development (R&D) analytics in order to forecast Weapon of Mass Destruction (WMD) events.

An HPC system is required to continue these efforts. An intense J9CXQ review, by onboard HPC Subject Matter Experts (SMEs), of all available HPC systems has been in progress over the last several years. Recently, J9CXQ nominated the most suited HPC system to the DoD High Performance Computing Modernization Program (HPCMP) as part of their annual Dedicated HPC Project Investment (DHPI) process to support unique HPC requirements that cannot be met by use of existing systems at DOD Shared Resource Centers (DSRC). The J9CXQ submission of the Convey HC2ex was selected in a competition held across all of DoD. This system features hybrid-core nodes that support application specific Field Programmable Gate Arrays (FPGA), x86 processors which each have a shared address space with scatter-gather memory, a large shared memory, and a minimum of 15 billion edge traversals per second (GTEPS). The Convey HC2ex system, manufactured exclusively by Micron Semiconductor Products, Inc., is a Hybrid Core Graph Analytics System comprised of two Scalable Units (Server assemblies). The combined units provide a total of sixteen (16) coprocessor-enhanced nodes connected by a high speed, low latency interconnect with a total 12TB memory, and 256TB of hard disk drive (HDD) storage. The hardware, software, set-up and operation guidance, programming consulting, and maintenance warranties/service is required. The period of performance for this requirement will be one (1) base year plus four (4) option years. The estimated total value of this requirement is

3. Statutory Authority

The statutory authority for this non-competitive action is:

10 United States Code (U.S.C.) 2304(c)(1) or 41 U.S.C. 253(c)(1), as implemented by FAR 6.302-1, Only One Responsible Source.

4. Applicability of Authority

Micron Semiconductor Products, Inc. is the only vendor that meets the J9CXQ size, power, and cooling requirements, along with the HPC features of shared memory access among hybrid-core node application specific FPGA and x86 processors. The Convey HC2ex features are critical to J9CXQ because it allows for the HPC to be integrated into J9CXQ’s current computing environment of x86 processors. And, the FPGAs allow for flexibility in future environments that are even more computationally intensive. No other vendor’s product, in today’s market, is able to integrate as quickly into J9CXQ’s data environment, and to ensure uninterrupted future integration as changes occur. The use of any other HPC architecture would negatively impact the R&D timeline due to additional efforts and manpower required to harmonize the systems.

5. Effort to Solicit Potential Sources

The J9CXQ team contacted Cray and Hewlett Packard, in addition to Micron Semiconductor Products, Inc., in soliciting potential sources. Cray and Hewlett Packard cannot fulfill this requirement.

Additionally, a Notice of Intent to Award Sole Source was posted on Federal Business Opportunities for a period of fifteen (15) days. Results of the special notice indicated that there are no other potential sources that can fulfill this requirement.

6. Fair and Reasonable Costs

The anticipated cost/price of this contract action to the Government will be determined to be fair and reasonable, based on a comparison of proposed prices to historical prices paid, whether by the Government or other than the Government, and comparison to prices obtained through market research, for the same or similar items.

7. Market Research

Market research was conducted to determine the availability of acceptable manufacturers and equipment to meet program requirements. It was determined that only Micron’s Convey HC2ex can meet J9CXQ’s environmental and architectural requirements.

There were many vendors and HPC architectures which have been discussed and considered.

These evaluations were conducted primarily by J9CXQ and National Laboratory SMEs. A primary reference used by all the SMEs is the Graph 500, an online HPC guide that provides a rating of supercomputer systems, focused on Data intensive loads. The main performance metric used to rank the supercomputers is GTEPS (10^9 Traversed Edges Per Second). The Graph 500 provided benchmarked performance metrics that were used to evaluate current available HPC systems. This benchmark provides an unclassified relevant proxy for the actual classified workload.

For the final consideration for the DHPI submittal, the field was narrowed down to three HPC vendors: Cray, Hewlett Packard and Micron. The judging criteria was based on a minimum processing 15 GTEPS performance output, using the minimum number of cores, and processors.

The three final candidates were assessed against J9CXQ’s program specific environmental and architectural requirements:

I. Environmental requirements consist of the Data Center’s physical infrastructure required to support an HPC. Specifically: (1) physical space constraints (# of racks); (2) electrical power load (KW); and (3) cooling requirements (BTUs). The data center is currently at 50% of total rack space available (20 of 40 racks). Current power usage is at 25% of capacity (50KW of 200KW). Cooling is at 54% (184K of 360K BTUs) of capacity. The J9CXQ data center has a finite footprint and load capacity. Sufficient reserve must be maintained for future to be determined build-outs. In general, the less space required, while using the least amount of power, and generating the least amount of heat, is desired. As noted in Table 1, the Convey HC2ex minimizes all three categories, and would maintain the Data Center beneath the critical maximum operating parameters.

The HP power requirement would exceed the power available to the Data Center. Both the Cray and HP heat loads would exceed the cooling capacity of the Data Center. (Note that “worst case”, i.e. full power, is used for all calculations, and a safety factor was not applied.)

HPC

Vendor/System

Size (Racks) Total power required (Kilo-Watts)

Total heat output (BTUs) (Assumes 100% Watt:BTU conversion) Micron/HC2ex 2 racks 37 KW 126 K-BTUs Cray/XMP 4 racks 112 KW 382 K-BTUs HP/Apollo 8000 2 racks 168 KW 573 K-BTUs Table 1: Comparison of HPC Environmental Requirements

II. Architectural requirements focus on ease of integration into the current IT data infrastructure, and scalability for future data infrastructure changes. HPCs were originally designed to do large batch standalone processing. The following example identifies the specific use case that influenced J9CXQ’s architectural specifications:

HPCs have been used for gene sequencing. HPCs have reduced the time to sequence genomes from months to 30 minutes. Usually, the HPC resource runs the sequencing overnight to conserve power. Often when those large batch processing jobs go away, those HPC systems get retired because their architecture limits their applicability to other problems.

The J9CXQ use case requires an HPC resource that can do traditional batch processing, but must also be able to concurrently conduct online transactional analysis to feed other processes. The HPC system must interact with other network resources, for example, standard information servers, while executing large data or analytical intensive “supercomputer type” compute jobs.

Architecture comparisons:

a. The Convey Hybrid-core uses heterogeneous architecture so it can be easily customized to optimize performance-critical operations. The Convey coprocessor is based on standard FPGAs coupled with standard multi-core Intel Xeon processors. Physically, each processing element has its own local memory;

logically, memory is globally addressable and coherent, making software development efforts similar to that of a standard x86 environment. Because of this flexible architecture, the Convey HPC will quickly integrate with J9CXQ’s current data architecture and will be able to augment processing within days.

Furthermore, the architecture flexibility and FPGA will promote future development of advanced analytical methodologies in order to more easily extract value from big data. Micron Semiconductor Products, Inc. is the only vendor that features hybrid-core nodes that support application specific Field Programmable Gate Array (FPGA) and x86 processors with scatter-gather memory, which allows this flexible architecture.

b. Both the Cray X-MT and the HP Apollo 8000 solutions do not integrate well with the current J9CXQ data architecture set up. They do have the same capacity to support 13 TB of memory like the Convey HC2ex system. However, the Cray and HP systems must share that memory across several thousand processors.

Those systems have diminished processing capacity because much of the processing capacity has to go to synchronizing the use of the large available memory amongst thousands of processors. Convey HC2ex has a dedicated co-processor that can address the total memory space with a single processor. Thus, the Convey HC2ex system can handle larger and more complicated computing projects more efficiently.

HPC

Vendor/System

Memory Data Storage Architecture

Micron/HC2ex 13TB 4 PB Hybrid-core nodes that support application specific Field Programmable Gate Array (FPGA) and x86 processors with scatter-gather memory.

Cray/X-MT 13TB 4 PB A shared-memory parallel vector processor with bipolar gate-array integrated circuits.

HP/Apollo 8000 13TB 4 PB Integrated cluster of commodity-based servers with performance enhanced with a liquid cooled heat dissipation system.

Table 2: Comparison of HPC Architectural Requirements

8. Other Facts

Additional input concerning Convey HC2ex was obtained from the Virginia Tech Bioinformatics Institute (VBI), and the National Science Foundation (NSF) Strategic technologies for

Cyberinfrastructure program. VBI has supported DTRA in modeling Pandemics/Epidemics for many years. Both VBI and NSF purchased Convey HPC HC-1 hybrid-core node systems. The Convey HPC HC-2ex system is an upgrade of the Convey HPC HC-1 systems. In conversations with VBI, they were pleased with the versatility of Micron’s hybrid HPC system that could be connected to a conventional client/server network without diminishing the intended HPC computational performance. VBI especially appreciated the performance gains through the direct programming of the Micron co-processor.

9. Interested Sources

The companies (listed below) were initially contacted for market research to gather information about which items they could provide and to formulate an estimate of the cost of the listed items:

1. Cray, Inc.

2. Hewlett-Packard Company

Additionally, a Notice of Intent to Award Sole Source was posted on Federal Business Opportunities for a period of fifteen (15) days. DTRA did not receive any interested responses to this posting.

10. Subsequent Actions

J9CXQ plans to use the DHPI process again when the Micron HPC has reached the end of its life cycle. The DHPI process in itself requires a review of all current HPC capabilities that can meet the Program’s technical goals and a detailed justification of a proposed manufacturer before an award is made. This justification is validated by experts in Government and industry. In the event where multiple performers can meet the technical goals, J9CXQ will complete a detailed comparison in order to promote competition and ensure Government requirements are met.