HDFS_Salient_Characteristics.docx

Highly Distributed File System (HDFS) SALIENT CHARACTERISTICS

· Commercial Off-The-Shelf Software (COTS) that can simultaneously distribute data, queries, statistical, and other advanced procedures across multiple servers (nodes) for processing. This is commonly known as a Highly Distributed File System or HDFS.

· Out-of-the-box configuration for Single Sign On (SSO) using two-factor authentication using smart cards (known within DoD as the Common Access Card or CAC).

· Include Apache Spark and/or a Data Science toolset (or similar) that can deploy advanced statistical and Natural Language Processing (NLP) across the file system.

· HDFS solution must operate with a high degree of fault tolerance and incrementally scale processing capability with additional nodes to add to the overall capacity of the entire system.

· Physical HDFS deployment onsite within a DISA-hosted Data Center to align with our IT department’s strategic plan.

Our Planned Environment:

· NIPR: 16 Physical Servers (Small Form Factor Nodes, approximately 64GB RAM, 3.6TB Hard Drive Disk Space, 16 Cores with 32 Processors, 2 sockets per node);

· SIPR: 16 Physical Servers (Small Form Factor Nodes, approximately 64GB RAM, 3.6TB Hard Drive Disk Space, 16 Cores with 32 Processors, 2 sockets per node);

· Red Hat Enterprise Linux (RHEL) Operating System software (version 5.5 or higher) across the entire cluster.

· Contractor shall provide the following software subscriptions (1 Year) for each NIPR and SIPR Install:

· HDFS software subscriptions for 16 nodes (12 Production + 4 Development)

· RStudio Pro (includes Python). Server, unlimited users.

· HDFS Tools such as SPARK, Workbench, etc. for at least 10 users.

· Contractor shall provide the following services for NIPR environment only:

· Installation & configuration of all software including single sign-on using DOD Common Access Card at remote facility (Montgomery Alabama).

· Documentation of installation and configuration.

· Migration of existing project to new environment and tools.

· Documentation of project migration functioning using new tools.

· User and Admin training.

· Contractor shall provide the following services for SIPR environment only:

· Installation & configuration of all software including single sign-on using DOD Common Access Card at remote facility (Montgomery Alabama).

· Documentation of installation and configuration.