ERDC HITS-U III Sample Tasks RFI_Posted 2025.04.16.pdf

DRAFT

ERDC DSRC TASK ORDER FOR HITSU-III

1.0 GENERAL INFORMATION

This task order performance work statement (PWS) consists of background, scope, and tasks specific to the ERDC DSRC. Any PWS sections not referred to in this task order PWS should align with the base contract PWS.

1.1 INTRODUCTION:

The Engineer Research and Development Center (ERDC) administers and operates one of the five DoD DSRCs established under the auspices of the DoD HPCMP. The ERDC DSRC is physically located at the Information Technology Laboratory (ITL) in Vicksburg, Mississippi, and provides large-scale HPC resources to serve the capability computing needs of DoD engineers and scientists. The ERDC DSRC’s unique program resources include the Data Analysis and Visualization Center (DAV-C), which provides scientific visualization, conceptual animation, and video production support to DoD HPCMP users. The ERDC DSRC also provides operational and administrative support for HPC and enterprise class machines not operating under HPCMP. These machines and users typically have requirements that do not fit within the HPCMPs provided offerings but have missions that align with the HPCMP’s mission. These systems often interact with HPCMP resources but can also be stand-alone computing systems.

1.2 BACKGROUND:

The U.S. Army Engineer Research and Development Center (ERDC) in Vicksburg, Mississippi, is the premier research and development laboratory complex for the Corps of Engineers. ERDC discovers, develops, and delivers innovative solutions to the nation's toughest challenges in military engineering, installations and operational environments, civil works, geospatial research and engineering, and engineered resilient systems.

Originally established as an Army Supercomputer Center in 1989, ERDC became the first High Performance Computing (HPC) Major Shared Resource Center (MSRC) in 1993 as part of the DoD HPC Modernization Program (HPCMP). The name was changed to the ERDC DoD Supercomputing Resource Center (DSRC) in 2009.

Located in the Information Technology Laboratory (ITL) at ERDC, the ERDC DSRC serves the high performance computing (HPC) needs of engineers and scientists throughout the DoD, providing a complete HPC environment, including hardware, http://itl.erdc.usace.army.mil/ software, data storage, archiving, visualization, training, and expertise in various Computational Technology Areas.

1.3 SCOPE:

The scope of this contracted effort includes: (1) essential ERDC DSRC operations support including system integration and operation, system administration, network administration, cybersecurity/information assurance, facilities support, database administration, storage administration, maintenance management and execution for the facility, hardware, and software; (2) delivery of HPC capabilities at unclassified and classified levels; (3) enterprise and user support requirements, such as HPC Help Desk, ERDC DSRC Help Desk, application software support, outreach, and support for other HPCMP components; (4) program and project management support, such as financial data and reports and process management, procurement support, and inventory support.

Operations support includes all services required for ERDC DSRC routine operations somewhat independent of HPC capabilities being delivered or customers supported.

The ERDC DSRC operates 24 hours/day, 7 days/week, even during federal holidays.

An enterprise HPC Help Desk provides support 5 days per week (Monday-Friday), 12 hours per day (0900-2100 CST), excluding federal holidays, to over 5,000 HPC users and HPCMP staff members (approximately 2,500 considered active users). An enterprise ERDC Help Desk provides support 5 days per week, (Monday-Friday), 8 hours per day (0800-1600 CST), excluding federal holidays, to approximately 50 users.

Cybersecurity is a critical requirement on this contract, where the ERDC DSRC must perform continuous risk assessments and management, as well as sustain the approvals to operate.

This contract will provide services to support the operations and management of the ERDC DSRC. Requirements at the ERDC DSRC include work requested and funded by HPCMP components other than HPCMP Centers, as well as customer-funded efforts outside the HPCMP.

2.0 PERIOD OF PERFORMANCE (POP):

The period of performance for this task order will consist of a five year ordering period.

Tasks will be issued during the five-year ordering period and will specify a period of performance for each task.

3.0 GENERAL SCOPE OF REQUIREMENTS:

3.1 ERDC DSRC Operations Support

This PWS section includes all minimal support required for basic operations of the ERDC DSRC to be ready to deliver HPC capabilities and user support. The requirements are somewhat independent of the number of HPC systems operating and the HPC users being supported. The requirements are inclusive for the ERDC DSRC facility with appropriate operational elements of total electrical power and mechanical cooling capacity (not tied specifically to any HPC systems but representative of overall total capacity), basic DREN, SDREN, and multiple Above Secret networking enclaves with various operational workstations and servers, safety and security programs, and business processes such as procurement, asset/inventory control, change management, and configuration management.

Baseline ERDC DSRC operations support will ensure the following:

• Operation and delivery of HPC capabilities 24 hours/day, 7 days/week, 365 days/year (includes federal holidays, although reduced staff expected during federal holidays).

• Facilities, support infrastructure, and non-HPC systems remain operational and reliable.

• Cybersecurity support is continuous and consistent with effective risk assessment and management so that network enclaves maintain appropriate authorities to operate.

• Hardware and software maintenance agreements are up to date.

• Configurations are managed effectively.

• HPCMP Help Desk support for HPC users.

• ERDC Help Desk support for local and remote users.

3.1.1 Operations Monitoring. The contractor shall provide support to operate and monitor, as directed, all HPCMP and ERDC DSRC systems as well as computing infrastructure assets within the ERDC DSRC. This support includes 100% of all networks (unclassified and classified), data storage, data archives, web services, asset management, and other ERDC DSRC and Enterprise systems. Support also includes the major HPC systems except for the base level Operating System (OS) and management software stack(s) that are provided and maintained by a system vendor or original equipment manufacturer (OEM). Further, it includes cybersecurity and other aspects of operating and maintaining HPC systems as identified by the government.

The contractor will be responsible for any additional emerging requirements including workload management, software virtualization and containers, container orchestrators, and user support, customer support, or both that are placed on the major HPC systems by the Program. The contractor shall develop automated systems to enhance the monitoring of operations, both in scope and response time.

3.1.2 Facilities Engineering Support. The contractor shall provide support related to, but not limited to, minor building alterations, environmental monitoring, electrical power and backup subsystems, and mechanical support systems. Services include planning through engineering designs and assessments, as well as other functions needed to execute facility projects. Environmental monitoring is largely provided via an automated system (EMMS). The ERDC DSRC has been increasing the level of automation in recent years, and the contractor shall assist further development of these automated systems.

3.1.3 Cybersecurity Support for DSRC Operations. The baseline operational DREN, SDREN and multiple Above-Secret network enclaves in the ERDC DSRC require authorization to operate and connect to the wide-area networks. The Contractor shall assist the government cybersecurity staff in attaining and sustaining the appropriate authorizations to operate. Cybersecurity requirements related to DSRC operations are identified in Section 3.7.

3.1.4 System Administration. For baseline ERDC DSRC operations, there are several non-HPC devices such as workstations, servers, and repositories. Baseline operations include a VDI environment at multiple classification levels that is unique to the ERDC DSRC. The Contractor shall provide system administration support in keeping these systems operating reliably and securely.

3.1.5 Network Engineering and Administration. The Contractor shall provide network engineering and system administration for the various network devices including, but not limited to, routers and switches. In addition, in accordance with DoD HPCMP direction and JFHQ-DODIN Comply-to-Connect (C2C) Concept of Operations instructions, the Contractor shall administer and maintain the necessary hardware and software (currently Forescout) at the ERDC DSRC to continue enforcing C2C compliance.

3.1.6 Database Administration. The ERDC DSRC maintains databases for baseline operations support. This includes, but is not limited to, environmental monitoring data, service tickets, maintenance logs, etc. The Contractor shall provide database administration support for these systems.

3.1.7 Software Development. The Contractor shall develop and support execution of specialized software used in ERDC DSRC operations. This includes, but is not limited to, applications to monitor and automate facility environmental controls, monitor and automate center operations, and automate operational and business processes.

3.1.8 Maintenance Support. The Contractor shall provide support for ERDC DSRC hardware and software maintenance agreements related to facility equipment and supporting infrastructure (networks, non-HPC servers, etc.) used in routine operations.

This includes both licenses and regularly used supplies. Note: Portions of the hardware and software maintenance are currently performed via OEM or third-party maintenance personnel under contracts acquired directly by the Government.

3.1.9 Inventory Management Support. The Government is responsible for managing Government Furnished Property (GFP) at the ERDC DSRC; however, the Contractor shall provide support to the ERDC DSRC in identifying and documenting equipment inventory for asset management and IAW local policy.

3.1.10 Configuration and Change Management. The contractor shall manage the ERDC DSRC’s configuration and change management database. The contractor shall provide support to the ERDC DSRC by identifying, maintaining, reporting, and verifying configuration management items. The contractor shall provide support to the ERDC DSRC by identifying, analyzing, documenting, and maintaining change requests IAW local process, procedure, and policy.

3.1.11 Procurement Support. The ERDC DSRC purchases various items needed to support routine operations. This includes, but is not limited to, IT supplies and equipment, facility infrastructure items, HPC infrastructure equipment, hardware maintenance, and software maintenance. The Contractor shall provide procurement support for all ERDC DSRC requirements within scope of this contract.

3.1.12 Continuity of Operations. Technical support is required to reconstitute critical operations, defined by the government, at alternative location(s) in the event of a disaster that causes the shutdown of ERDC DSRC operations at the primary site. The Contractor shall assist in establishing continuity of operations and disaster recovery for such events and shall support any exercises and simulations.

3.1.13 Technical Project Support. The ERDC DSRC executes studies and projects related to, but not limited to, facility upgrades, infrastructure changes, HPC capacity increases, and HPC system installation and integration. The Contractor shall provide technical support to ERDC DSRC studies and projects whether at the local or enterprise level. This support shall include technology planning and assessments and engineering support during project execution.

3.1.14 Storage Administration. Baseline ERDC DSRC operations utilizes several Storage devices such as backup appliance, NFS file servers, ZFS storage appliance, and External storage systems. Baseline operations include an unclassified NFS storage appliance, multiple external Storage systems for VDI environments, and NetBackup Appliance to backup all servers and workstations at multiple classification levels unique to the ERDC DSRC. The Contractor shall provide system administration support to maintain these systems operating reliably and securely.

3.2 Deliver Unclassified HPC Capabilities at the ERDC DSRC

Delivering HPC capabilities includes requirements in direct support of installing, integrating, and operating HPC systems. The baseline scope in this PWS section includes all unclassified HPC systems. Contractor support requirements include the HPC-related infrastructure that ties directly to delivering unclassified HPC capabilities such as data movement, data storage (mid-tier and archive), and user interfaces and portals to unclassified HPC systems. The ERDC DSRC generally hosts three to five large HPC systems as well as a varying number of smaller specialized systems.

Requirements for specialized systems and HPC systems funded outside the HPCMP will vary in level of support from the large HPCMP systems.

3.2.1 Infrastructure Support for Unclassified HPC Systems. The Contractor shall provide technical support to include, but not limited to, system engineering, system integration, system administration, storage administration, and database administration for HPC-related infrastructure directly tied to delivering unclassified HPC capabilities.

This includes, but is not limited to, HPC data movement and storage systems, portals, servers, routers, and switches that have direct relationships or connections to HPC systems. This does not include those systems or devices that are part of the ERDC DSRC baseline operations identified in paragraph 3.1 and associated subparagraphs.

3.2.2 Unclassified HPC System Installation and Integration. Contractor support requirements include preparation of the ERDC DSRC facilities for installing, integrating, accepting, and operating unclassified HPC systems as well as unclassified storage systems. The contractor shall provide engineering support for connecting unclassified HPC systems and unclassified HPC storage systems to the ERDC DSRC infrastructure, both facilities-related and HPC-delivery-related.

3.2.3 Application Management Support for Unclassified HPC. The HPC original OEMs typically provide levels of support, maintenance, and warranties for hardware and software delivered with an operational HPC system. For those software components that are to be installed and modified to support HPCMP workloads or use cases, the contractor shall install, modify, and manage those applications for unclassified HPC system utility. This includes, but is not limited to, job schedulers, reservation and dedicated partition services, data management and movement services, data visualizers, HPC applications, and user interfaces and portals.

3.2.4 Cybersecurity Support for Unclassified HPC Systems. This requirement is related to cybersecurity support for unclassified HPC needed in addition to the ERDC DSRC operations cybersecurity support identified in paragraph 3.1.3. The Contractor shall provide this additional cybersecurity support to include, but not limited to, HPC infrastructure and HPC systems. Cybersecurity requirements related to unclassified HPC systems are identified in Section 3.7.

3.2.5 Unclassified HPC System Support. Unclassified HPC systems are typically managed and administered by OEMs, third-party entities, government personnel, or the contractor. The contractor shall provide additional unclassified HPC system administration support including, but not limited to, job scheduler configuration, regular backups of critical data, assistance with troubleshooting issues, and implementation and maintenance of various persistent services on all unclassified HPC systems and storage systems at the ERDC DSRC.

3.2.6 Additional Unclassified HPC Requirements. To be determined.

3.3 Deliver Classified HPC Capabilities at the ERDC DSRC

integrating, and operating HPC systems. The baseline scope in this PWS section includes all classified HPC systems. Contractor support requirements include the HPC-related infrastructure that ties directly to delivering classified HPC capabilities such as data movement, data storage (mid-tier and archive), and user interfaces and portals to classified HPC systems. The ERDC DSRC generally hosts two to four large HPC systems as well as a varying number of smaller specialized systems across multiple classified networks. These classified networks include multiple Secret networks including SAP, multiple Above Secret including collateral Top Secret, TS/SCI and SAP/SAR. Requirements for specialized systems and HPC systems funded outside the HPCMP will vary in level of support from the large HPCMP systems.

3.3.1 Infrastructure Support for Classified HPC. The Contractor shall provide technical support to include, but not limited to, system engineering, system integration, system administration, and database administration for HPC-related infrastructure directly tied to delivering classified HPC capabilities. This includes, but is not limited to, HPC data movement and storage systems, portals, servers, routers, and switches that have direct relationships and connections to HPC systems. This does not include those systems or devices that are part of the ERDC DSRC baseline operations identified in paragraph 3.1 and associated subparagraphs.

3.3.2 Classified HPC System Installation and Integration. This requirement includes preparation of the ERDC DSRC facilities for installing, integrating, accepting, and operating classified HPC systems as well as classified storage systems. The Contractor shall provide engineering support for connecting the classified HPC systems and the classified storage systems to the ERDC DSRC infrastructure, both facilities-related and HPC-delivery-related.

3.3.3 Application Management Support for Classified HPC. The HPC OEMs typically provide levels of support, maintenance, and warranties for hardware and software delivered with an operational HPC system. For those software components that are to be installed and modified to support HPCMP workloads and use cases, the contractor shall install, modify, and manage those applications for classified HPC system utility. This includes, but is not limited to, job schedulers, reservation and dedicated partition services, data management and movement services, data visualizers, HPC applications, and user interfaces and portals.

3.3.4 Cybersecurity Support for Classified HPC Systems. This requirement is related to cybersecurity support for classified HPC needed in addition to the ERDC DSRC operations cybersecurity support identified in paragraph 3.1.3. The contractor shall provide this additional cybersecurity support to include, but not limited to, HPC infrastructure and HPC systems. Cybersecurity requirements related to classified HPC systems are identified in section 3.7.

3.3.5 Classified HPC System Support. Classified HPC systems may be managed and administered by OEMs, third-party entities, government personnel, or the contractor. The contractor shall provide additional classified HPC system administration support including, but not limited to, job scheduler configuration, regular backups of critical data, assistance with troubleshooting issues, and implementation and maintenance of various persistent services on all classified HPC systems at the ERDC

DSRC.

3.3.6 Additional classified HPC requirements. To be determined.

3.4 Deliver Specialized HPC Capabilities at the ERDC DSRC

integrating, and operating HPC systems for the HPCMP. The baseline scope in this section includes specialized HPC systems used to explore new technologies, alternative service delivery models, or emerging non-traditional workloads and use cases. These systems are typically not shared or allocated the same as the systems in paragraphs

3.2 and 3.3 and can be hosted in various environments (unclassified, classified, isolated, standalone). Contractor support requirements include the HPC-related infrastructure that ties directly to delivering specialized HPC capabilities such as data movement, data storage (mid-tier and archive), and user interfaces and portals to specialized HPC systems.

3.4.1 Infrastructure Support for Specialized HPC. The contractor shall provide technical support to include, but not limited to, system engineering, system integration, system administration, and database administration for HPC-related infrastructure directly tied to delivering specialized HPC capabilities. Contractor support includes, but is not limited to, HPC data movement and storage systems, portals, servers, routers, and switches that have direct relationships and connections to HPC systems. This support does not include those systems or devices that are part of the ERDC DSRC baseline operations identified in paragraph 3.1 and associated subparagraphs.

3.4.2 Specialized HPC System Installation and Integration. Contractor support requirements include preparation of the ERDC DSRC facilities for installing, integrating, accepting, and operating specialized HPC systems and specialized storage systems.

The contractor shall provide engineering support for connecting the specialized systems and the specialized storage systems to the ERDC DSRC infrastructure, both facilities-related and HPC-delivery-related.

3.4.3 Application Management Support for Specialized HPC. The HPC original OEMs typically provide levels of support, maintenance, and warranties for hardware and software delivered with an operational HPC system. For those software components that are to be installed and modified to support HPCMP workloads and use cases, the contractor shall install, modify, and manage those applications for specialized HPC system utility. This includes, but is not limited to, job schedulers, reservation and dedicated partition services, data management and movement services, data visualizers, HPC applications, and user interfaces and portals.

3.4.4 Cybersecurity Support for Specialized HPC Systems. Contractor support related to cybersecurity support for specialized HPC is needed in addition to the ERDC DSRC operations cybersecurity support identified in paragraph 3.1.3. The contractor shall provide this additional cybersecurity support to include, but not limited to, HPC infrastructure and HPC systems. Cybersecurity requirements related to Specialized HPC systems are identified in section 3.7.

3.4.5 Specialized HPC System Support. Specialized HPC systems are typically managed and administered by OEMs, third-party entities, government personnel, or the contractor. The contractor shall provide additional specialized HPC system administration support including, but not limited to, job scheduler configuration, regular backups of critical data, assistance with troubleshooting issues, and implementation and maintenance of various persistent services on all classified HPC systems at the ERDC

DSRC.

3.4.6 Additional specialized HPC requirements. To be determined.

3.5 Assist HPC Users at the ERDC DSRC

This section includes all help desk support required for assisting HPC users. The requirements that are ERDC DSRC specific include Help Desk basic support and higher levels of more involved support from experts at the ERDC DSRC. As some user assistance requirements are executed with a Centers enterprise perspective, contractor staff are located both at the ERDC DSRC and remotely. Classified requirements, however, will require staff to be on site to access classified resources.

3.5.1 Basic HPC Help Desk. The Centers component requires an enterprise-level basic Help Desk function to support all HPC users. Additional site specific requirements for the ERDC DSRC help desk include, but are not limited to, user account setup support and troubleshooting, basic HPC system information, limited HPC application support, basic Enterprise Services information, basic Virtual Desktop Infrastructure (VDI) information, and local and remote user or staff support. The contractor shall provide HPC customer support services (Monday-Friday, 8:00 am to 4:00 pm CST, 8 hours each day, excluding federal holidays) to local and remote users of ERDC DSRC unclassified and classified resources.

3.5.1 DSRC Help Desk Support. The Contractor shall provide support staff at the ERDC DSRC to follow up on service requests issued by the Basic HPC Help Desk in paragraph 3.5.1. These local ERDC DSRC support staff are required to be familiar with the ERDC DSRC’s HPC systems, infrastructure, and other support elements to address each Help Desk service request sent for follow-up. Support is required for both unclassified and classified HPC capabilities at the ERDC DSRC.

3.5.3 Higher Level User Support. The contractor shall provide additional ERDC DSRC user support (above the basic level) to address areas such as (but not limited to) HPC application software installation and software container development and support.

This level of support typically requires up to thirty (30) days for complex issues. Other requests for support that involve other components within the HPCMP or third parties such as HPC vendors shall be transferred appropriately.

3.5.4 HPC Application Management and Support. The Contractor shall provide support staff to assist in managing and supporting all HPC enterprise software applications that are accessible to ERDC DSRC users for their workflows and use cases. This includes government-owned, commercial, and open-source software installed (by the Centers component) and operating on ERDC DSRC HPC systems (both unclassified and classified), as well as license servers and managers.

DIRECTIVES:

The following directives are mandatory and shall be complied with:

ERDC Policy Memo #2 Ethical Uses of Communications Resources, 20 October 2006.

The following directives are provided as guidance:

AR 25-1 Army Information Technology, 15 July 2019 AR 25-2 Army Cybersecurity, 4 April 2019 AR 190-13 Army Physical Security Program, 27 June 2019 AR 380-5 Army Information Security Program, 25 March 2022 AR 380-10 Foreign Disclosure and Contacts with Foreign Representatives, 14 July 2015 AR 380-28 Army Sensitive Compartmented Information Security Program, 13 August 2018 AR 380-40 Safeguarding and Controlling Communications Security

Material, 6 September 2022 AR 380-53 Department of the Army Communications and Security

Monitoring, 21 April 2004 AR 380-67 Personnel Security Program, 24 January 2014 AR 380-381 Department of the Army Special Access Programs (SAPs) and

Sensitive Activities, 21 April 2004 AR 735-5 Relief of Responsibility and Accountability, 10 April 2024

ERDC CR 25-1-4 Information Management: Corporate Information Assurance Policy, 26 July 2001