SS DRAFT - Attachment 4 - Draft PERFORMANCE WORK STATEMENT - OBJECTIVE 3_ HPC Support Network Security.pdf

DRAFT DOCUMENT

PERFORMANCE WORK STATEMENT

Objective 3: High Performance Computing Support & Network Security

1.0 Background and Scope

The Contractor shall maintain a resilient, proactive defense posture for the NOAA Office of the Chief Information Officer (OCIO) Research and Development High Performance Computing Systems (RDHPCS) program environments and network infrastructure. The computing environment operates 24 hours a day, 365 days a year, requiring continuous security oversight to protect against emerging threats. See Attachment B for a description of RDHPCS Computing and Data Storage Resources.

The scope of this PWS encompasses the program, user support, and defense of computational clusters, the Hierarchical Storage Management Systems (HSMS), High-Performance File Systems (HPFS), HPC production nodes (e.g., Gaea, Hera, Ursa, etc.), and the network conduits that connect them, ensuring authorized researchers have seamless, secure access to computing resources. All security operations must strictly comply with Department of Commerce (DOC) and NOAA IT security policies.

2.0 Specific Task Requirements

2.1 Network Infrastructure Security and Data Protection

The Contractor shall secure the high-speed networks that enable GFDL's scientific mission.

1. Enterprise Network Integration and Defense: The Contractor shall actively secure and support operations across the NOAA N-wave Enterprise Network, which connects geographically dispurse production HPC systems, the HSMS, HPFS, and other infrastructure and service systems across the RDHPCS.

2. High-Volume Transfer Security: The Contractor shall ensure that the specialized tools and network pathways used for the reliable transfer of up to 80 Terabytes (TB)/day of data from remote production sites (e.g., ORNL) to storage and analysis systems remain encrypted, monitored, and protected from interception or corruption and shall utilize the specialized software and networking tools to manage continuous, massive data pipelines.

3. Access Control: The Contractor shall implement and manage robust identity and access management (IAM) protocols, including multi-factor authentication (MFA) and least-privilege access, for both federal personnel and the approximately 1,000 global collaborators sharing these resources.

2.2 Proactive Threat Mitigation and Vulnerability Management

The Contractor shall execute continuous monitoring and rapid incident response protocols. DRAFT

1. Continuous Monitoring: The Contractor shall utilize continuous diagnostic and mitigation (CDM) tools to monitor network traffic, identify anomalous behavior, and detect unauthorized access attempts across the HPC ecosystem.

2. Incident Response: The Contractor shall serve as the first line of defense during suspected cyber incidents, immediately isolating compromised nodes, preserving forensic data, and reporting incidents to the NOAA Information System Security Officer (ISSO) and NOAA Computer Incident Response Team (N-CIRT).

3. Continuous Patching and Scanning: The Contractor shall work with the RDHPCS integrator to execute a continuous monthly cycle of vulnerability scanning, patch management, and system updates across the entire RDHPCS inventory.

4. Vulnerability Remediation: The Contractor shall aggressively remediate identified vulnerabilities in network hardware (routers, switches, firewalls) and HPC infrastructure within federally mandated timeframes (e.g., CISA Emergency Directives).

5. Compliance Auditing & Reporting: Log all patch exceptions and endpoint remediation activities within the ITSM ticket system. Generate monthly patch compliance and vulnerability health reports for leadership, tracking SLAs for critical flaw remediation.

6. Federal Mandate Compliance: The Contractor shall ensure all devices comply with mandated federal security directives (e.g., CISA Emergency Directives, STIGs) within the required Government timeframes, aggressively remediating identified vulnerabilities.

2.3 High-Performance Computing (HPC) Systems Technical Services The Contractor shall provide direct, responsive support to the GFDL user base for all endpoint-related issues.

1. Help Desk Monitoring: The Contractor shall utilize Government-furnished IT Service Management (ITSM) tools to track, manage, and resolve user requests related to local hardware, OS errors, software installations, and peripheral connectivity.

2. Help Desk Operational SLA Metrics: Acknowledge new requests within 1 hour of receipt during hours of operation. Document all work performed in the IT ticketing application. Monitor all open tickets, provide monthly status reports summarized by category, and escalate open tickets not edited in 2 weeks or completed within 4 weeks.

Ensure tickets requesting IT service are completed within 4 weeks absent an exception approved in writing by the RDHPCS Senior IT Manager or designated alternate. Request customer feedback on quality of service.

3. Single Point of Contact (SPOC) & Ticket Triage: Operates as the SPOC for intake, ticket triage, routine account admin, and Tier I/II desktop and remote troubleshooting across multi-channel intake (phone, self-service portal, chat, email). Target First Contact Resolution (FCR) using approved Standard Operating Procedures (SOP) and analyze ticket trends to identify underlying system issues.

4. Account Administration & Access Operations: Perform routine password resets, account unlocks, basic MFA re-registrations using identity verification protocols, and process distribution list updates. Troubleshoot complex SSO, conditional access, and account sync errors across hybrid directory environments (Active Directory / Entra ID). DRAFT

5. Remote Diagnostics & Endpoint Troubleshooting: Conduct remote desktop sessions to resolve standard COTS software errors, basic browser configurations, localized VPN connectivity issues, and peripheral connectivity. Execute advanced diagnostic steps for OS corruptions, registry failures, line-of-business application crashes, and perform profile repairs without requiring full device re-imaging.

6. Knowledge Management & Process Enablement: Apply and maintain Knowledge Base (KB) and SOP articles during routine resolution workflows. Author, update, and publish technical KB and SOP articles for support staff and self-service articles and general usage documentation for end users, while mentoring Level I technicians on new technologies and escalation paths.

7. Application Packaging & Software Distribution: Push target application packages to individual endpoints via management consoles; troubleshoot local installation failures using standard runbooks; assist users with supported, user-managed software installation (e.g., Python and Conda). Package, script, wrap, and test software applications for enterprise silent deployment.

2.4 Hierarchical Storage Management System (HSMS) Operations

The Contractor shall assist the HSMS administrators with testing and optimizing the RDHPCS Hierarchical Storage Management System (HSMS) at two sites: GFDL and NESCC.

1. Capacity Management: The Contractor shall monitor usage trends and group quotas of nearly an Exabyte (and growing) of archived data, and work with the HSMS administrators ensure the HSMS meet the data generation rates and demands of the scientific use of the archived data across the RDHPCS.

2. Customized Tool Development: The Contractor shall design, develop and maintain a suite of tools to assist users in creating, storing, and accessing archived data. The contractor shall also design, develop and maintain a suite of tools to assist system administrators in maintaining, and monitoring maintaining the HPMS.

3. Data Usage Reporting: The Contractor shall maintain monthly usage reports of the

HSMS.

2.5 Cloud and DevSecOps Security Integration

The Contractor shall ensure security is integrated seamlessly into cloud environments and software development pipelines.

1. FEDRAMP Compliance: The Contractor shall enforce boundary protections and secure configurations for all GFDL cloud-based HPC efforts operating within FEDRAMP Moderate environments (Google Cloud, Microsoft Azure, AWS).

2. Code and Credential Security: The Contractor shall ensure that configurations and sensitive data (e.g., API keys, passwords) are strictly excluded from source code and are stored exclusively in an encrypted credential management system.

2.6 Security Authorization and Documentation

The Contractor shall provide a position and technical documentation required to maintain the system's Authority to Operate (ATO). DRAFT

1. Compliance Documentation: The Contractor shall continuously update and maintain the IT Security Plan, Disaster Recovery Plan, Risk Analysis Plan, and Accreditation/Self-Test documents.

2. System Auditing: The Contractor shall support annual NOAA IT security audits, providing required artifact data, log files, and architectural diagrams to federal assessors.

3. IT Security Officer (ITSO): support for high-performance computing systems, ensuring compliance with federal cybersecurity standards, identifying and mitigating system vulnerabilities, overseeing risk management processes, and providing security guidance and threat protection for scientific research operations.

3.0 Performance Standards and Quality Assurance

The Contractor’s performance under this PWS will be measured against the following standards. Failure to meet the Acceptable Quality Level (AQL) may result in rework or reduced payment per FAR 52.246-4.

Required Service Performance Standard Acceptable Quality Level (AQL)

2.1.1 Enterprise

Network Integration & Defense

Secure and support operations across the NOAA N-Wave Enterprise Network connecting production nodes, HSMS, HPFS, and support nodes.

99.9% network operational availability;

zero unmitigated security vulnerabilities or unauthorized network breaches.

2.1.2 High-Volume

Transfer Security

Ensure encrypted, monitored, and uncorrupted transfer of up to 80 TB/day from remote sites (e.g., ORNL) via pipeline tools.

100% compliance with encryption protocols; 99% successful daily transfer completion rate without data corruption or loss.

2.1.3 Access Control Enforce robust IAM protocols (MFA, least privilege) for federal staff and ~250 global collaborators.

100% compliance with MFA policies;

zero unauthorized access incidents;

access audits completed 100% on schedule.

2.2.1 Continuous

Monitoring

Deploy and utilize CDM tools across the HPC ecosystem to monitor traffic, detect anomalous behavior, and identify unauthorized access attempts.

100% CDM tool deployment coverage across network and HPC endpoints;

real-time threat detection and logging.

2.2.2 Incident Response Serve as the first line of defense; isolate compromised nodes, preserve forensic data, and report incidents to GFDL ISSO and N-CIRT.

Initial containment/isolation initiated within 15 minutes of detection; 100% compliance with ISSO/N-CIRT reporting and forensic preservation timelines.

2.2.3 Continuous

Patching and Scanning

Ensure all systems are patched, scanned, and compliant with federal security mandates on a continuous monthly cycle.

100% of endpoints receive critical security patches within 14 days of release; zero overdue CISA directive remediations.

2.2.4 Vulnerability

Remediation

Patch and remediate identified vulnerabilities in network hardware (routers, switches, firewalls) and HPC

100% remediation of critical/high vulnerabilities within federally DRAFT

Required Service Performance Standard Acceptable Quality Level (AQL) infrastructure within DOC/NOAA policies, CISA Emergency Directives, and other relevant federal requirements.

mandated timeframes (e.g., CISA Emergency Directives).

2.2.5 Compliance

Auditing & Reporting

All patch exceptions and endpoint remediation activities are tracked in the ITSM ticket system.

100% of all patch exceptions are logged, and all monthly patch compliance reports are received on time.

2.2.6 Federal Mandate

Compliance

Ensure all devices comply with federal security directives, and reports are submitted timely as required and directed.

100% device compliance, and all reports are submitted within federal mandated timelines.

2.3 HPC Systems

Technical Services

Resolve end-user technical issues rapidly. User tickets acknowledged within 1 hour during operational hours; complete requests within 4 weeks. Maintain technical and user documentation. Install and maintain software packages and distributions.

99.5% of standard endpoint tickets resolved within established Service Level Agreement (SLA) timeframes.

All support documentation maintained and tracked.

2.4 HSMS Operations Maintain the group quotas, and usage

trends on the HSMS, and submit timely usage and trend reports.

All reports are submitted on time.

2.5 Cloud & DevSecOps

Security Integration

Enforce FedRAMP Moderate boundary controls (AWS, Azure, GCP); keep secrets/credentials out of source code using encrypted management systems.

100% compliance with FedRAMP Moderate configuration baselines; zero hardcoded secrets/credentials detected in repositories.

2.6 Security

Authorization & Documentation

Maintain ATO documentation (System Security Plan, Disaster Recovery Plan, Risk Analysis) and support annual NOAA security audits with evidence artifacts.

100% of security plans and accreditation documents reviewed/updated annually or post-change; zero overdue audit artifact submissions.

DRAFT

3.0 Performance Standards and Quality Assurance