SS DRAFT - Attachment 2 - Draft PERFORMANCE WORK STATEMENT - OBJECTIVE 1_ Adaptive Earth System Model Development.pdf

DRAFT DOCUMENT

PERFORMANCE WORK STATEMENT -

Objective 1: Adaptive Earth System Model Development

1.0 Background and Scope

The Contractor shall provide agile, high-performance scientific software engineering, computational programming capacity, HPC technical support, and data science infrastructure to develop, sustain, optimize, containerize, and deploy the Geophysical Fluid Dynamics Laboratory's (GFDL) component and coupled Earth System Models (ESMs) and their supporting modeling infrastructure. This work is performed in direct support of GFDL's modeling and data science efforts and is executed within a fully integrated, cross-disciplinary environment alongside Federal research scientists, the Atmospheric and Oceanic Sciences (AOS) Program, University Corporation for Atmospheric Research (UCAR), the Cooperative Institute for Modeling the Earth System (CIMES), private entities, and other organizations.

The scientific technology stack within scope includes, but is not limited to:

1. Core Libraries & Frameworks: Flexible Modeling System (FMS) and FMS Runtime Environment (FRE).

2. Dynamical Cores & Models: Finite-Volume Cubed-Sphere Dynamical Core (FV3); Modular Ocean Model version 6 (MOM6) and other ocean, atmosphere, sea-ice, and land component and coupled models (e.g., WaveWatch, SIS, LMx, AMx, SHiELD, SPEAR, CMx, ESMx).

3. Domain Languages & AI/ML Workflows: NDSL, pyFV3, pySHiELD, PACE packages, and integrated AI Earth System Models.

4. Diagnostics & Data Platforms: Model Diagnostics Task Force (MDTF) diagnostics package, GFDL Data Portal, Earth System Grid Federation (ESGF) nodes, intake-ESM catalogs, and AI/ML data pipelines.

Work under this Objective is drawn dynamically from the Government's prioritized Master Portfolio Backlog and is executed under an Agile-Teams-as-a-Service model.

The Government retains authoritative Product Owner (PO) status. The computing environment operates 24 hours a day, 365 days a year; however, standard development work under this PWS is performed during normal business hours on Federal Government workdays except where otherwise directed. DRAFT

1.1 Contract Line Item Structure (CLINs)

This PWS is executed through the following contract line items:

CLIN 0001 – Baseline Development Sprint: One cross-functional Agile team per 4-week sprint block (Firm-Fixed-Price).

CLIN 0002 – Surge Development Sprint: On-demand additional Agile team labor capacity in 2-week increments exercised via written CO direction (Firm-Fixed-Price). Surge sprints represent additional capacity, not compression of baseline work.

CLIN 0003 – Embedded Support: Dedicated Contractor staff embedded within GFDL scientific divisions (Firm-Fixed-Price per FTE; currently estimated at four

(4) FTEs).

CLIN 0004 – Managed Services Projects (e.g. Scientific Software Stack Lifecycle Management): Managed services projects and known, recurring deliverables.

2.0 Specific Task Requirements

2.1 Agile Program Management and Capacity Execution (CLINs 0001 & 0002)

The Contractor shall deliver cross-functional scientific software development capacity as a managed Agile service. Technical tasks across all functional areas shall be drawn exclusively from the top of the Government-prioritized Master Portfolio Backlog during formal Sprint Planning.

1. Agile Ceremonies: The Contractor shall facilitate sprint planning, daily scrums, sprint reviews (including relevant product demos), sprint retrospectives, backlog estimation, and prioritization activities.

2. Sprint Cadence and Surge Mechanics: Baseline development shall be executed in 4-week sprint blocks (CLIN 0001). The Government may exercise optional 2-week surge sprints (CLIN 0002) via written direction from the Contracting Officer (CO) to support urgent, targeted, or unpredictable scientific requirements. Surge sprints represent additional team capacity and shall not compress baseline sprint deliverables.

3. Shared ALM Environment: The Contractor shall use and maintain all work items in the Government-furnished Application Lifecycle Management (ALM) tool (e.g., GitHub Projects, Jira, ServiceDesk). The Contractor shall maintain 100% bi-directional traceability mapping every task and user story to its parent Epic, DRAFT

Release, and GFDL Strategic Program Theme. Verbal tasking or informal completion confirmations are prohibited for reporting purposes.

4. Definition of Ready (DoR) and Definition of Done (DoD): User stories shall meet the mutually agreed DoR prior to sprint entry. Software increments shall satisfy 100% of the DoD prior to final acceptance and invoicing. Requirements for DoD include main-branch code integration with zero regression errors, automated unit/regression testing at agreed coverage thresholds, automated developer/user documentation, and peer-reviewed refactored code adhering to Separation of Concerns (SoC) and DOC/NOAA IT security directives.

5. Scope, Capacity, and Technical Guard Rails: The total estimated effort assigned during any sprint shall not exceed the historical or demonstrated velocity of the contracted team capacity. Team-level autonomy is granted for technical, architectural, and design decisions provided they do not (1) alter baseline scope, product vision, or budget; (2) compromise the modular design, Separation of Concerns, or HSD methodology of FMS/FRE/models; or (3) violate DOC/NOAA IT security policies. Decisions exceeding these boundaries require written direction from the Government Product Owner (PO), Contracting Officer Representative (COR), or CO.

2.2 Modular Architecture and Flexible Modeling System (FMS) Modernization

The Contractor shall optimize and modernize the Flexible Modeling System (FMS) infrastructure library to ensure GFDL's component models, coupled models, and FMS associated utilities (e.g. grid generation tools) remain maintainable, extensible, and performance-optimized, adhering strictly to the Separation of Concerns (SoC) principle.

FMS code shall remain highly portable across on-premise systems (RDHPCS) and cloud HPC architectures (e.g., AWS, Google Cloud, Microsoft Azure).

Key requirements include:

1. Co-manage a publicly accessible, version-controlled repository for the FMS infrastructure library and FMS Coupler codebases.

2. Execute continuous integration, regression testing, and quality assurance processes across supported computing platforms prior to code releases.

3. Enhance model component communication, general input/output (I/O) functionalities, temporal and spatial data interpolations, message passing interface (MPI) capabilities, and all FMS functionality.

4. Provide lower-level coupling functionalities and physical constant definitions supporting both internal and external modeling consortia.

5. Implement new features within FMS and update relevant model components using software engineering best practices. DRAFT

2.3 Workflow Management, FRE, and Runtime Environments

The Contractor shall develop, support, and maintain workflow solutions and the FMS Runtime Environment (FRE) enabling efficient model build, execution, post-processing, and diagnostics on HPC and cloud assets. Public repositories and documentation shall be maintained continuously.

Key requirements include:

1. Co-manage schema configurations to ensure repeatable model builds, execution, data transfers, post-processing, and archiving.

2. Develop, test, deploy, and maintain workflow frameworks, FRE infrastructure, and associated scientific tool integrations.

3. Develop, maintain, and enhance job performance data gathering platforms (such as EPMT) utilizing advanced statistical and machine learning analytics to optimize job streams.

4. Provide technical support for production and regression modeling workflows across designated supercomputing systems.

5. Incorporate emerging and standard workflow technologies and reproducibility standards into the testing and production environment.

6. Educate and train lab personnel on workflow standards, user tools, and best practices.

2.4 Containerization and Cloud Readiness

NOAA functions in a hybrid on-prem and cloud environment. The Contractor shall provide cloud-readiness procedures and workflows. Key requirements include:

1. Cloud-ready containerization of component/coupled model codes, FMS infrastructure, and workflows compatible with public cloud architectures (including AWS, Google Cloud, and Microsoft Azure) to ensure portability and scalability across hybrid computing environments.

2. Cloud managed images (AMI, Azure Compute Gallery, etc) and virtual machine configurations for use in earth system modeling and post processing.

3. FRE workflow to support cloud model development, compilation, runs, and post-processing

4. Cloud-ready CI/CD pipelines will be developed and maintained

5. Explore methods to reduce costs and increase the speed of adoption, including testing cloud applications and features and evaluating workflows that integrate with the cloud. DRAFT

2.5 CI/CD, Automated Testing, and Technical Debt Control

The Contractor shall implement rigorous Continuous Integration / Continuous Deployment (CI/CD) practices that automatically build and test all proposed code changes across development, staging, and production environments. Key requirements include:

1. Automated Testing Suite: Incorporate unit testing (required for 100% of new code), hierarchical testing, regression testing, quality assurance, and performance testing, striving for Test-Driven Development (TDD) compliance.

2. Code Health: Establish portable, self-documented code repositories that maintain passing status across pipelines.

3. Technical Debt Management: Assess code changes for technical debt, maintain code health metrics, and provide formal remediation plans.

2.6 Hierarchical System Development (HSD) Testing

The Contractor shall execute the Hierarchical System Development (HSD) methodology, entering the testing pipeline at multiple levels of complexity:

1. Component / mechanism level

2. Simplified testing environments

3. Fully integrated model systems

4. Supported end-to-end operational pipelines

Cross-platform testing shall be executed continuously across NOAA RDHPCS platforms, including both on-premise systems and cloud infrastructure

2.7 HPC Technical Support, Storage, and Systems Operations

The Contractor shall deliver technical support, storage architecture management, and user enablement across local, multi-site, and supercomputing environments. Key requirements include:

1. Diagnose, troubleshoot, and resolve complex HPC system issues, user execution errors, and file system performance bottlenecks.

2. Maintain and support multi-site data transfer infrastructure, including Globus services, Grid Community Toolkit configurations, and Data Transfer Nodes (DTNs).

3. Support operations, updates, and data movement across Hierarchical Storage Management Systems (HSMS, including HPE DMF) and high-performance storage platforms (including VAST). DRAFT

4. Implement multi-layer system health monitoring, metric collection, and visualization solutions for storage, network fabrics, and computing infrastructure to proactively identify performance issues.

5. Develop, maintain, and enhance core user interface scripts, login environments, and diagnostic utilities.

6. Serve as technical liaisons between scientific users, external supercomputing center operators, vendors, and system integrators.

2.8 Data Science, Services, Public Distribution, and AI/ML Integration

The Contractor shall manage internal data archives, maintain public data services compliant with federal open data mandates, GFDL Fair Use Policy, and FAIR principles, and integrate modern AI/ML data infrastructure. Key requirements include:

1. Open Data Platforms & Agentic AI Workflows: Build and maintain scalable data platforms leveraging Agentic AI workflows, AI/ML user agents, and Retrieval-Augmented Generation (RAG) to accelerate model data discovery, quality assurance, optimize resources, and reduce data-publishing turnaround.

2. Advance Diagnostic Frameworks: Deliver process-oriented and performance-diagnostic software and data products, leveraging and advancing the Model Diagnostics Task Force (MDTF) framework to foster multi-institutional alignment, strengthen internal development of GFDL Earth System Models, and partner on stakeholder engagement activities.

3. Modernized Data Pipelines & Object Storage: Modernize data pipelines by integrating cloud-native object storage layers (e.g., NetCDF/Zarr) with weather model data from GFDL and other NOAA systems (e.g., the Unified Forecast System - UFS). Test, enable, and document both emerging and established scientific data formats and storage options across local, HPC, and cloud environments.

4. Unified Data Archive (UDA): Obtain external data and update the UDA for wide-use at GFDL. Automate the request and data download process while maintaining a human-in-the-loop for accuracy and security purposes.

5. Data Cataloging & Discovery: Build, deploy, and support data discovery utilities, intake-ESM catalog integrations, automated catalog builders, and the UDA for streamlined internal access.

6. Federated Publishing (ESGF) & Metadata: Maintain, deploy, and operate Earth System Grid Federation (ESGF) nodes across bare-metal, container, and cloud architectures. Develop and maintain metadata rewriting toolsets to support community data publishing projects (e.g., CMIP), digital object identifier (DOI) integrations, and metadata schemas.

7. Data Governance & Usage Metrics: Assist scientists in data preparation, formatting, and manipulation; oversee data hosting and publication pipelines; and DRAFT track data usage metrics to assist scientific teams with compliance under applicable data management policies.

2.9 Advanced Computing, Domain-Specific Languages, and AI/ML Architecture

The Contractor shall advance modern programming paradigms and artificial intelligence workflows within Earth system modeling. Key requirements include:

1. Develop, test, and release domain-specific modeling tools and libraries, including NDSL, pyFV3, pySHiELD, and PACE packages.

2. Manage public-facing code repositories with robust automated testing and build integration.

3. Support porting, testing, and optimization of core model components to run across heterogeneous hardware, including GPUs, QPUs, and cloud infrastructure.

4. Research, implement, and integrate artificial intelligence and machine learning methodologies into modern model architectures, physics parameterizations, and full AI Earth system models.

5. Evaluate, document, advise on, and utilize recent and emerging technologies including, but not limited to, generative AI, Kubernetes, AI agents, and natural language processing (NLP) UIs to support Earth system modeling.

2.10 Model Diagnostics, Analysis Tools, and Frameworks

The Contractor shall develop, support, and advance modern diagnostic frameworks and scripting utilities for model evaluation. Key requirements include:

1. Diagnostic Integrations: Integrate data catalog tools into diagnostic frameworks and workflow tools.

2. Standardized Script Frameworks: Develop standardized analysis script frameworks capable of operating independently or integrated with primary workflow tools, ensuring compatibility with modern post-processing formats and visualization systems.

3. Community Alignment: Coordinate community contributions of Process-Oriented Diagnostics from external academic and federal partners and engage stakeholders.

2.11 Model Integration, Scientific Liaison, and Embedded Support (CLIN 0003)

The Contractor shall deliver specialized technical support directly to scientific model development teams through embedded staff and technical liaisons. DRAFT

2.11.1 Embedded Scientific Division Support (CLIN 0003):

The Contractor shall provide dedicated, embedded technical staff within designated GFDL scientific divisions to perform model development and code maintenance during regular business hours. Embedded staff are currently estimated at four (4) FTEs (currently: two (2) Scientific Programmer - III, one (1) Scientific Programmer - IV, and one (1) System Architect - V). The Government will notify the Contractor annually of required Labor Categories and FTE levels. Embedded staff operate under the same technical standards, security controls, and Definition of Done as the Agile teams, while working directly to the priorities of their host division.

2.11.2 Model Integration & Scientific Liaison Tasks:

1. Manage source code repositories (public GitHub/internal GitLab), including issue tracking, pull/merge request reviews, branch management, and CI pipelines.

2. Provide hands-on code management, guidance on best practices, branching strategies, and bug troubleshooting to scientific modeling teams.

3. Support maintenance, versioning, and configuration management of model execution files and maintain lists of supported models for regression testing.

4. Execute regression tests to verify bitwise reproducibility, science fidelity, and computational performance, thereby providing continuous sustainment, maintenance, and enhancement of the FMS framework and the FRE Runtime Environment as foundational, steady-state infrastructure underpinning all GFDL modeling activity.

5. Measure and optimize model scaling and execution performance on computing systems; evaluate compilers, dependencies, CPUs, GPUs, and QPUs, cloud, and other technologies, reporting issues as appropriate.

6. Participate in regular coordination meetings with science and technology groups, providing operational feedback to Modeling Systems Division (MSD) management.

2.12 Software Stack Lifecycle Management (CLIN 0004)

The Contractor shall build, manage, and continuously update a unified scientific software stack across supported lab computing platforms. [Under the incumbent contract, this work is currently delivered by one (1) full-time Systems Programmer/Analyst III who splits time between this activity (75%) and the sprint work (25%) and partners to fulfill this requirement with a full-time staff member on a separate vehicle.] Key requirements include:

1. Maintain and develop automated deployment tools, container environments, and packaging configurations to support climate and weather scientific software, models, and analysis. DRAFT

2. Update associated system software to use the latest supported compilers and package management tools.

3. Execute regular software updates with minimal disruption to ongoing research following lifecycle policies.

4. Release the GFDL software stack at least twice per year, including patching when necessary.

5. Produce comprehensive technical documentation, build scripts, system dependency guidelines, and maintain a robust test harness for packages, applications, and libraries.

6. Coordinate software updates with science teams to ensure software satisfies a diverse set of scientific requirements and process user software requests.

Deliverables & Release Schedule:

Deliverable ID Description Schedule / Cadence

Ongoing-1 Manage and fulfill user requests for new software applications, modules, and dependencies.

Continuous / As requested

Ongoing-2 Continuous review and triage of incoming software requests and maintenance tickets.

Weekly (Target Review Date: TBD)

D1 User-Facing GFDL Scientific Software Environment Test Release (Fall)

First Wednesday in September

D2 User-Facing GFDL Scientific Software Environment Production Release (Fall)

First Wednesday in October

D3 User-Facing GFDL Scientific Software Environment Test Release (Spring)

First Wednesday in March

D4 User-Facing GFDL Scientific Software Environment Production Release (Spring)

First Wednesday in April

D5 Robust Software Testing Harness Continuous

2.13 User Support, ITSM, and Community Engagement (Applies to all CLINs)

The Contractor shall deliver comprehensive user support and foster open-source community engagement across all contract activities. Key requirements include:

1. Provide comprehensive code documentation, developer/user guides, build scripts, and system dependency guidelines.

2. Deliver IT Service Management (ITSM)-tracked user support for code execution, system errors, and workflow troubleshooting. DRAFT

3. Facilitate scientific code hackathons, community sprints, and training workshops to advance Research-to-Operations (R2O) transition and open-source community engagement.

4. Educate and train lab personnel on workflow standards, user tools, data stewardship, and software engineering best practices.

3.0 Deliverables

All deliverables shall be submitted electronically, verified free of viruses, in the specified format. Deliverables under this PWS are in addition to, and consistent with, the master deliverables schedule in the base contract.

PWS

Section

Deliverable Description

Applicable

CLIN

Frequency / Target Schedule

Formal Acceptance Criteria / Standards

2.1 Agile Sprint

Increments

CLIN 0001 /

CLIN 0002

Every 4 weeks (Baseline) / Every 2 weeks (Surge)

100% compliance with DoD (main-branch code integration, zero regression errors, automated testing thresholds met, peer-reviewed refactoring, docs updated).

Demonstrated via live max 20-min software demo or verified logs.

2.1 Agile Sprint

Ceremonies Plan

CLIN 0001 /

CLIN 0002

One month from beginning of performance period

Agreed upon plan for accomplishing the sprint ceremonies

2.1 Bi-Directional

ALM Traceability

CLIN 0001 /

CLIN 0002

Continuous / Real-time

Every task and user story must map 100% to its parent Epic, Release, and GFDL Strategic Program Theme within the Government ALM tool (e.g., GitHub Projects/Jira).

2.2 / 2.11.2

FMS Repository & Core Optimization

CLIN 0001 /

CLIN 0002 /

CLIN 0003

Continuous / Per Release

Maintained public repository and documentation updated; automated CI/CD and regression tests passing on RDHPCS and/or cloud platforms; enhanced coupling, I/O, MPI, and temporal/spatial interpolation routines.

2.3 / 2.11.2

FRE & Workflow Platform Maintenance

CLIN 0001 /

CLIN 0002 /

CLIN 0003

Continuous / Per Release

Maintained public repositories and documentation updated; configuration schemas maintained for reproducible builds, execution, post-processing, and archiving; job metrics collection (e.g., EPMT) operational.

2.4 / 2.12

Cloud-Ready Containerization

CLIN 0001 / Per Backlog Task

Containerized components and coupled model codes compatible with public clouds DRAFT

PWS

Section

Deliverable Description

Applicable

CLIN

Frequency / Target Schedule

Formal Acceptance Criteria / Standards

CLIN 0002 /

CLIN 0004

(AWS, GCP, Azure) ensuring portability and scalability.

2.5 / 2.6 CI/CD Pipelines & HSD Testing Infrastructure

CLIN 0001 /

CLIN 0002

Continuous Automated test suite incorporating unit testing (100% coverage for new code), hierarchical (HSD) testing, regression testing, and TDD compliance; passing pipeline builds.

2.5 Technical Debt

Assessments & Remediation Plans

CLIN 0001 /

CLIN 0002

Per Sprint / As Needed

Formal code health evaluations and remediation proposals provided when technical debt threshold limits are exceeded.

2.7 HPC

Operations, Multi-Site Transfer & Health Monitoring

CLIN 0001 /

CLIN 0002

Continuous Multi-layer telemetry monitoring operational;

automated transfer protocols (Globus/DTNs) and storage systems (HSMS/DMF, VAST) optimized; issue resolution completed.

2.8 Data Platforms, Open Data Services & AI Pipelines

CLIN 0001 /

CLIN 0002

Continuous FAIR-compliant public services maintained;

Agentic AI/RAG data discovery platforms deployed; Zarr/NetCDF cloud-native object storage pipelines operational; ESGF nodes and intake-ESM catalogs maintained.

2.9 DSL Packages

Heterogeneous Computing Tools

CLIN 0001 /

CLIN 0002

Per Release / Sprint

NDSL, pyFV3, pySHiELD, and PACE packages maintained with passing CI tests;

yearly reports on code updates related to optimizing for heterogeneous hardware

2.10 MDTF

Framework & Standardized Diagnostic Toolsets

CLIN 0001 /

CLIN 0002

Continuous / Per Release

MDTF diagnostics package and CI testing suite maintained; process-oriented diagnostics integrated with catalog tools and modern post-processing formats.

2.11.2 Scientific

Liaison Tasks

CLIN 0001 /

CLIN 0002 /

CLIN 0003

Continuous / Per Sprint

Dependency remediation, cross-platform portability maintenance, defect resolution, and bitwise reproducibility verified. Performance evaluations for new models and code updates.

2.12 Ongoing-1: User

Software

CLIN 0004 Continuous / As Requested

User requests for new software applications, modules, and dependencies fulfilled within established service levels. DRAFT

PWS

Section

Deliverable Description

Applicable

CLIN

Frequency / Target Schedule

Formal Acceptance Criteria / Standards

Request Fulfillment

2.12 Ongoing-2:

Software Request & Maintenance Triage

CLIN 0004 Weekly Review and triage of incoming software requests and maintenance tickets completed and documented.

2.12 D1: User-Facing

GFDL Scientific Software Test Release (Fall)

CLIN 0004 Annual (1st Wednesday in September)

Test release of software stack deployed with automated build scripts, configuration files, dependency guidelines, and verified in the test environment.

2.12 D2: User-Facing

GFDL Scientific Software Production Release (Fall)

CLIN 0004 Annual (1st Wednesday in October)

Production software stack release deployed across supported platforms with zero critical system disruptions and comprehensive technical documentation and system guidelines.

2.12 D3: User-Facing

GFDL Scientific Software Test Release (Spring)

CLIN 0004 Annual (1st Wednesday in March)

Test release of software stack deployed with automated build scripts, configuration files, dependency guidelines, and verified in the test environment.

2.12 D4: User-Facing

GFDL Scientific Software Production Release (Spring)

CLIN 0004 Annual (1st Wednesday in April)

Production software stack release deployed across supported platforms with zero critical system disruptions and comprehensive technical documentation and system guidelines.

2.12 D5: Robust

Software Testing Harness

CLIN 0004 Continuous Dedicated, functional testing harness for packages, applications, and libraries maintained.

4.0 Performance Requirements Summary (PRS)

4.1 Quality Control Plan (QCP)

The Contractor is solely responsible for controlling the quality of all services provided under this PWS. The Contractor shall develop, implement, and maintain a Quality Control Plan (QCP) that establishes standardized procedures to monitor, inspect, and DRAFT document performance across all functional areas (Standardized & Secure Endpoints, Computer Operations, and Business Administration).

1. Submission: The Contractor shall submit a final QCP to the Contracting Officer (CO) and Contracting Officer’s Representative (COR) for approval within fourteen

(14) calendar days after contract award.

2. Updates: Any subsequent updates or revisions to the QCP during the period of performance are subject to prior written approval by the COR.

3. Core Elements: The QCP shall incorporate:

a. Automated and manual inspection/audit procedures covering routine operations and high-priority support services.

b. Discrepancy identification, root cause analysis, and corrective/preventive action procedures to prevent recurrence.

c. Trend analysis tracking and customer feedback/complaint resolution mechanisms.

4.2 Quality Assurance

The Government shall evaluate the Contractor’s performance under this contract in accordance with the Government's Quality Assurance Surveillance Plan (QASP). The QASP is a Government-administered evaluation framework designed to verify that the Contractor satisfies all contractual requirements and metrics. Government inspection and surveillance do not relieve the Contractor of responsibility for maintaining effective quality control.

4.3 Performance Requirements Summary (PRS)

The Contractor's performance under this PWS will be measured against the standards in this Performance Requirements Summary (PRS). Failure to meet the Acceptable Quality Level (AQL) may result in rework, CPARS rating deduction, or reduced payment per FAR 52.246-4.

Required Service Performance Standard Acceptable Quality Level (AQL)

Agile Program & Sprint Execution

(PWS 2.1)

Execute sprint deliverables drawn from the Master Portfolio Backlog;

deliver increment through sprint reviews (including relevant product demos). Maintain 100% bi-directional traceability mapping every task/user story to parent Epic, Release, and GFDL Program Theme in the ALM tool.

• Definition of Done (DoD): 100% compliance with DoD for all accepted stories.

• Traceability: Minimum 98% of work items mapped in ALM tool prior to sprint review.

• Demonstration: 100% of sprint reviews conducted on schedule. DRAFT

Required Service Performance Standard Acceptable Quality Level (AQL)

FMS Modernization & Modular Architecture

(PWS 2.2)

Maintain, optimize, and modernize FMS infrastructure libraries adhering strictly to Separation of Concerns (SoC). Code must remain portable across RDHPCS and cloud HPC (AWS, Azure, GCP).

• Zero regression errors introduced into core libraries.

• 100% of released code passes multi-platform CI/CD and regression testing pipelines.

• Public code repository updated continuously with automated build verification.

Workflow Management

& FRE (PWS 2.3 & 2.4)

Maintain and develop FRE runtime environments, schema configurations, job performance monitoring tools (e.g., EPMT), and associated software tools/plug-ins for repeatable model builds, execution, and post-processing across supercomputing and cloud environments.

• System uptime and availability for workflow tools at ≥ 99% during standard operational windows.

• Defect remediation initiated within 1 business day for critical workflow blockers.

• Bitwise reproducibility verified for supported model configurations.

• Public code repositories updated continuously with automated build verification.

CI/CD, Testing & Technical Debt Control

(PWS 2.5)

Maintain automated CI/CD pipelines incorporating unit, regression, hierarchical (HSD), and cross-platform testing across RDHPCS and cloud infrastructure.

Remediate technical debt proactively.

• Unit Testing: 100% code coverage required for all new code.

• Pipeline Reliability: Pipeline passing rate ≥ 95% on initial main-branch integration attempts.

• Technical Debt: Formal remediation plans provided for 100% of code changes exceeding established technical debt thresholds.

HPC Operations & Storage Support (PWS 2.7)

Diagnose and resolve HPC system issues, file system bottlenecks, and multi-site data transfers (Globus/DTNs). Support operations across HSMS (DMF) and high-performance storage (VAST).

• Incident Response: Initial triage of critical HPC execution errors within 2 hours during normal business hours.

• Data Movement: ≥ 95% successful automated data transfer completion without manual intervention. DRAFT

Data Science, Public Distribution & AI/ML

(PWS 2.8)

Operate FAIR-compliant open data services, Agentic AI/RAG discovery platforms, intake-ESM catalogs, Zarr/NetCDF cloud pipelines, and ESGF nodes. Maintain CMIP/DOI metadata toolsets.

• Data Platform Availability: ESGF nodes and public data services maintain ≥ 98% operational uptime.

• Metadata Compliance: 100% of published datasets pass automated FAIR and metadata schema validation.

Advanced Computing & DSLs (PWS 2.9)

Maintain domain-specific modeling packages (NDSL, pyFV3, pySHiELD, PACE) and optimize core models for heterogeneous architectures (GPUs, QPUs, cloud).

• 100% of DSL packages are maintained with active CI/CD integration.

• Demonstrated non-degradation of performance metrics when porting across targeted hardware architectures.

Model Diagnostics

(MDTF) (PWS 2.10)

Advance and maintain the MDTF diagnostics framework and associated CI/CD test suites;

integrate data catalog tools and deliver standardized analysis script frameworks.

• 100% of framework updates pass integrated diagnostic CI/CD testing suites.

• Diagnostics package releases fully aligned with primary model workflow versions.

Embedded Scientific Division Support (PWS 2.11.1)

Provide dedicated embedded technical staff (est. 4 FTEs) for hands-on repository management, PR/MR reviews, model execution support, and scaling evaluations within host scientific divisions.

• Staffing: Maintain ≥ 95% position coverage/fill rate for designated FTE levels.

• Code Management: PR/MR reviews completed within 3 business days of submission.

• Quality Standard: Embedded outputs meet 100% of DoD and security standards.

Model Integration & Scientific Liaison Tasks (PWS 2.11.2)

Provide code management, execute regression tests for reproducibility, measure and report model scaling performance, and participate in coordination meetings.

• PR/MR reviews completed within 3 business days

- 100% of required regression tests and performance evaluations executed without unauthorized delays

- Respond to emails from science and technology groups within 2 business days

Scientific Software Stack Lifecycle Management (PWS 2.12)

Build, manage, and release the unified scientific software stack (Spack, Lmod modules, container environments, conda) on scheduled dates (D1–D4), while providing continuous user request fulfillment and ticket triage.

• Scheduled Releases (D1–D4): 100% on-time delivery on designated Wednesdays.

• Weekly Triage (Ongoing-2): 100% of incoming tickets/requests triaged within 2 business days. DRAFT

4.4 Contract Discrepancy Reporting & Resolution Meetings

1. Issuance of CDR: The Government may issue a written Contract Discrepancy Report (CDR) whenever the Contractor's performance is determined to be unacceptable or fails to satisfy established AQLs.

2. Meeting Notification: The COR will notify the Contractor's Program Manager within two (2) business days of identifying a performance discrepancy to convene a contract discrepancy meeting.

3. Response Timeline: Within three (3) business days of receiving a written CDR, the Contractor shall submit a written response to the CO and COR detailing:

a. Root cause analysis explaining why the discrepancy occurred.

b. Specific corrective actions taken or planned to resolve the discrepancy.

c. Preventive measures instituted to ensure non-recurrence.

4. Remediation Execution: Corrective actions specified in the response plan shall be executed within two (2) business days following written acceptance by the COR and Contracting Officer.

DRAFT

2.13 User Support, ITSM, and Community Engagement (Applies to all CLINs)