Attachment 2 - TO1 PWS.pdf

Task Order 1 PWS EARTH PREDICTION INNOVATION CENTER (EPIC)

Department of Commerce (DOC)

National Oceanic and Atmospheric Administration (NOAA) Office of Oceanic and Atmospheric Research

(OAR)

Weather Program Office (WPO)

TASK ORDER ONE (TO1)

PERFORMANCE WORK STATEMENT (PWS)

Earth Prediction Innovation Center (EPIC)

Contract TBD

1. EXECUTIVE SUMMARY 3

2. BACKGROUND 3

3. OBJECTIVES AND SCOPE 4

4. TASKS 5

5. DELIVERABLES 16

6. PERIOD OF PERFORMANCE 21

7. PLACE OF PERFORMANCE 21

8. TRAVEL 21

9. CONTRACT TYPE 21

10. SECURITY REQUIREMENTS 21

11. PERFORMANCE REQUIREMENT SUMMARY (PRS) 22

12. APPENDICES 28

1. EXECUTIVE SUMMARY

This Performance Work Statement (PWS) outlines the Earth Prediction Innovation Center (EPIC) requirements for advancing weather modeling skill, reclaiming and maintaining international leadership in the area of numerical weather prediction, and improving the transition of research into operations. The general objective and scope of this PWS is to obtain state-of-the-art software architecture, software engineering and scientific and technical support services to maintain and continuously expand the community modeling infrastructure that facilitates robust user support and easy access for advanced physics-based and machine learning (ML) based Earth system models. This includes applications and components in the framework of the Unified Forecast System (UFS), a community-based, coupled, comprehensive Earth modeling system spanning spatial scales from local to global, and temporal scales from hourly to seasonal forecast scales.

The contractor shall facilitate innovation and accelerate research to operations including through an extramural, virtual center that will:

● Leverage the Weather Enterprise to provide expertise on removing barriers to improving numerical weather prediction;

● Enable scientists and engineers to effectively collaborate in areas important for improving operational numerical weather prediction skill, including model development, data assimilation techniques, systems architecture integration, and computational efficiencies;

● Strengthen the National Oceanic and Atmospheric Administration (NOAA)’s ability to undertake research projects in pursuit of substantial advancements in numerical weather prediction skill;

● Utilize and leverage existing resources across the NOAA enterprise; and

● Advance and create community weather research modeling systems that:

○ Are accessible by the public;

○ Meet end-user requirements for running on computers and networks apart from and not associated with secure NOAA information and technology systems; and

○ Utilize, whenever appropriate and cost-effective, innovative strategies and methods, including cloud-based computing capabilities, for hosting and management of part or all of the forecast systems.

EPIC will continuously integrate and develop the community-modeling environment with all partners in the Weather Enterprise through community-based, coupled, comprehensive Earth system models, to improve Numerical Weather Prediction (NWP) and accelerate the Research to Operations and Operations to Research (R2O2R) process. The NWP approach shall span local to global domains and predictive time scales from sub-hourly analyses to seasonal predictions, and will include both physics-based and data-driven ML-based ) methods. EPIC’s approach shall leverage the Weather Enterprise in alignment with NOAA Modeling Strategy 2024-20331. It is envisioned that EPIC will lead NOAA in providing strategies and infrastructure for data-driven / ML-based Earth system modeling.

The Contractor shall provide all Scientific and Technical Support Services supporting EPIC, as outlined in this PWS. All activities in this contract will be managed through the Weather Program Office (WPO) within NOAA’s Oceanic and Atmospheric Research (OAR) Line Office.

2. BACKGROUND

The Department of Commerce (DOC), NOAA, and OAR’s WPO support world-class weather research, leading to advancements in NOAA’s forecasting capabilities that save lives, reduce property damage and enhance the national economy.

In pursuit of NOAA’s vision and mission, WPO works in close partnership with OAR Laboratories, the National Weather Service (NWS), and the weather enterprise to improve extreme weather forecasts across a variety of phenomena and timescales, including severe weather, extreme rainfall, winter weather, subseasonal to seasonal forecasts, tropical weather, and fire weather and atmospheric composition. Through EPIC, WPO enables a community of practice including OAR Laboratories, the NWS, industry, academia, and other national and international organizations to improve the UFS and to help democratize operational numerical weather prediction. In short, WPO facilitates the improvement of the modeling suite leading to more seamless weather-to-climate numerical weather prediction on timescales from minutes to months

1 https://repository.library.noaa.gov/view/noaa/56332 and on spatial scales from individual clouds to the globe (WPO Strategic Plan, 2022-20262).

The Weather Research and Forecasting Innovation Act of 2017 (WRFIA, Public Law 115-25) directs NOAA to prioritize improving weather data, modeling, computing, forecasting, and warnings to protect life and property and enhance the national economy. The authorizing language for EPIC, in the National Integrated Drought Information System Reauthorization Act of 2018 (NIDISRA, Public Law 115- 423), calls for NOAA to accelerate community-developed scientific and technological enhancements into the operational applications for NWP, which requires “advancing weather modeling skills, reclaiming and maintaining international leadership in the area of numerical weather prediction, and improving the transition of research into operations.” EPIC’s vision is to enable the most accurate and reliable operational numerical forecast model in the world, and its mission is to be the catalyst for community research and modeling advances that continually inform and accelerate advances in our nation’s operational forecast modeling systems (EPIC Strategic Plan3). The EPIC-supported community modeling framework also supports innovation in modeling by facilitating the public’s access to NOAA model codes, as mandated in the LEGEND Act of 20214.

Over the last 6 years, EPIC has successfully supported open science solutions to advance community-based modeling at NOAA. This new contract continues to support this work, focusing on proven methods that enhance the R2O2R pipeline while also expanding into ML-based models, which have the potential to revolutionize the field of numerical weather prediction over the next few years. With this contract, EPIC is poised to be at the center of this transformation within NOAA by providing the core infrastructure and community support for these new systems.

3. OBJECTIVES AND SCOPE

The objective of this task order is to make continuous progress building upon the strategic outcomes achieved from activities associated with the EPIC Program in years 1-5, including:

1. Maintenance and enhancement of the Earth Prediction Innovation Center;

2. Enabling strong collaboration with the Earth and Atmospheric Sciences community to nurture a representative and unifying modeling community of practice across the Weather Enterprise;

3. Maintenance and enhancement of a publicly accessible end-to-end testing and development environment, from pre- to post-processing, for core Earth system model applications;

The contractor shall commit to maintain and continuously enhance key elements of the Earth Prediction Innovation Center using state-of-the-art technology and project management principles:

● Project Management. Provide a consistent project management approach based on the principles of Agile project management that prioritizes strategic goals, integrating priorities across technical teams, program office, and stakeholders, ensuring a high-rate of objectives completion.

● Infrastructure. Support and enhance a cross-cutting community platform as the public-facing interface of the UFS, including the EPIC Community Portal (ECP, epic.noaa.gov), integrated with continuous integration and continuous delivery (CI/CD) pipelines to enable a public-facing testing and development environment that leverages on-premise and cloud high-performance computing resources provided by NOAA and partners, hosted in an EPIC cloud account.

● Code Management. Support and enhance platform-agnostic containerized versions of the UFS code, including configurations, applications and selected components, and associated regression-testing (RT) to ensure portability and usability of on-premise and cloud High Performance Computing (HPC) platforms. Support and expand a code management framework for UFS code that follows an agreed-upon system architecture and infrastructure design and best practice following an open-source and open-development paradigm. This system architecture should facilitate the transition of research to open source code and operations within NOAA, taking into account the needs of research and operational modeling communities.

● Systems Integration. Support end-to-end prototypes and continuous integration of UFS configurations, applications and selected components, a unified build-test package management system for the UFS including but not limited to the data assimilation framework (e.g., Joint Effort for Data Assimilation Integration, JEDI), and continuously update the ECP to ensure web artifacts are up-to-date with the status of UFS and EPIC.

4 https://www.congress.gov/bill/117th-congress/senate-bill/1127 3 https://wpo.noaa.gov/wp-content/uploads/2022/09/Signed_EpicStrategicPlan2021-0903.pdf 2 https://wpo.noaa.gov/wp-content/uploads/2022/11/WPO_2022_2026_Strategic_Plan_Final.pdf

● Data Management. Support data management and ingestion requirements for unit, regression and scientific testing of UFS configurations, applications and selected components.

● Tracking performance and skill improvement. Support and enhance computational performance and forecast skill.

● Community Releases. Support community releases for UFS applications and components, including release plans and a framework for continuous coordination and support.

● Advanced User Support. Consolidate and expand an advanced user support framework.

● Community Engagement and Communications. Support and update a UFS community engagement and communications plan in collaboration with UFS application and cross-cutting teams. Facilitates engagement between the UFS modeling community and other modeling communities in the United States.

● Continuous Stakeholder Engagement. In collaboration with the Government, support and participate in a community governance framework to enhance and support community access to the UFS, define and implement best practices for community development, outline evaluation criteria for community contributions, and document the processes to guide those contributions back into the UFS and where applicable, operations. Support various teams within the community governance framework, including but not limited to functions/teams for architecture, infrastructure, education, outreach, performance, evaluation, and releases.

● Machine Learning (ML) for NWP. Support and enhance a public-facing community modeling infrastructure to allow for easy initialization, training, and visualization of data-driven NWP models based on ML and hybrid NWP models that combine the strength of data-driven and physically-based models using UFS/NOAA analyses and data products.

The general scope of this task order is to ensure a smooth transition to maintain the foundational aspect of EPIC following established procedures and protocols, and provide a strategic implementation plan for potential enhancement in follow-up task orders. In addition, this task order will also increase support for ML-based model infrastructure and will focus R2O2R activities on high priority applications.

4. TASKS

All tasks are associated with deliverables described in Section 5 Deliverables. Requirements for deliverables are provided in Section 11 Performance Requirement Summary (PRS). Tasks, deliverables and requirements listed in this PWS are subject to modifications, extensions and new scope of work to be determined by the Government prior to contract award.

The contractor shall:

4.1. Develop a Project Management Plan (PMP) to fulfill the requirements of this task order featuring a consistent project management approach that prioritizes strategic goals, integrating priorities across Agile Teams, and stakeholders, ensuring a high-rate of objectives completionThe PMP must meet the following criteria:

4.1.1. Integrates priorities across Agile Teams and stakeholders.

4.1.2. Aligns with the EPIC contract PMP and strategic plan.

4.1.3. Follows template approved by the EPIC Program Team (EPT).

4.1.4. A milestone table is created that outlines all of the requirements and a timeline of when they are expected to be delivered.

4.1.5. The milestone table is to be created within the approved project management platform to be maintained as the baseline, and is updated regularly whenever there are timeline changes.

4.1.6. A project management platform is included, aligned with the principles of Agile project management.

4.1.7. Chosen project management platform facilitates rapid and flexible development through features like Kanban or Scrum boards, backlog management, sprint planning, and collaborative communication.

4.2. Develop implementation plans to sustain foundational functions of EPIC and integrate proposed innovative solutions to improve UFS software architecture/infrastructure, computational performance and/or forecast skill.

4.2.1. Develop an implementation plan for sustaining, maintaining and enhancing the foundational functions of EPIC.

4.2.2. Develop a strategic implementation plan to integrate proposed innovative solutions to improve UFS software architecture/infrastructure, computational performance and/or forecast skill based on the most recent EPIC strategic plan. The plan must meet the following criteria:

4.2.2.1. Describes innovative solutions that add value to the UFS model computational performance and/or forecast skill in the context of potential integration to next generation models and application, or EPIC community modeling infrastructure.

4.2.2.2. Provides an estimated timeline and cost projections associated with the development of approaches, products and services5.

4.3. Support a cross-cutting public-facing community infrastructure platform to accelerate research to operations and operations to research of the Nation’s numerical weather prediction capabilities.

4.3.1. Maintain the EPIC continuous integration and continuous delivery (CI/CD) pipeline across cloud and on-premise HPC resources, ensuring that the platform agnostic UFS WM code is built and tested automatically on supported platforms. This task must meet the following criteria:

4.3.1.1. Supports at least five Tier-1 platforms and required software stack deployment supported by the EPIC contract.

4.3.1.2. The CI/CD pipeline is consistent with EPIC’s containerization based testing framework, checks expected subcomponent versions in the model code are used, is consolidated into the UFS testing framework mimicking selected Tier-1 platforms, and consistent with NOAA R2O2R and transition to operations (T2O) approach.

4.3.1.3. The regression-testing (RT) framework is compatible with the proposed containerization approach, using UFS WM and supported UFS Apps updated consistently with the latest containers, and capable of building a UFS WM executable stand-alone and several executables simultaneously, running single, multiple or full RTs, with code quality tests (e.g. ORT) for new feature tests.

4.3.1.4. Containers are released and documented in UFS and/or EPIC github repos as appropriate, leveraging on-premise and cloud high-performance computing and data dissemination resources provided by NOAA and partners, integrated and consistent with NOAA Information Technology Requirements, with prototyping consistent with NCEP Central Operations (NCO) High Performance Computing Implementation Standards.

4.3.1.5. Creates interfaces for absorbing scientific innovation provided by the community towards improving the UFS WM and applications, ensuring community innovations do not disrupt or slow down NOAA operations processes.

4.3.1.6. Provides a capability to assess the performance of new developments relative to NOAA’s operational codes, recommendations when public releases are looking to innovate and can sync up with operational codes, and a flexible architecture to facilitate testing new system architectural innovation.

4.3.1.7. CI/CD pipeline tools have the same functionality as the current UFS WM and compatible performance integrates stages and gates indicating readiness levels of innovations and identifying operational readiness.

4.3.1.8. Documentation includes full technical description, benchmarking results and relevant metrics across HPC resources.

4.3.2. Support an automated build and build-testing framework for the UFS WM and supported applications using spack-stack as the package management system and state-of-the-art containerization technology.

4.3.2.1. Continuously updates and supports deployment of up-to-date, platform-agnostic spack-stack, UFS WM, Apps and configurations consistent with the UFS testing framework in 4.3.1 to enable automatic building of specified libraries and dependent libraries in spack-stack, automatic building spack-stack with specified compiler/MPI versions, automatic testing with updated spack-stack libraries in UFS WM, and ensuring all source codes and workflows are on open-source code repositories.

5 Note: actual funds of the proposed innovative solutions will be negotiated separately and will not be included as part of this TO. The estimates are requested for the purpose of assessing feasibility.

4.3.2.2. Provides a staffing plan to co-develop spack-stack in coordination with a joint spackstack team to ensure continuity in service on Tier-1 platform as documented in the spackstack documentation.

4.3.2.3. Ensures consistency with compilers that support community development work and transitions to operational applications, addressing requirements for data dependencies, and supporting the hierarchical systems development (HSD) framework as appropriate.

4.3.3. Maintain and develop cloud sandboxes, enhanced with the platform agnostic software infrastructure as code for ECP, UFS deployment, including internal- and external-facing configurations. This task must meet the following criteria:

4.3.3.1. Continuously supports the cloud sandboxes within NOAA3000 boundary for EPIC operation and development.

4.3.3.2. Continuously updates EPIC Workload Baselines in coordination with the OAR ITMO team in compliance with NOAA IT requirements .

4.3.3.3. Utilization alarms shall be set to avoid exceeding a cost threshold of $6,000/month;

Approval from EPIC PM and COR required to exceed the set threshold.

4.3.3.4. Sandbox is supported and functional throughout the lifetime of this task order.

4.3.4. Improve the UFS-WM software process by shifting UFS-WM components toward unit testing, libraries and enforcing code quality in collaboration with teams under the UFS Governance. This task must meet the following criteria:

4.3.4.1. An updated plan is submitted and followed toward improving the UFS-WM software process to shift from submodules to libraries in the UFS-WM, with clearly articulated roles and responsibilities for the EPIC team developed in collaboration with the UFS System Architecture and Infrastructure Cross-Cutting team by the end of the second quarter in the TO POP.

4.3.4.2. The plan shall ensure proper testing of any new libraries in the UFS-WM CI/CD pipeline, smooth transition from submodules to libraries using a sound code/configuration approach, corresponding code management documents are kept up to date, and quarterly updates are provided with a summary of progress made toward the plan.

4.3.5. Continuously support a community modeling infrastructure for consolidating the hierarchical system development (HSD) framework with appropriate testing capabilities, and a public facing interface containing technical documentation and entry points for enabling community contribution and collaborations for facilitating associated scientific documentation and discussions. This task must meet the following criteria:

4.3.5.1. The HSD infrastructure includes updated scientific validated cases selected in collaboration with stakeholders, testing capabilities and a public facing interface containing technical documentation, and entry points for enabling community discussions and collaborations.

4.3.5.2. The HSD infrastructure consolidates the hierarchical system development (HSD) framework with appropriate testing capabilities, contains a public facing interface with technical documentation and entry points for enabling community discussions and collaborations, and supports users guide release, including software features and documentation for supported platforms.

4.3.5.3. The HSD infrastructure ensures connection and proper documentation of preprocessing scripts or code (e.g., for idealized cases), verification and validation packages/tools, and visualization of outputs, mimicking components in the WRF Tutorial, and documents the process for supported platforms and continuously makes available on a wiki page.

4.3.5.4. Works closely with the UFS Community, Education, and Outreach Team and EPT to collect inputs from key stakeholders, including NWS, NOAA Labs and members of the Weather Enterprise, to extend HSD and allow keeping pace with case studies relevant for community users, and to advocate the HSD capabilities .

4.3.6. Continuously support a public-facing community modeling infrastructure to allow for easy initialization, training, inference, evaluation, and visualization of ML-based numerical weather prediction models using UFS/NOAA analyses at multiple resolutions, including lightweight versions that can be easily trained by new students and the public. This task must meet the https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html https://spack-stack.readthedocs.io/en/latest/PreConfiguredSites.html following criteria:

4.3.6.1. The community modeling infrastructure is consistent with the public-facing Anemoi framework repositories to work on NOAA-accessible hardware platforms such as DOE computers, RDHPC Ursa, and on the Cloud Providers, in collaboration with NOAA scientists, and applying the framework to train and validate models by maintaining and updating the external (e.g., ERA5, CONUS404) and internal NOAA (e.g., GDAS and HRRR analyses and the UFS replay) datasets that are part of the framework.

4.3.6.2. The infrastructure includes datasets of interest into the training pipeline, trains and validates global and limited area ML models using the supported Anemoi framework, and establishes a data management framework addressing the needs of this effort.

4.3.6.3. The associated ML framework shall allow addressing needs of users from a broad range of expertise and computer resource availability levels (e.g., reduced resolution and/or cadence configurations).

4.3.6.4. The infrastructure provides an environment for developing code management, CI testing, and training artifact recording infrastructure for the Anemoi-based models, advancing Anemoi framework to support NOAA and US centric requirements, establishing pipelines to support the ML framework and verification, and enables benchmarks for training and performance assessment on selected platforms.

4.3.6.5. The infrastructure allows users to adapt the trained outcomes to additional custom datasets wherever possible, establishes a leadership team composed of EPIC, NCEP and NOAA Lab members for high-level decision making and setting roles and responsibilities, and follows an updated SOP or white paper outlining processes, procedures, roles and responsibilities, wherever applicable.

4.3.6.6. The infrastructure enables the development and supports the framework including a near real-time workflow that allows for deployment of deterministic and ensemble-based configurations of the ML models, and ensures that key stakeholders and their requirements are properly recorded in associated online artifacts, WE2E tests are developed and integrated for the ML models based on key stakeholder inputs, verification and validation packages are integrated based on key stakeholder inputs, preprocessing, postprocessing and visualization tools are included, and leverages UFS cross-cutting tools (e.g., EPIC data management framework, UWF toolkit, UFS ARCO, UPP, METPlus)

4.3.6.7. Documentation follows format and standards developed in collaboration with the UFS community, and a flexible workflow is developed that allows rapid inclusion of NOAA-developed and external ML models into the framework as applicable.

4.4. Support and coordinate the code management framework for the UFS Weather Model (UFS-WM), selected components, and cross-cutting infrastructure code repositories.

4.4.1. Coordinate and support the code and repository management of the UFS WM, UFS Apps and configurations, and the ML-based numerical weather prediction model training and near real-time prediction workflows; This management framework should be based on industry best practices.This task must meet the following criteria:

4.4.1.1. Coordinate code integration and porting to supported platforms of UFS WM and Apps and configurations consistent with sub-components, refactoring efforts and aligned with the use of UFS libraries on relevant HPC systems and./or containers.

4.4.1.2. Support the containerization and continuous development of spack-stack, and code management meetings for the UFS WM and all relevant UFS Apps within scope of the EPIC Program.

4.4.1.3. Ensure that the schedule of code integration follows priorities of UFS apps and subcomponents, communicating status to app managers in UFS application team meetings, reviewing and tracking UFS WM Pull Requests (PRs) daily, identifies issues/conflicts, and works with developers to ensure PRs are addressed.

4.4.1.4. Evaluate and coordinates new software/libraries/input data in the UFS WM, discussing with developers and SMEs impacts to operations, and continuously coordinating subcomponent repository updates and UFS WM testing for software releases including https://epic.noaa.gov/get-code/ai/ system, dependent libraries and subcomponents, ensuring submodules and the testing framework are synced with authoritative repositories every two weeks reflecting latest development and code changes in different applications.

4.4.1.5. Develop plans to improve the UFS WM testing framework to run tests efficiently and ensure code quality prior to every UFS WM and application release, continuously coordinating model code changes with workflow and downstream applications, and aligning efforts to facilitate the integration of novel UFS WM or UFS App capabilities including additional dynamic cores (dycores), such as the Model for Prediction Across Scales (MPAS), as appropriate.

4.4.1.6. Provide and archives meeting minutes, updated within 48h following every code management meeting, supporting developers on syncing/merging branches, run UFS WM RT (or ORT) on the supported platforms or in containers consistent with the transition plan in 4.3.1, monitor and resolve testing, debugging performance issues.

4.4.1.7. Maintain UFS WM RT, CI, and ORT tests, input data and baselines on HPC resources, updating the EPIC Code and Repository Management Plan to include code management support to the Anemoi-based ML repos consistent with the overall EPIC code management framework.

4.4.2. Support the seamless integration of EPIC code and repository management environment.This task must meet the following criteria:

4.4.2.1. Ensure that applications connect to the individual component models after changes are made to the application code repository, documenting dependencies of application repos on individual components on GitHub repo wikis, and ensuring that as components get upgraded, applications are tested with every upgrade so that component code changes do not break the system across the different Tier-1 platforms (including containers).

4.4.2.2. Deploy benchmarking retrospectives to ensure new developments are not degrading accepted operational forecast skill and computational performance, also ensuring that new features introduced in component repositories meet implementation standards, are thoroughly tested using available operational workflows, and include a tracking system, wiki and user support to the code management tool used by community developers.

4.4.3. Coordinate the Continuous Integration (CI) Teams to support the UFS WM and UFS Apps in collaboration with key stakeholders. This task must meet the following criteria:

4.4.3.1. CI teams must be established within one month after support to a new app begins as part of CI team daily activities ensuring requirements of the integrated EPIC code and repository management environment are met, maintaining regression, cross platform and integration tests, enhancing the regression test framework with new features, and integrating new model functional tests into the regression test framework.

4.4.3.2. Continue to support Tier-1 platforms or in containers consistent with the platform and containerization plan in 4.3.1, updating supported compiler versions as appropriate, coordinating regression testing amongst the various members of the team, and ensuring latest results of regression tests are captured within the UFS model repositories.

4.4.3.3. Develop activities in collaboration with key stakeholders, working with component code managers to test upcoming features and alpha/beta releases of the components, investigating regression test failure when required, making code updates when/where necessary, verifying and validating test results, creating new baselines for expected result changes, and tracking unexpected changes.

4.4.3.4. Ensure computational performance is maintained through communication with component managers on a regular basis, and archive meeting minutes and notes with availability to the EPT and community.

4.4.4. Support the development and integration of the CICE component in the UFS WM, including design, code and repository management and continuous integration, development and documentation of CICE component software, development practices and processes. This task must meet the following criteria:

4.4.4.1. Include design, code and repository management and continuous integration, development and documentation of component software, development practices and processes.

4.4.4.2. Develop model run and test scripts, testing strategy, and test implementation including ability to provide continuous integration and development for the UFS community, providing software engineering support to CICE and Icepack code separation and interfaces.

4.4.4.3. Help review and validate proposed model source code changes, evaluating and optimizing model porting and performance EPIC supported platforms or containers consistent with 4.3.1.

4.4.4.4. Provide broad oversight of the code to ensure compatibility and software best practices, and software consulting support for team members in coordination with the UFS code management team, developing documentation following standards and requirements applicable to the UFS WM and applications.

4.4.5. Support the UFS Stochastic Physics team in their code repository management activities consistent with code management practice guidelines in Appendix I. This task must meet the following criteria:

4.4.5.1. Ensure GitHub actions artifact is in place that checks for documentation for future PRs.

4.4.5.2. Ensure documentation is up to date reflecting all capabilities in the physics repository, working with key stakeholders to ensure that proper logging of test results is being done by the GitHub actions.

4.4.5.3. Testing is automated and executed with each PR. Minimizes manual testing that has to be done by the code managers, updating the established CI/CD pipeline as appropriate.

4.4.6. Support a data management framework for findable, accessible, interoperable, and reusable data.

This task must meet the following criteria:

4.4.6.1. The data management framework is consistent with the NOAA Public Access to Research Results (PARR) plan (https://repository.library.noaa.gov/view/noaa/10169).

4.4.6.2. The framework updates the current procedures and infrastructures accordingly, in support of the UFS community data needs for development, testing, and evaluation of the UFS WM, UFS Apps, data assimilation systems, post-processing, and ML models.

4.4.6.3. Maintain a database available for download, or being staged, containing data for UFS WM, UFS Apps, ML models, and components for running relevant test cases, including representative forecast runs and leveraging the already-existing UFS Case Studies Platform, updating datasets supporting UFS Apps and ML models as part of code releases.

4.4.6.4. Include full resolution or thinned data depending on the case study and documented user needs, and observational datasets available using conventional and satellite observations available from sources such as NCEP/NOMADS/NCEI operational and research archives for relevant weather and oceanographic events.

4.4.6.5. Include pre-processing tools to prepare appropriate BCs, ICs on cloud and on-premise HPC platforms, and datasets matching test cases appropriate for case studies and updated reflecting needs of UFS App and ML model releases.

4.4.6.6. Establish a data pipeline by engaging the appropriate POCs from data source organizations, ensuring datasets are available in a public repository in the cloud and via the EPIC Community Portal.

4.4.6.7. Provide documentation describing data, formats and applications, including data mapping algorithms to facilitate the data management, complying with requirements related to Findability, Accessibility, Interoperability, and Reuse of the data, and ensuring availability and easy access to relevant observational dataset with appropriate BCs, ICs.

4.4.6.8. Continuously update a unified data mapping framework that ensures dependent data in the cloud is available for processing by UFS WMs running Tier-1 platforms (including containers), ensuring datasets are available in a public repository in the cloud, are accessible via the EPIC Community Portal, and include documentation describing data, formats and applications.

4.5. Continuously support development and integration of: the UFS Short-range Weather (SRW) App, a https://epic.noaa.gov/get-code/stochastic-physics/

UFS-based community Data Assimilation (DA) system; UFS Global Apps; and ML-based Global and Limited-area Ensemble forecast system(s) that can facilitate R2O2R.

4.5.1. Coordinate reviews and integrate pull requests into the authoritative repositories based on the established EPIC code management framework (Appendix I). This task must meet the following criteria:

4.5.1.1. Coordinate code integration and porting to supported platforms of UFS Apps and configurations consistent with sub-components, supporting code management meetings for the UFS SRW, UFS-based community Data Assimilation (DA) system, Global App, and ML model as needed, and following approved UFS application system architecture and infrastructure document and UFS Application Software Governance.

4.5.1.2. Communicate status to app managers in UFS application team meetings, continuously coordinating subcomponent repository updates and model code changes with workflow and downstream applications, ensuring submodules are synced with authoritative repositories every two weeks.

4.5.1.3. Continuously coordinate model code changes with workflow and downstream applications, providing archives containing meeting minutes updated within 48h following every code management meeting.

4.5.1.4. Support developers on syncing/merging branches, run ORT on Tier-1 platforms, monitor and resolve testing, debugging performance issues, maintaining ORT tests, input data and baselines on EPIC supported Tier-1 platforms, and assisting community developers with code commits into UFS repositories and helping determine which tests need to be run (besides minimal set of WE2E tests).

4.5.1.5. Follow the approved WE2E plan for each supported app or configurations, adding new WE2E tests for new features.

4.5.1.6. Ensure all integration activities are consistent with application pipelines and with GitHub Actions or Jenkins.

4.5.2. Develop and deploy a minimal set of workflows end-to-end (WE2E) tests in each authoritative repository for supported applications (i.e., UFS SRW App, UFS-based Community DA system, Global App, and ML, listed at https://epic.noaa.gov/get-code). This task must meet the following criteria:

4.5.2.1. Follow the approved WE2E plan for each supported app or configurations, adding new WE2E tests as needed for new features, and ensuring a consistent Application pipeline with GitHub Actions or Jenkins.

4.5.3. Integrate verification and validation metrics in coordination with the application teams for supported applications. This task must meet the following criteria:

4.5.3.1. Provide a plan to enable a workflow to integrate postprocessing utilities, verification and validation metrics, and visualization packages to an EPIC forecast skill dashboard with an external web interface on EPIC website

4.5.3.2. The workflow shall allow side-by-side comparison of outputs from physically based vs data-driven numerical weather prediction models, including deterministic runs and ensembles.

4.5.3.3. App documentation shall include step-by-step instructions to customize the V&V workflow using at least one case study as an example.

4.5.3.4. Case selection is made in coordination with the application teams for supported applications, ensuring consistency in V&V packages between physically based and ML-based models.

4.5.4. Improve supported components or supported applications with enhanced DA, preprocessing, postprocessing, and verification and validation capabilities in coordination with the corresponding application teams. This task must meet the following criteria:

4.5.4.1. Create a plan with workflow elements or features in components that require improvement identified in coordination with key stakeholders for each supported App, ensuring improvements are submitted as PRs to the corresponding application repo or cross-cutting repos (e.g., DA, UWF).

4.5.4.2. Ensure alignment with priorities of the Consortium for Advanced Data Assimilation

Research and Education (CADRE), and documentation and wiki pages are updated to reflect improvements.

4.5.5. Benchmark the computational performance of end-to-end configurations for at least one prototype of each supported application on selected computing platforms. This task must meet the following criteria

4.5.5.1. Create a benchmarking plan with the configurations and prototype identified in coordination with key stakeholders for each supported App.

4.5.5.2. Conduct the benchmarking of each app on selected computing platforms or approved by EPT, ensuring benchmark results are advertised to the broadest extent to highlight what the current capability is.

4.5.6. Deliver optimized prototypes for supported applications with documentation. This task must meet the following criteria:

4.5.6.1. Documentation is updated with all integrated features in 4.5.3 – 4.5.5 linked to UFS document databases (Application Documentation github rep, wiki pages).

4.5.6.2. Prototype codes are on open-source code repositories, including software stacks that support community development work and transitions to operational applications, addressing requirements for data dependencies, and providing user-friendly evaluation tools, input datasets and documentation.

4.6. Support planned community releases and capability announcements of UFS Apps and configurations.

4.6.1. Update the EPIC Release Process Documentation (Appendix II) and Release Coordination Cross Cutting Team (CCT) Charter (Appendix III). This task must meet the following criteria:

4.6.1.1. Collect inputs and incorporate perspectives of key stakeholders for supported configurations/applications (e.g,, HSD, SRW App, UFS Community DA System, Global App, ML-based models) in coordination with the government.

4.6.1.2. Release prioritization should be based on the stakeholder engagement process in 4.10, in collaboration with the Government.

4.6.2. Support release coordination efforts based on the approved release process and charter in 4.6.2.

This task must meet the following criteria:

4.6.2.1. Follows the approved UFS Release Coordination Vision and Release Coordination CCT Charter from 4.6.1.

4.6.3. Support capability announcements of selected physically-based and/or ML-based Apps/framework. This task must meet the following criteria:

4.6.3.1. Capability announcements are made reflecting program and community priorities of UFS-WM and ML components and Apps UFS including, but not limited, to the HSD framework, the UFS SRW App, the UFS Community DA System, the UFS Global Apps, and Anemoi framework.

4.6.3.2. Determine feasibility and level of support to HAFS on the basis of discussions with relevant stakeholders, coordinating with the JEDI Configuration Builder (JCB) team to identify suitable JCB configurations for UFS community DA system capability announcements, and consistent with UFS Public Release Plans.

4.6.3.3. Release plan indicates clearly key stakeholders for each release, and are made available and easily accessible to relevant key stakeholders.

4.6.3.4. Establishes a SOP for releasing reanalysis data from selected modeling centers (e.g., ECMWF or NCEP) as the meteorological driver, if appropriate.

4.6.3.5. Deploys final platform-agnostic containerized versions of the UFS WM and relevant components and applications, where appropriate, ensuring code is on open-source code repositories and runs consistently in on-premise, cloud, and limited testing environments, containers have all necessary infrastructure for ensuring UFS WM and App codes are properly built for applications being rolled out, are consistent with compilers that support community development work and transitions to operational applications, and address requirements for data dependencies associated with running regression and usability tests and case studies, providing user-friendly evaluation tools, work datasets and documentation integrated with each public UFS App release.

4.6.3.6. Develop a finalized minimal set of workflow end-to-end (WE2E) tests for each PR submitted into the UFS WM and relevant applications consistent with CI/CD pipeline, collaborating with community developers to determine which tests need to be run (besides minimal set of WE2E tests), adding new WE2E tests as needed for new features.

4.6.3.7. Evaluates whether unit tests are needed for PR, and using automation server and testing approaches consistent with the UFS community framework.

4.6.3.8. Documentation is updated reflecting new features added to the UFS WM repository as pertains to the SRW release, following format and standards developed in collaboration with the UFS community.

4.6.4. Support the releases of selected physically-based and/or ML-based Apps/framework. This task must meet the following criteria:

4.6.4.1. Releases are consistent with stakeholder inputs from 4.6.1 and 4.6.2, and leverage the UFS cross-cutting tools (e.g., EPIC data management framework, UWF toolkit, UFS ARCO, UPP, METPlus)

4.6.4.2. Documentation follows format and standards developed in collaboration with the UFS community.

4.7. Support an advanced user support framework for UFS-WM, UFS HSD framework, UFS Community DA System, Global, SRW Apps, ML models, UWF, UPP, and stochastic physics.

4.7.1. Coordinate and manage user support forums integrated to the ECP. This task must meet the following criteria:

4.7.1.1. Establish a community engagement team to ensure seamless integration to the ECP, ensuring consistency with DTC and UFS community needs, and continuity of existing UFS forums.

4.7.1.2. Monitor daily forum postings from the community and provide technical advice engaging appropriate SMEs in collaboration with UFS working groups, developing a list of SMEs for the various UFS components and applications, providing answers or contacting UFS SMEs for support within a reasonable time frame (eg, 48 hours).

4.7.1.3. Document a support team with subject matter expertise to provide answers reducing dependency on SMEs, providing an SOP for activities within the PoP of this task order, and tutorials for UFS developers in code management, continuous integration, and DevOps in forms of wiki pages linked to EPIC supported GitHub repositories (i.e., UFS-WM, SRW,, UWF,, Community DA System, Global App), videos, and training workshops.

4.7.1.4. Submit a plan to address this requirement in the second sprint of the PoP.

4.7.1.5. Support the HAFS Development coordination meeting.

4.7.2. Develop tutorials and instructional materials dedicated to community modeling in Earth sciences, understanding, running and developing the UFS. This task must meet the following criteria:

4.7.2.1. Materials are accessible to users of all backgrounds and levels of expertise, addresses requirements reflecting needs of EPIC and UFS communities.

4.7.2.2. Videos follow all accessibility standards (i.e., 508 compliance, subtitles), and content is updated, as needed within an agreed mutually timeframe, to reflect changing content.

4.7.2.3. Technical support is provided to developing content for tutorials, engaging with the community engagement teams, ensuring technical details of all tutorials are accurate, and that tutorials involving code build and running case studies are fully functional and ready for cloud and supported on-premise platforms.

4.7.2.4. Technical support is provided to develop content relevant to the success of the EPIC program, including EPIC processes and CI/CD pipeline, community code development, testing.

4.7.2.5. Development of tutorials and instructional materials is aligned with the EPIC educational programs coordinated by EPT, such as the UFS faculty and student ambassador programs.

4.7.2.6. Outreach materials in this task are made available via the ECP, and contents are reviewed by technical and non-technical teams to ensure quality and accessibility.

4.7.3. Develop documentation supporting the UFS Apps. This task must meet the following criteria:

4.7.3.1. Documentation is updated reflecting new features added to UFS and EPIC repositories, following format and standards developed in collaboration with the UFS community.

4.7.3.2. Documentation is accessible via public engagement resources, including contents, links and guidance in the ECP, GitHub wikis and relevant resources, in coordination with the UFS release coordination working group.

4.7.4. Support and enhance the interface providing selected users access to cloud-based sandboxes or other suitable compute environments for testing of innovations, attending workshops and learning events, and changes to the UFS models. This task must meet the following criteria:

4.7.4.1. Continuously support the cloud sandboxes within appropriate NOAA IT boundaries for EPIC operation and development.

4.7.4.2. Continuously update EPIC Workload Baselines in compliance with NOAA IT requirements.

4.7.4.3. Include utilization alarms to avoid exceeding a monthly cost threshold to be determined post-award.

4.7.4.4. The sandbox is supported and functional throughout the lifetime of this task order.

4.7.4.5. Documentation is provided and broadly disseminated describing Interface allowing sandbox access and user access controls.

4.7.4.6. User support is available to ensure access is successful ahead of workshops and events.;

4.8. Support and update a community engagement strategy associated with EPIC and the UFS community mission and strategic goals.

4.8.1. Update the EPIC communications strategy document. This task must meet the following criteria:

4.8.1.1. The EPIC communication strategy document must be reviewed and updated quarterly (and published annually or as requested by the EPT) to ensure goals, outcomes, and metrics are communicated broadly to the community; strategy aligns UFS Governance and Structure; previous plans and documents owned by the Program Team shall be updated to remain current; updates are consistent and aligned with the UFS Community, Education and Outreach (CEO) team; identify metrics for tracking goals, outcomes, and measuring the success of communications efforts.

4.8.1.2. Rollout plans are created for each release and workshops in collaboration with the EPT.

4.8.1.3. Ensure participation in scheduled UFS CEO team and UFS Steering Committee meetings.

4.8.2. Support and enhance the EPIC Community Portal (ECP). This task must meet the following criteria:

4.8.2.1. Portal includes dashboards with metrics, integration to the community (E.g., GitHub) tools, to community forums, to ticketing systems, and sandbox access, and interfaces for user access to GitHub code repositories, integrated access to EPIC and UFS GitHub wikis, ticketing/issues, user forums, and access to a user sandbox when applicable.

4.8.2.2. Portal follows responsive design and 508 compliance and human-centered design best practices, is Search-Engine Optimized (SEO), and follows style guidelines established in the EPIC communications strategy.

4.8.2.3. Portal addresses requirements of this task order as indicated in other PWS items and their respective requirements. (Verified only during the fourth PI in the PoP after all other items are accepted).

4.8.2.4. Portal reflects user feedback and technology needs and opportunities, including regular feedback from community members.

4.8.2.5. Content approvals follow the EPIC Website Standard Operating Procedures, whereby contract and program teams are…