R.1 -- Attachment 4 - 1305M320RNRMN0002 - PWS Sample Task Order One - Amendment 01 - Track Changes.pdf

ATTACHMENT 4 – R1 EARTH PREDICTION INNOVATION CENTER (EPIC)

Department of Commerce (DOC) National Oceanic and Atmospheric Administration (NOAA)

Office of Oceanic and Atmospheric Research (OAR) Office of Weather and Air Quality (OWAQ)

SAMPLE TASK ORDER ONE

PERFORMANCE WORK STATEMENT (PWS)

Earth Prediction Innovation Center (EPIC)

ATTACHMENT 4

1305M320RNRMN0002

REVISION 1

Contents

1. INFORMATION 3

2. BACKGROUND 3

3. OBJECTIVES 4

4. SCOPE OF WORK 5

5. TASKS 5

6. DELIVERABLES 10

7. PERIOD OF PERFORMANCE 12

8. PLACE OF PERFORMANCE 12

9. CONTRACT TYPE & RISK 13

10. PERFORMANCE REQUIREMENT SUMMARY (PRS) 13

1. INFORMATION

This Performance Work Statement (PWS) outlines the Earth Prediction Innovation Center (EPIC) requirements for obtaining software engineering and scientific/technical support services on cloud infrastructure and to provide robust user support for easy access to the latest version of the National Oceanic and Atmospheric Administration (NOAA) Unified Forecast System (UFS) which is a community-based, coupled, comprehensive Earth modeling system. The efforts are intended to accelerate community-developed scientific and technological enhancements into the operational applications for Numerical Weather Prediction (NWP).

NOAA is leading the development of the EPIC and the associated scientific work practices that will change the way community integrated earth system modeling is achieved in the United States. EPIC will continuously integrate and develop the community-modeling environment with all partners invested in EPIC through a community-based, coupled, comprehensive Earth modeling system (i.e. UFS) to improve NWP and to accelerate the Research to Operations and Operations to Research (R2O/O2R)R2O/O2R paradigm. The UFS numerical applications span local to global domains and predictive time scales from sub-hourly analyses to seasonal predictions. It is designed to support the Weather Enterprise and to be the source system for NOAA's operational NWP applications.

The Government will rely upon the Contractor to provide EPIC with Scientific and Technical Support Services directly supporting EPIC as outlined in this PWS. The resultant contract task order award will be managed through the Office of Weather and Air Quality (OWAQ) within NOAA’s Oceanic and Atmospheric Research (OAR) Line Office.

2. BACKGROUND

The Department of Commerce (DOC), NOAA, and OAR’s OWAQ supports world-class weather and air quality research that leads to, among other things, advancements to NOAA’s NWP that ultimately saves lives, reduces property damage, and enhances the national economy.

In pursuit of its vision and mission, OWAQ works closely with the National Weather Service (NWS) to help develop and transition to NWS’ operational suite of forecast models weather and air quality research, including hurricanes, severe thunderstorms, heavy precipitation, and air pollution. Additionally, OWAQ utilizes social science to learn how to deal with the uncertainties weather presents and to inform its engagement and communication with researchers, funders, and the public.

Furthermore, OWAQ selects and funds research that supports and fosters collaborations within NOAA's research laboratories and across the weather enterprise (i.e., NOAA, other Federal agencies and entities, state and local governments, academia, and the private sector).

The Weather Research and Forecasting Innovation Act of 2017 (WRFIA, Public Law 115-25) directs NOAA to prioritize improving weather data, modeling, computing, forecasting, and warnings for the protection of life and property and for the enhancement of the national economy. The authorizing language for EPIC, in the National Integrated Drought Information System Reauthorization Act of 2018 (NIDISRA, Public Law 115- 423), calls for NOAA to accelerate community-developed scientific and technological enhancements into the operational applications for NWP, which requires “advancing weather modeling skills, reclaiming and maintaining international leadership in the area of numerical weather prediction, and improving the transition of research into operations.”

The devastating impacts of hazardous events on life, property, and the national economy can be mitigated with accurate forecasts. To meet this need while reclaiming global leadership in weather prediction, NOAA must serve the nation by building a community-modeling environment for innovative, integrated earth system modeling.

EPIC is the catalyst for accurate weather forecasts in the near-term and integrated earth system modeling in the long-term. In the near-term, this means developing NWP skills in the UFS. In the long-term, this means expansion to other operational model applications such as high-resolution convective-allowing models and fully coupled subseasonal to seasonal, water and ocean forecast system. EPIC is working to fulfill its mandate and to exceed customer expectations. For additional information on EPIC, see https://owaq.noaa.gov/Programs/EPIC and on UFS, see https://ufscommunity.org/.

3. OBJECTIVES

The objective of this first task order is to provide NOAA with an assessment and project management plan for successful implementation of the Statement of Objectives for the overall EPIC Contract and for supporting and building of the UFS applications. The contractor will provide EPIC with affordable and effective Professional Scientific and Technical Support Services that enable EPIC to meet its community mission requirements for making UFS accessible as an open source system with access through GitHub repositories to meet R2O/O2R functions. Additionally, this task order will build the UFS codes and dependent software on Cloud ecosystems using platforms such as AWS, Google, and Azure with a focus on building a system that is interoperable and portable from Cloud to on premise High Performance Computing (HPC) architectures. The contractor will integrate into the UFS governance structure to provide user/customer support for the cloud-based UFS system and HPC platforms relevant for the benefit of the wider EPIC and UFS community.

The primary areas of focus of this task order are described below:

● Provide an overall assessment of the UFS cloud related activities and develop a project management plan for the EPIC Contract including comprehensive support of all UFS applications.

● Develop a community-based UFS weather application modeling system that runs on Cloud HPC with robust user support, easy access to the latest version of code, input and output data; a code repository maintained under version control software (Github); thorough, understandable documentation; user-friendly workflow; software infrastructure; tutorials, workshops, help desk services; and ensuring that developers are involved in user support.

● Communicate broadly EPIC activities that deliver short-term success online and publicly at conferences

The success of this contract is solely dependent on integration of EPIC into the UFS infrastructure and governance to support community-based coupled Earth system model development by providing the community of users with a modern and flexible UFS framework that works on Cloud and can easily transition to conventional HPC owned by UFS partners. It’s imperative to engage the extensive research enterprise in the U.S. for EPIC to succeed. EPIC will create an environment to more efficiently and effectively translate research advances into operational outcomes. Ultimately, EPIC’s support of the UFS will enable NOAA to simplify its production suite of forecasting models from a great many independent systems, each of which has to be improved and maintained, to a single seamless coupled Earth modeling system.

4. SCOPE OF WORK

This Contractor shall:

4.1 Develop a Project Management Plan to fulfill the requirements of this Task Order and develop a second Project Management Plan as a part of this Task Order to implement the full scope of the EPIC Contract.

4.2 Develop a platform-agnostic containerized version of the UFS weather application that can be used on Cloud HPC, meaning there is a documented process for running element(s) of the UFS weather application on Cloud HPC via multiple vendors, and passes the graduate student test (tThe Graduate Student Test means that a student can easily: access code, run code, change code, test code for correct operation, and evaluate code with standard diagnostic packages; obtain code documentation, user support, and training; and understand what is needed for their code changes to transition to operations).

4.32 Provide the Weather Enterprise with access to UFS weather application via regular code releasesoperational model code, user-support services, and work closely with NOAA on with a method for the secure ingest of community code into NOAA applications (with open source on Github and open data policies based in NOAA’s and other agency best-practices and policies).

4.43 Align computer scientists and software engineers with earth scientists to: (1) create well-documented cloud-friendly code; and (2) free earth scientists to perform research. Communicate short-term successes.

4.54 Provide solutions for accelerating the research to operations pipeline and supply a user-support team with individual accountabilities for answering questions and updating the frequently-asked questions.

The primary focus of this service will be to ensure access and usability by all community members, regardless of the level of expertise and nationality.

NOAA’s current capacity to provide operational and research-related HPCHigh-Performance Computing is a limitation to the advancement of numerical weather prediction. The transition of research and development to a cloud environment has the potential to dramatically improve NOAA’s capacity. It is NOAA's intention with EPIC to fully test the limits of this possibility, consequently, the plan that is developed under this RFP shall incorporate this.

5. TASKS

5.1 : Assessment of R2O/O2R of the Current and Future Unified Forecast System

The Ccontractor shall:

5.1.1 Describe your vision to meet the mandates of the WRFIA and NIDIS reauthorization with respect to EPIC to accelerate community-based scientific and technological enhancements into the operational applications for NWP improve and supply NOAA with community-based advanced UFS applications through for engagement ing with the Weather Enterprise and in alignment with the EPIC Strategic Plan and Governance Principles (currently in draft form) (i.e., UFS users contributing code for potential operational use within NOAA and methods to grow the usergrow user base) to meet the mandates of the WRFIA.

5.1.2 Recruit and establish a team of professional software, systems architecture, infrastructure, data scientists, and computational engineers to execute this Performance Work Statement who will work directly with physical scientists in an integrated, semi-virtual, cross-disciplinary environment to ensure that the UFS is supported, easily maintainable, extensible and performance optimized.Define, and develop, and execute a full project management plan to include project scope management plan, milestones, schedule management plan, cost management plan, risk management plan, risk register, release management plan, quality management plan, communication management plan, and staff management plan necessaryclear project plan with deliverables and success metrics to implement the vision outlined in section 5.1.1. in conjunction with the Government’s objectives as outlined in the EPIC Statement of Objectives. that outline the path to supporting the UFS

5.1.3 Define and develop a full project management plan to include project scope management plan, milestones, schedule management plan, cost management plan, risk management plan, risk register, release management plan, quality management plan, communication management plan, and staff management plan necessary to implement the vision outlined in section 5.1.1. in conjunction with the Government’s objectives as outlined in the EPIC Statement of Objectives.Recruit and establish a team of professional software, systems architecture, infrastructure, data scientists, and computational engineers to work directly with physical scientists in an integrated, semi-virtual, cross-disciplinary environment to ensure that the UFS code is supported, easily maintainable, extensible and performance optimized.

5.1.4 Project Management Plan, specified at 5.1.3 above, shall include the following elements/themes:

5.1.4.1 Conduct an assessment that Ddescribes and defines the optimal mechanisms for supporting and accelerating the R2O/O2R process at NOAA.through this contract.

5.1.4.25 Define the community of users and define the pathways into the UFS repositories with the UFS governance structure..

5.1.4.3 Describe how you will enable community based research and development through public releases of the UFS in aAlignment with the UFS science goals, strategic implementation plan, and EPIC Strategic Plan. science goals of the UFS Applications.

5.1.4.46 Conduct an aAssessment and provide a comprehensive report of on UFS cloud activities to avoid duplication of effort and .assess the UFS run on the cloud for three platforms (AWS, Google and Azure).

5.1.4.57 Assess and develop plan for agile development and Continuous Integration Continuous Deployment (CI/CD) pipeline, secure ingest, data services, cloud storage, debugging, model integration, reforecasts/reanalysisrealanysis, and retrospective of Work with parties to define the vetting process of UFS code back into the UFS master repository.

5.1.4.68 Conduct a review and aAssessment of the current UFS planning documents (for example Repository Management Plan and the UFS Research to Operations Plan) to ensure optimal success and provide feedback to the UFS community.

5.1.4.79 Conduct an assessment and evaluation of the of current UFS communications platform and determine if it is optimal, given available alternatives, for interaction within the UFS community that includes researchers, developers and users from NOAA, educational institutions, other federal agencies, and the private sector.5.1.9 Survey, assess, and provide a report on current UFS related activities within NOAA utilizing cloud services such as continuous integration (CI), continuous development (CD), data services, debugging, model integration, reforecasts/reanalysis, and retrospectives to avoid duplication of effort and identify gaps.

5.1.4.810 Conduct an assessment of UFS code and published coding standardsProvide advice and best practices to to determine whether these are sufficient to that ensures that the a UFS code that is innovative, modern, maintainable, extensible, and performance optimized on modern computer architectures. Include best practices on CI/CD approaches that automatically build and test all proposed code changes in its testing infrastructure.

5.1.4.911 Work with relevant NOAA labs and centers (i.e., EMC) to develop CI/CD process for the UFS. Conduct an assessment on Ensure the the hierarchical testing framework for the UFS applications and works across all UFS modeling components.

Provide a description of how to design and implement an enterprise-level agile approach to accelerate software testing and integration of the UFS through secure ingest procedures and accelerating the R2O/O2R process.

5.1.4.1012 Conduct an assessment on community-based and operational workflows (i.e. Common Infrastructure for Modeling the Earth (CIME), ecFlow, and Rocoto) running on cloud environments.

5.1.4.1113 Assess cloud readiness of Common Infrastructure for Modeling the Earth (CIME) workflows in three cloud environments (i.e., AWS, Google and Azure).

5.1.4.12 Provide an assessment on current and future needs for community engagement to build a strong community user base for EPIC.

5.1.514 Finalize assessments and submit report on the EPIC project management plan according to the following schedule:.

Project Management Plan Development Schedule:

Community Interviews and Artifact Review 90 days after award

50% Submission (Updated Project Management Plan Summary)

30 days after completion of Interviews and artifact review

50% Review Period 20 days after receipt of 50% submission

50% EPIC Project Management Plan Briefing

Immediately following review period

90% Submission (Near Final Project Management Plan)

90 days after NOAA Program Management Briefing

90% Review Period 20 days after receipt of 90% submission

90% EPIC Project Management Briefing Immediately following review period

100% Submission 20 days after 90% NOAA Program Management Briefing

100% Review Period 20 day after receipt of 100% Submission

Finalization and submission 20 days after receipt of NOAA’s comments on 100% Submission

EPIC Community Project Management Plan Presentation

To be determined based on NOAA’s Scheduling;

most likely at the EPIC Annual Summer Workshop

5.2: Develop and document a cloud-based Unified Forecast System Weather Application

The Ccontractor shall:

5.2.1 Develop cloud-based Establish access of the cloud-based Unified Forecast System (UFS) weather application, source codes, and dependent libraries (or other open source services) using the UFSus UFS Github repository.iesfor the EPIC community users.

5.2.2 Populate necessary dependent data provided by through NOAA provided services andto support research and development of UFS weather application on Cloud Platforms (e.g., AWS, Google and Azure) infrastructure. NOAA anticipates two releases of the UFS weather application during this Task Order performance period. NOAA will provide dependent data for each release, and may provide additional data for re-population between code releases. It is unknown at this time whether or how frequently this may be required.

5.2.3 Work with the communityDistribute a on the design and development of workflow (i.e., CIME) for cloud-based UFS applications that are portable across cloud and on-premise platforms; while the emphasis of this Task Order is with the Weather Application, the workflow that is developed shall be applicable to UFS applications generally. on Github repositories.

5.2.4 Create a Pportable version of and assess the UFS Wweather Aapplication Unified Forecast System (UFS) on Cloud Platforms (e.g., AWS, Google and Azure). fitness for use.

5.2.5 Provide a containerized (e.g., docker, singularity, charliecloud, kubernetes) version of the UFS wWeather aApplication and provide and support container technology to improve developer productivity, performance, efficiency, and application portability. The base performance for a containerized UFS weather application is 7 minutes per forecast day which currently meets or exceeds the current performance of the operational model run on NOAA’s Weather and Climate Operational Supercomputing System (WCOSS).

5.2.6 Port UFS to cloud services for the purpose of UFS activities such as continuous integration, data services, model integration, reforecasts/reanalysis, and ensuring portability.Work with provided communications platform(s) for interaction within the UFS community that includes researchers, developers and users from NOAA, educational institutions, other federal agencies, and the private sector.

5.2.7 Work with the UFS community on the next public UFS weather application release. Each successful release will include new capabilities that are necessary to run the UFS end to end (starting from observations to product generation).

56.2.9 Define the community of users and define the pathways into the UFS repositories with the UFS governance structure..

56.2.11 Develop and maintain online documentation (English language), user’s guides, and developers guides (e.g., a wiki) of the UFS as living documents.

56.2.13 Generate tutorials as short videos (YouTube) and consistent webinars (Google Hangouts) on how-to run the UFS weather application on different cloud platforms.

56.2.14 Connect scientists with software engineers through online tutorials.

56.2.15 Conduct workshops at AMS and AGU during the period of performance for this task order award.

5.2.16 Conduct annual EPIC summer workshop focused on user engagement and outreach. Contractor shall provide a suitable space and workshop arrangements for 200 people and it must be approved by the NOAA EPIC Program Manager prior to scheduling; in the event of disagreements, the Contracting Officer shall prevail.

56.2.17 Conduct 2 code sprints and 2 hack-a-thons (minimum 2) as opportunities to work on bounded problem areas; build partnerships or even pair with other developers; and build camaraderie around the projects.

Contractor shallwill be able to provide a suitable space for these types of gatherings of up to 50 people and it must be approved by the NOAA Program Manager prior to scheduling.

56.2.18 Participate in the design of up to 8 meetings for strategic planning, scientific workshops, tutorials, and/or hackathons related to the UFS.

Develop and document cloud-based UFS weather application on 3 CSPs

90 days after award

Populate data for running cloud-based UFS weather application on 3 CSPs

Concurrent with above activity

Define, develop, and document the workflow and any updates for cloud-based UFS weather application on 3 CSPs

Concurrent with above activity

Work with UFS community to design, build, test, the next public release of the UFS Weather Application

180 days after award

Work with UFS community to design, build, test, the next public release of the UFS Weather Application

360 days after award

5.3 User Support

The Contractor shall:

5.3.1 Work with provided communications platform(s) for interaction within the UFS community that includes researchers, developers and users from NOAA, educational institutions, other federal agencies, and the private sector.

5.3.2 Develop and maintain online documentation (English language), user’s guides, and developers guides (e.g., a wiki) of the UFS as living documents.

5.3.3 Generate tutorials as short videos (YouTube) and consistent webinars (Google Hangouts) on how-to run the UFS weather application on different cloud platforms.

5.3.4 Connect scientists with software engineers through online tutorials.

5.3.5 Design and monitor dashboard (highlighting success metrics such as top contributor and promoting contribution of innovation across the community) for users of UFS weather application.

5.3.6 Generate and maintain the inventory of the transitions of community code into NOAA applications (count; lines of code) as sample success metric.

5.3.7 Provide technical and user support for cloud-based UFS weather application and support transition of community-based UFS weather application to operations through well-defined R2O/O2R process.

5.4 Community Engagement

The Contractor shall:

5.4.1 Conduct workshops at AMS and AGU during the period of performance for this task order award.

5.4.2 Conduct annual EPIC summer workshop focused on user engagement and outreach. Contractor shall provide a suitable space and workshop arrangements for 200 people and it must be approved by the NOAA EPIC Program Manager prior to scheduling.

5.4.3 Conduct 2 code sprints and 2 hack-a-thons as opportunities to work on bounded problem areas;

build partnerships with other developers; and build camaraderie around the projects. Contractor shall provide a suitable space for up to 50 people and it must be approved by the NOAA Program Manager prior to scheduling.

5.4.4 Participate in weekly virtual meetings related to EPIC. In person meetings are required for workshops, conferences, and NOAA briefings. Assuming post-COVID activities are able to resume, we estimate 8-12 travel meetings annually.

6. DELIVERABLES

Deliverable Frequency Medium/Format/# of Copies

Submit To

Kick Off Meeting Within 10 business days of award.

At location specified by

NOAA

COR/TM

Project Management Plan for Sample Task Order One

In accordance with the scheduled specified at 5.1.5.

MS Word Report delivered by way of email

EPIC Program Manager (PM) and COR

Provide an overall assessment of the UFS cloud related activities and develop a Pproject

In accordance with the scheduled specified at 5.1.5.Within 3-6

MS Word Report delivered by way of email

Review Meetings with

EPIC Program Manager (PM) and COR

Management Pplan for implementing the overall ojbectives of the EPIC Contract comprehensive support of all UFS applications months of award

NOAA at 50%, 90%, and 100% intervals at location specified by NOAA

Community presentation at EPIC Summer Workshop

Define, Ddevelop, and Ddocument, and release a Containerized Cloud-based UFS Weather Application (includes container version) on 3 CSPs

In accordance with the scheduled specified at 5.2.7

Online through UFS Portal and UFS Github Repository

Documentation online

Define, develop, document, and release Cloud-based UFS Weather Application (includes container version) on 3 CSPs

In accordance with the scheduled specified at 5.2.7Within 12 months of award

Online through UFS Portal and UFS Github Repository

Documentation online

Kick Off Meeting Within 10 business days of award

COR/TM

Monthly Activity Report

By the 15th of every month or with invoice?

via email PM/COR/TM

Weekly Status Report Once a week Meeting with PM or otherwise specified

PM/COR/TM

Level of Effort Spend Plan Within 10 business days of award; ongoing updated on a monthly basis by the 15th of the month via email PM/COR/TM

Task 1: Deliverables

6.1.1 Conduct overall assessment of cloud readiness on the current UFS applications from the existing level of effort. Assess which pieces of the UFS are ready for the cloud on three platforms (AWS, Google and Azure).

6.1.2 Assess performance of container vs. non-container version of the current release of the UFS.

6.1.3 Assess cloud readiness of Common Infrastructure for Modeling the Earth (CIME) workflows in three cloud environments (i.e., AWS, Google and Azure).

6.1.4 Port the UFS system on three primary (AWS, Google and Azure) cloud platforms and benchmark the performance on all platforms for UFS releases.

6.1.5 Create report on EPIC’s UFS project plan.

6.1.6 Review UFS open source codes for understanding and potential gaps for cloud HPC performance.

Task 2: Deliverables

6.2.1 Provide technical and user support for cloud-based UFS and support transition of community-based UFS code to operations through the R2O/O2R process.

6.2.2 Define the community of users and define the pathways into the UFS repositories with the UFS governance structure.

6.2.3 Provide a dashboard for users of UFS

6.2.4 Develop and maintain online documentation of the UFS as living documents.

6.2.5 Generate the inventory of the transitions of community code into NOAA applications (count; lines of code).

6.2.6 Provide user support services through the USF community online forum established by NOAA.

6.2.7 Provide documentation functions within collaborative GitHub repositories, web-based documentation, user’s guides, and developers guides (e.g., a wiki).

6.2.8 Generate tutorials as short videos (YouTube) and consistent webinars (Google Hangouts) on how-to run UFS on three cloud platforms (AWS, Google and Azure).

6.2.9 Connect scientists with software engineers through online tutorials.

6.2.10 Conduct workshops as routine in-person user-focused workshops and sessions/fora at meetings

(e.g., AMS and AGU).

6.2.11 Perform code sprints (minimum 2) and hack-a-thons (minimum 2) as opportunities to work on bounded problem areas; build partnerships or even pair with other developers; and build camaraderie around the projects.

6.2.12 Participate in the design and implementation of up to 8 meetings for strategic planning, scientific workshops, tutorials, and/or hackathons related to the UFS.

7. PERIOD OF PERFORMANCE

The Task Order Period of Performance (POP) will be one (1) year from date of award with one (1), one-year option period.

8. PLACE OF PERFORMANCE

The place of performance is to be determined (TBD) and shall be proposed by the contractor. Remote work shall be authorized for this requirement.

9. CONTRACT TYPE & RISK

This is a firm-fixed price (FFP) order.

This Task Order has been deemed as a Low-Risk Contract (Security Processing Requirements).

10. PERFORMANCE REQUIREMENT SUMMARY (PRS)

Required Service

PWS

Section Performance Standard

Method(s) of Surveillance

Incentive

Sample Task Order Project Management Plan

4.1 Submitted within 30

days of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

In alignment with the vision for EPIC, develop a project management plan that enables a cross-disciplinary team to work together to ensure that the UFS code is supported, easily maintainable, extensible and performance optimized; and recruit and develop a team of professional software, systems, infrastructure, data scientists, and computational engineers who will implement the plan

43., 5.1.1, 5.1.2, 5.1.3, 6.1.1, and 5.1.46.1.

Adherence to schedule specified at 5.1.5– due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Develop a platform-agnostic containerized version of the UFS Weather Application that can be used on Cloud HPC, meaning there is a documented process for running element(s) of the UFS on Cloud HPC via multiple vendors, and passes the graduate student test.

PWS 4.2

and 5.2.51

Adherence to software design requirements;

Functionality meets or exceeds NOAA’s and the weather community’s business and technical needs and completes in 7 minutes per forecast day;

Design accommodates ongoing software revisions and extensions with minimal effort;

Documentation is complete, accurate, and understandable for use by the weather community

Adherence to schedule

Review of design specification document (if applicable) by Program Manager or designee(s);

Potentially independent testing and verification of Contractor software by NOAA technical staff;

Periodic inspection;

Random monitoring;

Customer feedback;

Consultation with Program Manager, members of the weather community, and other subject matter experts

Possible Inclusion in Past Performance Review

Provide the Weather Enterprise with access to UFS operational model code, user-support services, and NOAA with a method for the secure ingest of community code into NOAA applications

PWS

4.3, 5.2.1, 5.2.2, 5.2.4, 5.2.6, and 5.2.8

Adherence to software requirements, design specification, and development methodology (with open source on Github and open data policies based in NOAA’s and other agency best-practices and policies);

Adherence to schedule

Potentially independent testing and verification of software by NOAA technical staff;

Approval by NOAA designated Change Management Board for deployment;

Random inspections and evaluation by NOAA technical staff/SME’s, under direction of Program Manager, COR, or both

Possible Inclusion in Past Performance Review

Align software engineers with scientists to: (1) create well-documented cloud-friendly code;

and (2) free scientists to perform research.

Communicate short-term successes.

PWS 4.4

5.1.2, 5.2.8, and 5.2.93

Adherence to software requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs;

Adherence to schedule

Random inspections;

Customer feedback;

Consultation with Program Manager, members of the weather community, and other subject matter experts

Possible Inclusion in Past Performance Review

Provide a user-support team with individual accountabilities for answering questions and updating the frequently-asked questions.

PWS 4.5

and 5.34

Access and usability by all community members, regardless of the level of expertise and nationality

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Describe and define the optimal mechanisms for supporting and accelerating the R2O/O2R process through this contract

PWS

5.1.4

Adherence to schedule – due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Align with science goals of the UFS Applications

PWS

5.1.5

Adherence to schedule – due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Assess and provide a comprehensive report on UFS cloud activities to avoid duplication of effort

PWS

5.1.6

Adherence to schedule – due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Work with parties to define the vetting process of UFS code back into the master

PWS

5.1.7

Adherence to software requirements, design specification, and development

100% inspection by the PM and COR as well as consultation with relevant

Possible Inclusion in Past repository methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs;

Adherence to schedule – due within 3-6 months of award

NOAA technical staff Review

Assess the UFS Repository Management Plan and the UFS Research to Operations Plan to provide feedback to the UFS community

PWS

5.1.8

Adherence to schedule – due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Survey, assess, and provide a report on current UFS related activities within NOAA utilizing cloud services such as continuous integration (CI), continuous development (CD), data services, debugging, model integration, reforecasts/reanalysis, and retrospectives to avoid duplication of effort and identify gaps

PWS

5.1.9

Adherence to schedule – due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Provide advice and best practices to ensure a UFS code that is innovative, modern, maintainable, extensible, and performance optimized on modern computer architectures. Include best practices on CI/CD approaches that automatically build and test all proposed code changes in its testing infrastructure

PWS

5.1.10

Adherence to software requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs;

Adherence to schedule – due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Work with relevant NOAA labs and centers (i.e., EMC) to develop CI/CD process

PWS

5.1.11

Adherence to software requirements, design specification, and development methodology

Consultation with relevant NOAA technical staff

Possible Inclusion in Past

Ensure the hierarchical testing framework works across all UFS modeling components

PWS

5.1.12

Adherence to software requirements, design specification, and development methodology;

Functionality of framework meets or exceeds NOAA’s and the weather community’s needs

Review of design specification document (if applicable) by PM or designee(s);

Potentially independent testing and verification of Contractor software by NOAA technical staff;

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Provide a description of how to design and implement an enterprise-level agile approach to accelerate software testing and integration of the UFS through secure ingest procedures and accelerating the R2O/O2R process

PWS

5.1.13

Adherence to software requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs;

Adherence to schedule – due within 3-6 months of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Develop Establish access of the cloud-based Unified Forecast System (UFS) weather application ), source codes, and dependent libraries (or other open source services) on Github repositoryrepositories for the EPIC community users

PWS

5.2.1

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Populate necessary dependent data provided by through NOAA to support research and development of UFS weather application on Cloud Platforms (e.g., AWS, Google and Azure)provided services and infrastructure

PWS

5.2.2

Complete, accurate, reliable and functional data management, datasets, and application operation

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Work with the community on Distribute a workflow (i.e., CIME) for cloud-

PWS

5.2.3

Functionality of workflow meets or exceeds NOAA’s and the weather community’s

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in based UFS on applications that are portable across platformsGithub repositories needs Review

Create a portable version ofPort and assess the Weather Application Unified Forecast System (UFS) on Cloud Platforms (e.g., AWS, Google and Azure) fitness for use

PWS

5.2.4

Adherence to software requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Provide container version of the UFS Weather Application and provide and support container technology to improve developer productivity, performance, efficiency, and application portability

PWS

5.2.5

Adherence to software requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs; base performance for a containerized UFS weather application is 7 minutes per forecast day

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Port UFS to cloud services for the purpose of UFS activities such as continuous integration, data services, model integration, reforecasts/reanalysis, and ensuring portability

PWS

5.2.6

Adherence to software requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Work with provided communications platform(s) for interaction within the UFS community on the next public UFS weather application releasethat includes researchers, developers and users from NOAA, educational institutions, other federal agencies, and the private sector

PWS

5.2.7

Include new capabilities necessary to run the UFS end to end (starting from observations to product generation)Timely, relevant, effective, and responsive interactions with the UFS community

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past

Provide a dashboard for users of UFS weather application

5.2.9 Highlights success

metrics such as top contributor and promoting contribution of innovation across the community

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Generate and maintain the inventory of the transitions of community code into NOAA applications (count; lines of code

5.2.10 Within 30 days of each

code release accepted by NOAA into its operational forecast suite

Periodic inspection Possible Inclusion in Past Performance Review

Work with provided communications platform(s) for interaction within the UFS community

5.3.1 Timely, relevant, effective, and responsive interactions with the UFS community

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Online documentation (English language), user’s guides, and developers guides (e.g., a wiki) of the UFS as living documents are developed and maintained

5.3.2 Documentation is

developed and kept current

Periodic inspection;

Customer feedback

Possible Inclusion in Past Performance Review

Tutorials as short videos (YouTube) and consistent webinars (Google Hangouts) on how-to run the UFS weather application on different cloud platforms are developed

5.3.3 Are developed and kept

current

Periodic inspection;

Customer feedback

Possible Inclusion in Past Performance Review

Earth scientists are connected with software engineers through on-line

5.3.4 Are developed and kept

current

Periodic inspection;

Customer feedback

Possible Inclusion in tutorials Review

Regularly and effectively engaged in workshops, code sprint, hackathons, tutorials, and other community engagement events

5.4 Consistent engagement as

detailed in the Task Order

Periodic inspection;

Customer feedback

Possible Inclusion in Past