R.2 -- Attachment 4 - 1305M320RNRMN0002 - PWS Sample Task Order One - Amendment 02 - Clean Copy.pdf

ATTACHMENT 4 – R2 EARTH PREDICTION INNOVATION CENTER (EPIC)

Department of Commerce (DOC) National Oceanic and Atmospheric Administration (NOAA)

Office of Oceanic and Atmospheric Research (OAR) Office of Weather and Air Quality (OWAQ)

SAMPLE TASK ORDER ONE

PERFORMANCE WORK STATEMENT (PWS)

Earth Prediction Innovation Center (EPIC)

ATTACHMENT 4

1305M320RNRMN0002

REVISION 2

Contents

1. INFORMATION 3

2. BACKGROUND 3

3. OBJECTIVES 4

4. SCOPE OF WORK 5

5. TASKS 5

6. DELIVERABLES 9

7. PERIOD OF PERFORMANCE 10

8. PLACE OF PERFORMANCE 10

9. CONTRACT TYPE & RISK 11

10. PERFORMANCE REQUIREMENT SUMMARY (PRS) 11

1. INFORMATION

This Performance Work Statement (PWS) outlines the Earth Prediction Innovation Center (EPIC) requirements for obtaining software engineering and scientific/technical support services on cloud infrastructure and to provide robust user support for easy access to the latest version of the National Oceanic and Atmospheric Administration (NOAA) Unified Forecast System (UFS) which is a community-based, coupled, comprehensive Earth modeling system. The efforts are intended to accelerate community-developed scientific and technological enhancements into the operational applications for Numerical Weather Prediction (NWP).

NOAA is leading the development of the EPIC and the associated scientific work practices that will change the way community integrated earth system modeling is achieved in the United States. EPIC will continuously integrate and develop the community-modeling environment with all partners invested in EPIC through a community-based, coupled, comprehensive Earth modeling system (i.e. UFS) to improve NWP and to accelerate the Research to Operations and Operations to Research (R2O/O2R)R2O/O2R paradigm. The UFS numerical applications span local to global domains and predictive time scales from sub-hourly analyses to seasonal predictions. It is designed to support the Weather Enterprise and to be the source system for NOAA's operational NWP applications.

The Government will rely upon the Contractor to provide EPIC with Scientific and Technical Support Services directly supporting EPIC as outlined in this PWS. The resultant contract task order award will be managed through the Office of Weather and Air Quality (OWAQ) within NOAA’s Oceanic and Atmospheric Research (OAR) Line Office.

2. BACKGROUND

The Department of Commerce (DOC), NOAA, and OAR’s OWAQ supports world-class weather and air quality research that leads to, among other things, advancements to NOAA’s NWP that ultimately saves lives, reduces property damage, and enhances the national economy.

In pursuit of its vision and mission, OWAQ works closely with the National Weather Service (NWS) to help develop and transition to NWS’ operational suite of forecast models weather and air quality research, including hurricanes, severe thunderstorms, heavy precipitation, and air pollution. Additionally, OWAQ utilizes social science to learn how to deal with the uncertainties weather presents and to inform its engagement and communication with researchers, funders, and the public.

Furthermore, OWAQ selects and funds research that supports and fosters collaborations within NOAA's research laboratories and across the weather enterprise (i.e., NOAA, other Federal agencies and entities, state and local governments, academia, and the private sector).

The Weather Research and Forecasting Innovation Act of 2017 (WRFIA, Public Law 115-25) directs NOAA to prioritize improving weather data, modeling, computing, forecasting, and warnings for the protection of life and property and for the enhancement of the national economy. The authorizing language for EPIC, in the National Integrated Drought Information System Reauthorization Act of 2018 (NIDISRA, Public Law 115- 423), calls for NOAA to accelerate community-developed scientific and technological enhancements into the operational applications for NWP, which requires “advancing weather modeling skills, reclaiming and maintaining international leadership in the area of numerical weather prediction, and improving the transition of research into operations.”

The devastating impacts of hazardous events on life, property, and the national economy can be mitigated with accurate forecasts. To meet this need while reclaiming global leadership in weather prediction, NOAA must serve the nation by building a community-modeling environment for innovative, integrated earth system modeling.

EPIC is the catalyst for accurate weather forecasts in the near-term and integrated earth system modeling in the long-term. In the near-term, this means developing NWP skills in the UFS. In the long-term, this means expansion to other operational model applications such as high-resolution convective-allowing models and fully coupled subseasonal to seasonal, water and ocean forecast system. EPIC is working to fulfill its mandate and to exceed customer expectations. For additional information on EPIC, see https://owaq.noaa.gov/Programs/EPIC and on UFS, see https://ufscommunity.org/.

Moving forward, the terms develop or development is in reference to software development. Software development is the process of conceiving, specifying, designing, programming, documenting, testing, and bug fixing of applications or software components related to the UFS. Development may include research, new development, prototyping, modification, reuse, re-engineering, maintenance, or any other activities that result in new/updated software products for the UFS.

3. OBJECTIVES

The objective of this first task order is to provide NOAA with an assessment and project management plan for successful implementation of the Statement of Objectives for the overall EPIC Contract and for supporting and building the UFS applications. The contractor will provide EPIC with affordable and effective Professional Scientific and Technical Support Services that enable EPIC to meet its community mission requirements for making UFS accessible as an open source system with access through GitHub repositories to meet R2O/O2R functions. Additionally, this task order will build the UFS codes and dependent software on Cloud ecosystems using platforms such as AWS, Google, and Azure with a focus on building a system that is interoperable and portable from Cloud to on premise High Performance Computing (HPC) architectures. The contractor will integrate into the UFS governance structure to provide user/customer support for the cloud-based UFS system and HPC platforms relevant for the benefit of the wider EPIC and UFS community.

The primary areas of focus of this task order are described below:

● Provide an overall assessment of the UFS cloud related activities and develop a project management plan for the EPIC Contract including comprehensive support of all UFS applications.

● Develop a community-based UFS weather application that runs on Cloud HPC with robust user support, easy access to the latest version of code, input and output data; a code repository maintained under version control software (Github); thorough, understandable documentation; user-friendly workflow; software infrastructure; tutorials, workshops, help desk services; and ensuring that developers are involved in user support.

● Communicate broadly EPIC activities that deliver short-term success online and publicly at conferences

The success of this contract is solely dependent on integration of EPIC into the UFS infrastructure and governance to support community-based coupled Earth system model development by providing the community of users with a modern and flexible UFS framework that works on Cloud and can easily transition to conventional HPC owned by UFS partners. It’s imperative to engage the extensive research enterprise in the https://owaq.noaa.gov/Programs/EPIC https://ufscommunity.org/

U.S. for EPIC to succeed. EPIC will create an environment to more efficiently and effectively translate research advances into operational outcomes. Ultimately, EPIC’s support of the UFS will enable NOAA to simplify its production suite of forecasting models from a great many independent systems, each of which has to be improved and maintained, to a seamless coupled Earth modeling system.

4. SCOPE OF WORK

This Contractor shall:

4.1 Develop a Project Management Plan to fulfill the requirements of this Task Order and develop a second Project Management Plan as a part of this Task Order to implement the full scope of the EPIC Contract.

4.2 Develop a platform-agnostic containerized version of the UFS weather application that can be used on Cloud HPC, meaning there is a documented process for running element(s) of the UFS weather application on Cloud HPC via multiple vendors, and passes the graduate student test (the Graduate Student Test means that a student can easily: access code, run code, change code, test code for correct operation, and evaluate code with standard diagnostic packages; obtain code documentation, user support, and training; and understand what is needed for their code changes to transition to operations).

4.3 Provide the Weather Enterprise with access to UFS weather application via regular code releases and work closely with NOAA on a method for the secure ingest of community code into NOAA applications (with open source on Github and open data policies based in NOAA’s and other agency best-practices and policies).

4.4 Align computer scientists and software engineers with earth scientists to: (1) create well-documented cloud-friendly code; and (2) free earth scientists to perform research. Communicate short-term successes.

4.5 Provide solutions for accelerating the research to operations pipeline and supply a user-support team with individual accountabilities for answering questions and updating the frequently-asked questions.

The primary focus of this service will be to ensure access and usability by all community members, regardless of the level of expertise and nationality.

NOAA’s current capacity to provide operational and research-related HPC is a limitation to the advancement of numerical weather prediction. The transition of research and development to a cloud environment has the potential to dramatically improve NOAA’s capacity. It is NOAA's intention with EPIC to fully test the limits of this possibility, consequently, the plan that is developed under this RFP shall incorporate this.

5. TASKS

5.1 : Assessment of R2O/O2R of the Current and Future Unified Forecast System

The Contractor shall:

5.1.1 Describe your vision to meet the mandates of the WRFIA and NIDIS reauthorization with respect to EPIC to accelerate community-based scientific and technological enhancements into the advanced UFS applications through engagement with the Weather Enterprise and in alignment http://ufs-dev.rap.ucar.edu/index.html#/science/gst with the EPIC Strategic Plan and Governance Principles (currently in draft form).

5.1.2 Recruit and establish a team of professional software, systems architecture, infrastructure, data scientists, and computational engineers to execute this Performance Work Statement who will work directly with physical scientists in an integrated, semi-virtual, cross-disciplinary environment to ensure that the UFS is supported, easily maintainable, extensible and performance optimized.

5.1.3 Define and develop a full project management plan to include project scope management plan, milestones, schedule management plan, cost management plan, risk management plan, risk register, release management plan, quality management plan, communication management plan, and staff management plan necessary to implement the vision outlined in section 5.1.1. in conjunction with the Government’s objectives as outlined in the EPIC Statement of Objectives.

5.1.4 Project Management Plan, specified at 5.1.3 above, shall include the following elements/themes:

5.1.4.1 Conduct an assessment that describes and defines the optimal mechanisms for supporting and accelerating the R2O/O2R process at NOAA..

5.1.4.2 Define the community of users and define the pathways into the UFS repositories with the

UFS governance structure..

5.1.4.3 Describe how you will enable community based research and development through public releases of the UFS in alignment with the UFS science goals, strategic implementation plan, and EPIC Strategic Plan.

5.1.4.4 Conduct an assessment of UFS cloud activities to avoid duplication of effort and assess the UFS run on the cloud for three platforms (AWS, Google and Azure).

5.1.4.5 Assess and develop plan for agile development and Continuous Integration Continuous Deployment (CI/CD) pipeline, secure ingest, data services, cloud storage, debugging, model integration, reforecasts/reanalysis, and retrospective of UFS code back into the UFS repository.

5.1.4.6 Conduct a review and assessment of the current UFS planning documents (for example Repository Management Plan and UFS Research to Operations Plan) to ensure optimal success and provide feedback to the UFS community.

5.1.4.7 Conduct an assessment and evaluation of the current UFS communications platform and determine if it is optimal, given available alternatives, for interaction within the UFS community that includes researchers, developers and users from NOAA, educational institutions, other federal agencies, and the private sector.

5.1.4.8 Conduct an assessment of UFS code and published coding standards to determine whether these are sufficient to ensure that the UFS code is innovative, modern, maintainable, extensible, and performance optimized on modern computer architectures. Include best practices on CI/CD approaches that automatically build and test all proposed code changes in its testing infrastructure.

5.1.4.9 Conduct an assessment on the hierarchical testing framework for the UFS applications and modeling components.

5.1.4.10 Conduct an assessment on community-based and operational workflows (i.e. Common Infrastructure for Modeling the Earth (CIME), ecFlow, and Rocoto) running on cloud environments.

5.1.4.11 Assess cloud readiness of Common Infrastructure for Modeling the Earth (CIME) workflows in three cloud environments (i.e., AWS, Google and Azure).

5.1.4.12 Provide an assessment on current and future needs for community engagement to build a strong community user base for EPIC.

5.1.5 Finalize assessments and submit report on the EPIC Contract project management plan according to the following schedule:

EPIC Contract Project Management Plan Development Schedule:

Community Interviews and Artifact Review 90 days after award

50% Submission (Updated Project Management Plan Summary)

30 days after completion of Interviews and artifact review

50% Review Period 20 days after receipt of 50% submission

50% EPIC Project Management Plan Briefing

Immediately following review period

90% Submission (Near Final Project Management Plan)

90 days after NOAA Program Management Briefing

90% Review Period 20 days after receipt of 90% submission

90% EPIC Project Management Briefing Immediately following review period

100% Submission 20 days after 90% NOAA Program Management Briefing

100% Review Period 20 day after receipt of 100% Submission

Finalization and submission 20 days after receipt of NOAA’s comments on 100% Submission

EPIC Community Project Management Plan Presentation

To be determined based on NOAA’s Scheduling;

most likely at the EPIC Annual Summer Workshop

5.2: Develop and document a cloud-based Unified Forecast System Weather Application

5.2.1 Develop cloud-based UFS weather application source codes and dependent libraries (or other open source services) using the UFS Github repository.

5.2.2 Populate necessary dependent data provided by NOAA to support research and development of the UFS weather application on Cloud Platforms (e.g., AWS, Google and Azure). NOAA anticipates two releases of the UFS weather application during this Task Order performance period. NOAA will provide dependent data for each release, and may provide additional data for re-population between code releases. It is unknown at this time whether or how frequently this may be required.

5.2.3 Work with the community on the design and development of workflow (i.e., CIME) for cloud-based UFS applications that are portable across cloud and on-premise platforms; while the emphasis of this Task Order is with the Weather Application, the workflow that is developed shall be applicable to UFS applications generally.

5.2.4 Create a portable version of the UFS weather application on Cloud Platforms (e.g., AWS, Google and Azure).

5.2.5 Provide a containerized (e.g., docker, singularity, charliecloud, kubernetes) version of the UFS weather application and provide and support container technology to improve developer productivity, performance, efficiency, and application portability. The base performance for a containerized UFS weather application is 7 minutes per forecast day which currently meets or exceeds the current performance of the operational model run on NOAA’s Weather and Climate Operational Supercomputing System (WCOSS).

5.2.6 Port UFS to cloud services for the purpose of continuous integration, data services, model integration, reforecasts/reanalysis, and ensuring portability.

5.2.7 Work with the UFS community on the next public UFS weather application release. Each successful release will include new capabilities that are necessary to run the UFS end to end (starting from observations to product generation).

Develop and document cloud-based UFS weather application on 3 CSPs

90 days after award

Populate data for running cloud-based UFS weather application on 3 CSPs

Concurrent with above activity

Define (to explain or identify the nature or essential qualities of; describe), develop (to set forth or make clear by degrees or in detail), and document the workflow and any updates for cloud-based UFS weather application on 3 CSPs

Concurrent with above activity

Work with UFS community to design, build, test, the next public release of the UFS Weather Application

180 days after award

Work with UFS community to design, build, test, the next public release of the UFS Weather Application

360 days after award

5.3 User Support

5.3.1 Work with provided communications platform(s) for interaction within the UFS community that includes researchers, developers and users from NOAA, educational institutions, other federal agencies, and the private sector.

5.3.2 Develop and maintain online documentation (English language), user’s guides, and developers guides (e.g., a wiki) of the UFS as living documents.

5.3.3 Generate tutorials as short videos (YouTube) and consistent webinars (Google Hangouts) on how-to run the UFS weather application on different cloud platforms.

5.3.4 Connect scientists with software engineers through online tutorials.

5.3.5 Design and monitor dashboard (highlighting success metrics such as top contributor and promoting contribution of innovation across the community) for users of UFS weather application.

5.3.6 Generate and maintain the inventory of the transitions of community code into NOAA applications (count; lines of code) as sample success metric.

5.3.7 Provide technical and user support for cloud-based UFS weather application and support transition of community-based UFS weather application to operations through well-defined R2O/O2R process.

5.4 Community Engagement

5.4.1 Conduct workshops at AMS and AGU during the period of performance for this task order award.

5.4.2 Conduct annual EPIC summer workshop, lasting two to four days, focused on user engagement and outreach. Contractor shall provide a suitable space and workshop arrangements for 200 people and it must be approved by the NOAA EPIC Program Manager prior to scheduling.

5.4.3 Conduct 2 code sprints and 2 hack-a-thons as opportunities to work on bounded problem areas;

build partnerships with other developers; and build camaraderie around the projects. It is typical for these types of events to last 24-48 hours, roughly translated to two days, each Contractor shall provide a suitable space for up to 50 people and it must be approved by the NOAA Program Manager prior to scheduling.

5.4.4 Participate in weekly virtual meetings related to EPIC. In person meetings are required for workshops, conferences, and NOAA briefings. Assuming post-COVID activities are able to resume, we estimate 8-12 travel meetings annually.

6. DELIVERABLES

Deliverable Frequency Medium/Format/# of Copies

Submit To

Kick Off Meeting Within 10 business days of award.

At location specified by

NOAA

COR/TM

Sample Task Order Project Management Plan

Within 30 days of award

MS Word Report delivered by way of email

EPIC Program Manager (PM) and COR

Provide an overall assessment of the UFS cloud related activities and develop the EPIC Contract Project Management Plan for implementing the overall objectives of the EPIC Contract

In accordance with the scheduled specified at 5.1.5.

MS Word Report delivered by way of email

Review Meetings with NOAA at 50%, 90%, and 100% intervals at location specified by NOAA

Community presentation at EPIC Summer Workshop

EPIC Program Manager (PM) and COR

Define, develop, document, and release Cloud-based UFS Weather Application (includes container version) on 3 CSPs

In accordance with the scheduled specified at 5.2.7

Online through UFS Portal and UFS Github Repository

Documentation online

Monthly Activity Report

By the 15th of every month or with invoice?

via email PM/COR/TM

Weekly Status Report Once a week Meeting with PM or otherwise specified

PM/COR/TM

Level of Effort Spend Plan Within 10 business days of award; ongoing updated on a monthly basis by the 15th of the month via email PM/COR/TM

7. PERIOD OF PERFORMANCE

The Task Order Period of Performance (POP) will be one (1) year from date of award.

8. PLACE OF PERFORMANCE

The place of performance is to be determined (TBD) and shall be proposed by the contractor. Remote work shall be authorized for this requirement.

9. CONTRACT TYPE & RISK

This is a firm-fixed price (FFP) order.

This Task Order has been deemed as a Low-Risk Contract (Security Processing Requirements).

10. PERFORMANCE REQUIREMENT SUMMARY (PRS)

Required Service PWS Section Performance Standard Method(s) of

Surveillance Incentive

Sample Task Order Project Management Plan

4.1 Submitted within 30

days of award

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

In alignment with the vision for EPIC, develop a project management plan that enables a cross-disciplinary team to work together to ensure that the UFS code is supported, easily maintainable, extensible and performance optimized; and recruit and develop a team of professional software, systems, infrastructure, data scientists, and computational engineers who will implement the plan

4.1, 5.1.2, 5.1.3, and 5.1.4

Adherence to schedule specified at 5.1.5

100% inspection by the PM and COR as well as consultation with relevant NOAA technical staff

Possible Inclusion in Past Performance Review

Develop a platform-agnostic containerized version of the UFS Weather Application that can be used on Cloud HPC, meaning there is a documented process for running element(s) of the UFS on Cloud HPC via multiple vendors, and passes the graduate student test.

4.2 and 5.2.5

Adherence to software design requirements;

Functionality meets or exceeds NOAA’s and the weather community’s business and technical needs and completes in 7 minutes per forecast day;

Design accommodates ongoing software revisions and extensions with minimal effort;

Documentation is complete, accurate, and

Review of design specification document (if applicable) by Program Manager or designee(s);

Potentially independent testing and verification of Contractor software by NOAA technical staff;

Periodic inspection;

Random monitoring;

Customer feedback;

Consultation with Program Manager, Possible Inclusion in Past Performance Review understandable for use by the weather community Adherence to schedule members of the weather community, and other subject matter experts

Provide the Weather Enterprise with access to UFS operational model code, user-support services, and NOAA with a method for the secure ingest of community code into NOAA applications

4.3 and 5.2.

Adherence to software requirements, design specification, and development methodology (with open source on Github and open data policies based in NOAA’s and other agency best-practices and policies);

Adherence to schedule

Potentially independent testing and verification of software by NOAA technical staff;

Approval by NOAA designated Change Management Board for deployment;

Random inspections and evaluation by NOAA technical staff/SME’s, under direction of Program Manager, COR, or both

Possible Inclusion in Past Performance Review

Align software engineers with scientists to: (1) create well-documented cloud-friendly code;

and (2) free scientists to perform research.

Communicate short-term successes.

4.4 Adherence to software

requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs;

Adherence to schedule

Random inspections;

Customer feedback;

Consultation with Program Manager, members of the weather community, and other subject matter experts

Possible Inclusion in Past Performance Review

Provide a user-support team with individual accountabilities for answering questions and updating the frequently-asked questions.

4.5 and 5.3

Access and usability by all community members, regardless of the level of expertise and nationality

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Develop Unified Forecast System (UFS) weather application source codes, and dependent libraries (or other open source services) on Github repository

5.2.1 Functionality of

containerized code meets or exceeds NOAA’s and the weather community’s needs

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Populate necessary dependent data provided by NOAA to support research and

5.2.2 Complete, accurate, reliable and functional data management, datasets, and application

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance development of UFS weather application on Cloud Platforms (e.g., AWS, Google and Azure) operation Review

Work with the community on a workflow (i.e., CIME) for cloud-based UFS on applications that are portable across platforms

5.2.3 Functionality of

workflow meets or exceeds NOAA’s and the weather community’s needs

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Create a portable version of the Weather Application Unified Forecast System (UFS) on Cloud Platforms (e.g., AWS, Google and Azure)

5.2.4 Adherence to software

requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Provide container version of the UFS Weather Application and provide and support container technology to improve developer productivity, performance, efficiency, and application portability

5.2.5 Adherence to software

requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs; base performance for a containerized UFS weather application is 7 minutes per forecast day

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Port UFS to cloud services for the purpose of UFS activities such as continuous integration, data services, model integration, reforecasts/reanalysis, and ensuring portability

5.2.6 Adherence to software

requirements, design specification, and development methodology;

Functionality of containerized code meets or exceeds NOAA’s and the weather community’s needs

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Work with the UFS community on the next public UFS

5.2.7 Include new capabilities

necessary to run the UFS end to end (starting from

Periodic inspection;

Random monitoring;

Possible Inclusion in Past weather application release observations to product generation)

Customer feedback Performance Review

Provide a dashboard for users of UFS weather application

5.3.5 Highlights success

metrics such as top contributor and promoting contribution of innovation across the community

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Generate and maintain the inventory of the transitions of community code into NOAA applications (count; lines of code

5.3.6 Within 30 days of each

code release accepted by NOAA into its operational forecast suite

Periodic inspection Possible Inclusion in Past Performance Review

Work with provided communications platform(s) for interaction within the UFS community

5.3.1 Timely, relevant, effective, and responsive interactions with the UFS community

Periodic inspection;

Random monitoring;

Customer feedback

Possible Inclusion in Past Performance Review

Online documentation (English language), user’s guides, and developers guides (e.g., a wiki) of the UFS as living documents are developed and maintained

5.3.2 Documentation is

developed and kept current

Periodic inspection;

Customer feedback

Possible Inclusion in Past Performance Review

Tutorials as short videos (YouTube) and consistent webinars (Google Hangouts) on how-to run the UFS weather application on different cloud platforms are developed

5.3.3 Are developed and kept

current

Periodic inspection;

Customer feedback

Possible Inclusion in Past Performance

Earth scientists are 5.3.4 Are developed and kept Periodic inspection; Possible connected with software engineers through on-line tutorials current Customer feedback

Inclusion in Past Performance Review

Provide technical and user support for cloud-based UFS weather application and support transition of community-based UFS weather application to operations through well-defined R2O/O2R process

5.3.7 Consistent engagement as

detailed in the Task Order

Periodic inspection;

Customer feedback

Possible Inclusion in Past Performance Review

Regularly and effectively engaged in workshops, code sprint, hackathons, tutorials, and other community engagement events

5.4 Consistent engagement as

detailed in the Task Order

Periodic inspection;

Customer feedback

Possible Inclusion in Past Performance