EODT Q and A_SAM.Gov_FINAL.pdf

BROAD AGENCY ANNOUNCEMENT (BAA): Earth Observations – Digital Twin

Questions & Answers (Q&A) # Questions Received Government Response

Will sensor data be provided via an Application JSON like NASA extraterrestrial information, or will a given sample datasets of the response 200- API JSON result be provided?

The plan as we envisioned it is that sensor data will be provided in its original format, and provide readers associated with them. So not in a JSON app as asked about. We are unaware of any environmental observational or model output data available in JSON. We expect data will be provided in formats such as netcdf, hdf, BUFR, and grib/grib2, primarily.

2 Will the project involve any classified data? The project will not include classified data.

Satellite data anomaly detection: Does this refer to outages in the data feed or anomalies within the data products themselves?

This refers to anomalies in the data products themselves.

Calibration monitoring and correction: Would the performer be required to detect and correct for sensor drift over time?

The expectation is that part of the assessment and visualization components would include the aspect of generating time series that might highlight drifts over time.

To what extent should the performer expect to integrate existing algorithms versus developing additional?

The performer is expected to integrate algorithms that exist already within NOAA. It is not excluded however that the performer might generate or implement publicly available (or published) algorithms.

It is understood that the desired goal is to create a global DT, but some sensors in Section F do not provide global coverage. Can we assume that this initial effort would focus on a spatial subset and if so, can a region be specified?

The goal is indeed to have a global EO-DT. The fact that some sensors might have only a regional coverage will mean that these sensors will only refresh certain parts of the globe with their observations. There would be other sensors that cover other parts of the globe. The DT will still be global. That being said, the DT should be tailorable by the user seeking access to it, using the tailoring criteria listed in the BAA.

Is it expected that the price portion of the white paper include estimates of compute costs or will computing resources be provided by the government?

Compute resources will not be provided by the Government.

The offerors will be expected to use their own compute resources. NOAA will use its own IT resources for testing and validation.

Will NOAA provide the hardware or cloud platform for the Demonstration and/or will they provide any specs on the computer hardware or cloud platform?

To offer maximum flexibility, NOAA is not putting hard constraints on the specs of the computer hardware or cloud platform that the EO-DT should run on, since this is a demonstration project. We will, however, plan to provide guidance and recommendations to the offeror, through the Government team, during the execution of the project, in order to maximize the chances that the final product will be compatible with the NOAA environment. The offeror is expected to use its own hardware and/or cloud resources to develop the EO-DT. NOAA will use its own infrastructure to test and validate.

Also, does the BAA require a small business based on the below wording? “Small business subcontracting plans will be required from any large business proposals selected for negotiation or further solicitation/award in accordance with the applicable Federal Acquisition Regulation (FAR), Commerce Acquisition Regulation (CAR), Commerce Acquisition Manual (CAM) and NOAA provisions/clauses/regulations.”

If your white paper ROM exceeds $750K, and is selected to move to Stage 2 of the BAA process, you will receive an invitation for proposal that will include clause 52.219-9 - Small Business Subcontracting Plan. The inclusion of this clause in all solicitations over $750K is mandatory, where it has been determined that subcontracting possibilities exist.

10 Is there an estimated award time frame/ contracts start date? Any information you can provide will be helpful.

The anticipated award date for any study contracts resulting from this BAA is estimated at September 7, 2022. Please refer to the updated Community Day slide deck included in the Amendment 3 posting for a more detailed anticipated timeline.

11 Are you open to ownership / licensing approaches such as the following?

The question you've submitted is a good one, and one the Government team has devoted quite a bit of

1) Ownership of code by the government, with a full perpetual license back to the partner?

2) Ownership of the code by the partner, with a full perpetual license to the government?

thought/discussion to since its submission. We understand your concerns about not limiting your company's ability to engage in future R&D in this area, and have taken concerns like yours into consideration as we revised this section of the BAA to be clearer about what code ownership arrangements NOAA envisions. The updated language provided in Section 2, Sub-section C. b. (pgs.9-10) should provide additional clarification.

Can we address the goals of the Digital Twin using Geo Sounder data? Or do we need to explicitly use a dataset on the list?

The Digital Twin is envisioned to have representation of the different Earth system components, including atmosphere, ocean, space weather, etc., and that is why Section F proposes a list of sensors from representative satellites that are known to produce information in the different components of the Earth system (see Table E). The offeror should feel free to suggest alternative satellites and/or sensors, to cover alternative variables (from those listed in Sections E and F), but the preference is to achieve a digital twin that covers/represents all Earth system components with few variables.

For the BAA, we saw the following text:

"It is required for all prospective offerors and partners from industry, academia, and Government to pre-register for the Virtual Community Day Plenary Sessions."

Does this mean you have to have pre-registered and attended the Virtual Community Day sessions in order to submit a white paper for consideration?"

No, attendance at the Community Day wasn't a prerequisite for submitting a white paper in response to the BAA. We were trying to ensure that our platform for the virtual community day would be able to support the number of people calling in, which is why we requested people to preregister for the event.

The BAA solicitation states the US government will own all code developed for this project. Can the Government clarify that this statement only applies to solutions proposing software development, and does not apply to commercial products that are proposed?

After consideration of this and similar questions, the Government team has revised Section 2, Sub-section C. b.

(pgs.9-10), to be clearer about what code ownership arrangements NOAA envisions.

In regard to the 10 minute processing requirement, can the Government provide guidance on what triggers the 10 minutes to start?

The offeror should feel free to propose a trigger mechanism for generating the digital twin. One option suggested (as an example) is that the digital twin execution (an occurrence) would be generated every hour or half hour (or at a cadence defined by the user) at which time, the system would look for available data that were collected or that exist for around that time (within a +/- interval) and use that data for processing the digital twin occurrence. Recall that input data will include satellite data, ground based data, as well as previous forecast fields in order to generate the digital twin analysis with as little gap as possible, and with data fusion/assimilation techniques to merge multiple sources of data within single grids.

Can the Government describe how data will be made accessible to industry? For example, will industry be required to set up data feeds directly from source systems, or will NOAA re-broadcast data to vendors?

Sample data will be provided by the government for 'canned testing.' There is no need to set up real time data feeds since this is not meant to be an operational system, but rather, a proof of concept. However, if the offeror determines that it is feasible and there is value to set up a routine data feed, and process the data continuously, the NOAA technical team will support the offeror to set it up, should a study be awarded.

With new requirements recently made available to industry and limited time to administer teaming agreements, will the Government allow Offerors to propose teammates with the RFP submission if down-selected?

NOAA hopes that the extension of the white paper due date until June 7, 2022 will allow sufficient time for teaming arrangements to be finalized. However, if the offeror is unable to have teaming arrangements fully established at the time of white paper submission, they may indicate in their white paper that they are in the process of establishing a team. Anticipated team members and their role in the proposed study should be included in the white paper, but the team must be finalized by proposal submission time, if the offeror is selected to move to Stage 2 of the BAA process.

18 Will the same Tech Evaluation Board will also evaluate the cost and other non-technical aspects?

No. The price will be evaluated by the Acquisition team with feedback on technical on cost reasonableness from the Technical Evaluation Board.

19 Does this EO-DT focus on satellite data only? No. Please reference BAA Section F for all the sensors included, which includes ground based sensor data.

20 Is there any minimum Technology Readiness Level (TRL) for proposed technologies related to this BAA? No

21 Can you discuss some of the US Government agencies with which NOAA's wants to partner?

We have no preference. See page 4 of the BAA for some restrictions on who can compete.

Can you speak to integrating government provided data readers & algorithms? (Ex. types of source code, compiled binaries, or APIs etc.)

There will be an internal government team that will provide sample files, source codes, and readers to read the sample data.

23 From a platform compatibility perspective, what are the “major NOAA systems” to consider? Two systems are listed in the BAA as examples: GSI and UFS.

24 Does this EO-DT prefer to develop on existing DT framework/platform or from scratch/ground-zero? There is no preference.

Can you share more about what you hope the system will do for interpolation and correction of the data using ML?

For correction, it’s correcting anomalies. The solution should be able to detect anomalies and remove them. For interpolation, it’s when there are gaps. The solution should be able to combine what information exists and forecasts to fill the gap.

The scope of effort is quite comprehensive (encompassing ML/AI, data assimilation/data fusion, modeling, software system architecture). What is the expected time length of the projects in this activity?

Will proposals be selected such that there is team coordination?

If you want to put together a team, you can do that and submit the white paper as one entity. If you are selected to Stage 2, a contract will be awarded to the Prime and the Prime will manage the team. For companies less familiar with NOAA, NOAA will provide tactical guidance, such as if there is an existing AI algorithm, we’ll provide it.

27 Can you speak more about the 10 min timeline for the Digital Twin?

This is the amount of time we are targeting for a full global processing.

Is the goal to visualize the NOAA satellite data individually and a fused product, or only the data fused product globally?

Visualization is about the Digital Twin output, not input. The intent is to visualize the output of the digital twin (not the inputs). The output should be flexible: global or as tailored by the DT user.

Does NOAA want a Digital Twin of the satellites and their ability to observe Earth Science phenomenology or just an Earth model that incorporates the various data archives for ML purposes?

The first. A Digital Twin of the satellites and their ability to observe Earth Science phenomenology. We are looking for DT for observations, not modeling. The Digital Twin will provide “moving” output for the past 15 days and archiving of earlier observations is not expected.

Is there an expectation to only use NOAA affiliated data? Also is the ML usage only related to Earth Observation super-resolution, anomaly detection/correction? Is there any scope of using ML to build models for tracking environmental prognostics?

It does not have to be only NOAA data. The design should be flexible so it can later be expanded to more data. We are looking for earth observations at the original resolution. The Digital Twin is not for modeling predictions. It is a Digital Twin of current environmental observations.

Will NOAA experts be responsible for providing labeled data for training ML algorithms (such as anomaly detection)?

In AI, “labeled data” has specific meaning, so the answer is “no”. The NOAA Team will guide you through what is an anomaly in the data.

If we propose a solution based on an existing product -would the government need to own the code of initial solution as well?

Should a proposed solution build upon existing proprietary software, the offeror must identify a licensing approach that ensures that the US Government has long term rights to use and modify that software, and to authorize others to use and modify that software. Please see the updated language provided in Section 2, Sub-section C. b. (pgs.9-10) for additional details.

33 Would we need to provide data rights?

The Government will own the data and we can provide a non-exclusive right to the data back to the Offeror. If desired, the Offerors can propose something different in the White Paper.

The BAA mentions interpretability/explainability for ML models. Could you please elaborate on that since there are no predictions to be made using ML.

AI/ML algorithms have techniques to ensure the results are interpretable and explainable that we encourage the offerors to explore. This is meant to build confidence in the AI algorithms that are generated as part of this project. Another way to ensure explainability is to account for the option to include physical constraints in the ML/AI/CV algorithms.

Is the idea that the 10 minute processing would eventually supplement or replace data assimilation for the weather models?

This study is exploratory, but the idea is to infuse efficiency in data fusion and data assimilation for future model input.

36 Is there a restriction on using proprietary ocean sensing data? Or all data has to be open sourced? Please see Section 2, Subsection F of the BAA.

37 Will Simplified Acquisition Procedures be allowed? Can it be requested?

That will depend on the proposed price in the white paper.

Simplified Acquisition Procedures can only be used under certain dollar thresholds.

38 Does NOAA envision giving multiple awards? We envision one or multiple awards, based on the technical merits of the white papers submitted, and available funding.

39 Will NOAA provide the environment/IT infrastructure for the software development?

No, the Vendor should provide the IT infrastructure/environment.

40 Can NOAA staff be Co-I on the proposal? This may present a personal financial conflict of interest.

41 Can we talk to NOAA scientists while preparing our whitepaper?

The offeror should consider whether involving NOAA staff in the formulation of its white paper would create an issue with personal conflicts of interest that could impact its eligibility for award.