KMNLP BAA_Questions and Answers.pdf

KM/NLP BAA: Vendor Questions & Answers (Q&A)

# Vendor Question Government Answer

Sections 1 and 3.2. Throughout the BAA document, submissions are referred to as proposals, however it is noted that, "there is no planning for activity beyond a study and demonstration". Could the government provide clarity on the next steps following "proposal" submissions?

Sometimes BAA's request white papers prior to proposals.

White papers are a summarized version of the idea the offeror is presenting to allow the Government to review prior to determining if the Government is interested in a full proposal on the topic. In this BAA, due to time constraints, the Government is requesting only full proposals. At this time, the Government does not know what will materialize from the studies conducted in this BAA. This would be determined at a later time.

Is the government looking for recommendations to procure potential tools separately, or looking for quotes services and solution quotes as part of the response to this BAA?

No, the government, at this time, is interested in the results of the study and will use those results in determining what 'next steps' if any will be taken following the completion of this BAA study.

Can the government clarify if there are any incumbents to contracts related to work described in this BAA?

While there are "incumbents" to the mentioned "Earth Observing Digital Twin" project, the KM/NLP BAA is not a continuation of that project, but is a separate BAA project that builds upon that EO-DT concept. So, there aren't any "incumbents" to the work described in this BAA.

For this study, is the government also looking for assistance in their records management systems?

The government is only looking for proposals that address needs outlined and described in the BAA in terms of study topic. The outlined needs in terms of Mesh grooming may provide an opportunity to propose studies that might develop and demonstrate capabilities related to certain aspects of records management.

Is the government utilizing any open source ontologies?

The government is currently primarily leveraging an RDF encoding of the Open Archival Information System (OAIS) concepts of Information Object and Information Package (SIP, AIU, AIC, DIP). Examples of other open source ontologies that may be expressed within this upper archetype-oriented ontology and that are of interest to the government are listed in the BAA.

Per the Community Day, it was stated that vendors can submit more than one proposal.

Does that mean that vendors will have to submit multiple Past Performance Questionnaires from the same project to NOAA for different proposals?

Correct, it should be treated as a separate new proposal and submitted with the same instructions as any new proposal.

Would the government consider allowing contractors to include an appendix with additional information outside of the technical 10 page limit?

No, only the 10 pages for technical are being requested.

Would the government permit a Table of Contents to be excluded from the technical 10-page limit?

The initial submission is limited to a cover page, one-page Quad Chart, and Proposals not to exceed 10 pages. This results in a submission packet not to exceed 12 pages (not counting past performance submission.)

Will solutions powered by in-house LLMs be preferred over paid models?

Both solution configurations will be considered and there is no explicit preference. It is important for vendors to consider the broader context of what the BAA is asking for when considering what aspects of the solution are specified.

If our proposed solution contains a proprietary or novel pattern for synthesizing RDF graphs with LLMS, does NOAA retain the right to provide the

No, the government treats each proposal as proprietary and will not share other submitted proposals with the selected contractor(s).

selected contractor (should it not be us) with our proposed, proprietary pattern?

Will the contract be awarded in full to a single vendor, or is it possible that partial awards will be made to multiple vendors? Also, if NOAA finds specific parts of a contractor’s solution preferable, is there a possibility of awarding only those parts of the proposal?

As stated in BAA section 3.2: "The Government anticipates awarding multiple contracts, but reserves the right to award no contracts, or a combination of contracts or contract options for project work based upon availability of funds and the needs of the NESDIS Portfolio."

The links to the NSF workshop recording and artifact referenced on Page 20 do not work; can NOAA please provide?

Yes. The BAA has been updated with the correct working links.

Can NOAA provide details as to what extent has the knowledge graph base been implemented already (in reference to the diagrams in Appendix A)?

The process framework (bounded by central dashed lines, including API-exposed process infrastructure and data/metadata buckets) has been developed (TRL 6/7) and is working toward both open release and operationalization.

The CI/CD components indicated through arrows going in and out of the process framework has been implemented (TRL 6). Many of the connected systems (ecosystem) have been implemented, although these are illustrative of the larger goals. The foundation model endpoint does not exist.

Are there other teams or contracting organizations that we will need to interact with to get a better understanding of their architecture and mock solution-specific metadata using the knowledge mesh API?

There are no requirements related to interactions with other teams or contracting organizations. The intent is to provide the working mesh software, software documentation including notebooks that demonstrate core capabilities, and example mesh data from existing implemented streams to the selected offerors. A NOAA team will be the primary interface to selected offerors during the period of performance.

What is the anticipated number of concurrent users? Would they be daily users or ad-hoc requests?

The mesh is intended to eventually support the entire process surface (orchestrated managed processes) of the NCCF and its Consumers/users. This includes processes developed and run by data managers, data/physical scientists, operators, and the public. Solutions should ideally qualify which users are intended for support for the specific solution.

When does NOAA plan to release the anticipated contract clauses for any resulting award?

Contract clauses and provisions will be released at the award stage.

Would the government consider a one-week extension with a due date of June 19 and June 14 for the past performance to ensure ample time for government clients to complete the past performance questionnaires?

We're unable to provide a one-week extension for the proposal. The Past Performance Questionnaire has been updated to match the due date of proposals. Please see the updated attachment in Amendment 0001.

Can one proposal cover only one of the two aspects listed in Phase C? Is the technology integration more important for Phase C than technology baseline?

Any given proposal can cover any part or combination of the described phases. A given proposal for example might include some aspect of phase A (construction) and phase C (exploitation). A given proposal should describe how the specific configuration supports successfully addressing the use case(s) described by the proposal.

Can you explain 'portability' a little more? Portability in the context of the BAA generally means some measure of the ability of a particular solution aspect to be transferred or transported to another configuration.

Depending on solution specifics this might relate to the ability of a model or software to be deployed in the same state without changes or with minor changes on different infrastructures (e.g. different cloud providers); it might refer to the ability to pass well defined processes across different agents in contexts of real-time federated process planning and execution; it might refer to the ability of a system to be independently understandable for long term preservation.

The specific meaning of portability should be qualified by the offeror when describing the particular aspects of their solution.

Do we have to use existing knowledge mesh? No, there is not a requirement to use the existing knowledge mesh and this is called out explicitly in the BAA. There is a desire to see solutions that at least align with the described architecture so that the outcomes of the study may be directly compared with the direction of the NCCF knowledge mesh. Solutions that do leverage the existing knowledge mesh architecture will provide more straightforward alignment for evaluation with and potential adoption by the NCCF developed and operationally oriented system.

Do we have access to NOAA users during the proposal phase?

The BAA team will consist of NOAA technical staff that may be considered a user subset. Proposals are encouraged to describe user testing intentions.

What type of multi-modal nodes are envisioned in this prototype?

The knowledge mesh is capable of storing and contextualizing (directly or through links) arbitrary data which might include non-textual data. NOAA and partner data sources that might be necessary or at least useful in supporting persona-driven request/response ecosystems contain things like video, image, audio, raster, vector, and other non-textual data along with textual data. It is valid for solution proposals to include use of these multimodal sources in process descriptions. It is also valid to describe multi-modal delivery of processed data in the Access Aid context for particular solution process(es) if the solution is proposing to facilitate use by a user persona that might more easily consume response in some form other than text.

How do we tie into the NOAA NCCF? Solutions developed within the guidelines described by the BAA should be naturally aligned with parallel NCCF development that is operationally focused simply by following the BAA intent. Solutions may be regularly evaluated to see if features that they have developed could be adopted by NCCF operational systems. At the end of the study, solutions may be evaluated by the NOAA team to see if they (in part or whole) would support NCCF integration. No direct NCCF integration by the study offerors is expected.

Is there a specific interest in NOAA fishery data or should we be looking at all NOAA sources?

The fisheries case is provided as an example for a specific use case and should be treated as an example use case by solution offerors. All NOAA data sources are 'fair game' for this BAA. Offerors should consider that JVP is a NESDIS program. NESDIS manages the satellite data program for NOAA and is also home to the national environmental archive, which ostensibly is responsible for handling all of NOAA's diverse data. With the Digital Twins tie-in, persona foci that might further support ongoing digital twins efforts such as the coastal zone digital twin, wildland fire digital twins, flooding and socioeconomic impact digital twins, or extreme tropical weather digital twins are of interest.

Would the use of proprietary tools for background processes such as orchestration disqualify a proposal if they do not impact the ability of making the final product available (for free) to the general public?

No, this is explicitly allowed in the BAA language.

We know that there’s existing work developing knowledge graphs at NOAA. Would access to those APIs or data be available for development of these demonstrations?

Yes, the software including API software will be made available to selected offerors. Example mesh managed data will be made available to selected offerors.

Will people who work on the project need to be US citizens? (note- the company is a US company, but some staff are not US citizens)

Non-US citizens may be a part of solution execution teams.

Will people working on this project need to be badged?

No, badging is not a requirement.

29 Will the slides be available later? Yes, the slides are available at sam.gov.

Can you describe the extent to which you're looking for demonstration of capabilities versus production ready systems?

The BAA process is explicitly about developing and executing studies. This BAA is framed in a way that will allow studies to align with separate but parallel operational work.

Demonstration capabilities are the focus of BAA, alignment with the architecture described in the BAA is the extent of production readiness that is intended.

Could we please get a one-week extension on the due date?

We are unable to provide a one-week extension. All proposals must be submitted by 12 June.

Has this opportunity been funded? Funding has been identified and is currently budgeted for this BAA. Note that this funding status can change.

Would the government consider licensing proprietary applications to some extent?

Licensing proprietary applications can be considered;

however, the desire is to have the KM/NLP BAA as open as possible.

Data sources, how many data sources will we be working with?

This depends on the particular solution(s) proposed by the offeror. Data sources should support the goals outlined in the BAA (demonstrating multi-user/persona request/response processing against a shared knowledge mesh ecosystem composed of potentially many advanced agents and technologies).

Do we have access to these NOAA data experts? The BAA execution will include regular touchpoints with a small team of NOAA technical experts.

Section of study phases, are they referenced the same as the study options or are they referencing something else?

Study phases and study options are roughly equivalent in the context described. Phases specifically are capability categories of construction, grooming, and exploitation. The offeror has the option to propose solutions that incorporate one or more of these phases. E.g. a solution might include some aspect of construction that deals with materializing data stores into process or context models; then the solution might show how these processes are executed (exploited) to support a specific persona.

Is the NOAA global data explorer what we need to work with or are we building something new?

The NOAA global data explorer is a valid data source and/or access aid reference. The specific data intended for use should be described by the offeror in the solution proposal context.

This template instructs reference providers to return no later than June 7, 2024. With the short timeline for submissions, would the Government consider accepting PPQ questionnaires up to close of business on June 10, 2024?

Yes, please see details in amendment 01. Past Performance Questionnaires are due the same time and date as the proposal.

With the volume of information requested for the quad chart, would the government consider allowing a font size of 10 to be used and would the government consider this font size for text within graphics and tables as well?

The font size for technical proposals must be 12-point font.

In BAA, Best responses to tell a story, what most important piece want to see in 10 pages?

The BAA is framed in terms of being "capable of dynamically leveraging APIs using the knowledge mesh capability for the purpose of answering highly contextual queries from a number of personas representing a diverse swath of NOAA users", and the intended method to achieve that is to "examine, deliver, and iteratively demonstrate independently validatable, and working models, features, and capabilities related to the development and management of an enterprise information processing system backed by an interoperable knowledge graph ecosystem (‘knowledge mesh’) of process-contextualized, semantically enabled facts, exposed as a unified access interface".

Proposals should tell stories that explain the 'how' of the 'why'.

KM has great value for NOAA data, is intent to show KM to integrate emerging technology into products?

This is definitely a primary goal. The context is how this is validated (the 'why'), which is demonstrating that solutions support the mesh ecosystem as being "capable of dynamically leveraging APIs using the knowledge mesh capability for the purpose of answering highly contextual queries from a number of personas representing a diverse swath of NOAA users".

How do we prioritize between legitimacy of data set (data quality) traceability, credibility, etc., who is qualifying the data as legitimate?

Leveraging existing data sources as indicated by the BAA examples (NOAA datasets and partner datasets) can provide a baseline legitimacy. It might also be of interest to explore datasets that are not legitimate and how the system might support invalidation of processes that leverage those resources. Leveraging or aligning with the outlined and described mesh model (the OAIS information packages and overlaid process framework that binds these concepts horizontally) supports mesh level encoding of provenance information that covers more specific categories like traceability and credibility. Proposing solutions that help address questions posed by personas that might cover questions of trust (e.g. pre-flight process planning or retrospective analysis of process execution to understand result trustworthiness) by demonstrating how the mesh encoding of such provenance would be valuable areas of proposed study.

how do we submit pricing info? within quad chart and technical or separate?

Pricing shall be submitted as a separate volume labeled as pricing.

For the knowledge mesh described, can we design to and add onto what NOAA has currently?

Yes, this is valid.

Can you elaborate on the idea of system of solutions?

As a process-oriented mesh, the ecosystem supports integration of various solutions. Solutions may be bundled into groups of processes and process contextualized resources, with a desire that all resources available are described in the mesh (explicit network topology). In a broad sense the desired mesh will provide the ability to share, inspect, use, reuse, combine, federate, and otherwise interoperate various solutions of various types.

How much of the BAA does our proposal need to address

The proposal is free to address as much or as little as described by the BAA intentions and goals. The 10-page proposals should cover how the described solution(s) satisfy the persona-validatable intentions of the BAA with what phases, and how the phases are supported with what technologies and/or approaches.

Would a knowledge mesh be seen as an add on or a replacement? Is that a desirable deliverable?

What is an 'Agent'

The knowledge mesh should be seen as an integrative interoperability framework that provides unified view over the NOAA information ecosystem, both in terms of process and product. The mesh provides the concept of a leveling framework so that solutions may be described at a very high level and then iteratively broken down into more specific processes and product resources so that more and more of NOAA process and product is described natively in graph form, with an overall north star goal of achieving full enterprise level 5-star-linked-open-data. An agent is a complete system that may consist of its own processes and governance to solve particular process related tasks and produce specific products, and interoperate within the broader ecosystem through use of the mesh as the interoperability layer.

Are you looking for a system of solutions vs 1 overall solution?

The mesh will compose multiple solutions. In terms of solution proposals, offerors are free to tailor their proposal to describe address of a system of solutions or 1 solution.

Is any part of this BAA apportioned to women owned or small businesses?

No, this BAA does not have a specific apportion, but strongly encourages woman-owned and small businesses to participate.

What specific capabilities are needed? The specific capabilities desired for study are all well described in the BAA.

Are there any datasets that they can access to help with development?

The value of a particular dataset depends on the offeror solution(s) that are proposed. Examples of specific datasets and datasets that may be of value to particular use cases are well described in the BAA. These datasets are illustrative and offerors should select/target datasets that support their particular solution goal(s).

Does NOAA need process model output into text or graphic output?

This depends on the specific solution and possibly the specific aspect of solution proposed by the offeror. In general, the offeror may propose solutions that support some persona-valuable interface that leverages text or graphical output from process models. However, it should be clearly stated that the intent of the BAA is not to deeply study digital twin type modeling activity.

Small company, how can we establish a partnership to work with?

The BAA does not restrict the ability of small companies to participate.

Do we need to develop a contextual or multi modal system?

There is no explicit requirement for developing a multi -modal system, and the specific contexts implemented will depend on the proposed offeror solution(s).

Does it need to be open source and modular? Open source in solutions are specified as preferential in the BAA as a general requirement for open science and basis for many desirable and/or required aspects of an operational mesh in terms of trust, intra-process communication, and legal interoperability access issues for process and mesh federation. Modularity as related to portability is desired.

Specific dimensions of modularity of concern for broader interoperability are defined by the mesh architecture and described in the BAA.

Cost per query vs Fixed cost? Some upfront cost for resources needed, how should those be handled?

Proposals should include as much cost information as possible that supports estimation of solution implementation, which is assumed would deal primarily with fixed cost. Cost per query should be included in terms of the actual solution processing. In other words, fixed cost should be covered that deals with baseline solution implementation, and cost per query should be included in terms of simulating process execution in terms of 'how much would it cost me to run this process, at a per-job basis, so I can pre-flight plan my process'.

Do you need traceability of users? Traceability of users is a desired capability for development as it relates to use-driven discovery interests, tailoring and/or predicting experience or use, and enabling qualitative and/or quantitative measurement of how processes and their resources are being used.

Does it need to be multi lingual in the future? Eventually a multi-lingual user base would be supported by an operational mesh as the full NOAA user base supports multi-lingual users.

What will be the end state of intellectual property, who owns S/W after development?

This will be only resolved when proposals are received and reviewed. Note that solutions that exemplify open source, open data, and open science practices are emphasized in the BAA. That said, there are several types of licensing that may be considered depending on the merit of the specific proposal. In addition to Open Source, these include Government Purpose Rights, Unlimited Rights, Limited or Restricted Rights, Specifically Negotiated Rights.

There are 3 domains or phases within the BAA, can we offer solutions to specific phases?

Yes, the configuration of phase address is entirely up to the offeror in their proposal based on the solution(s) they choose to focus on.

If we are using NOAA foundational models should we have servers to run and store model data?

There currently are no operational NOAA foundation models.

The proposal should clearly describe any plans for foundation model development and use. Development and use of the foundation model as part of a selected study should be done on offeror provisioned infrastructure.

Should we develop to autonomous or human interactive solutions?

There is an option for offerors to propose solutions that address autonomous solutions and/or human interactive solutions.

Are there model sets that are outside of the spec of BAA? Are there other standards that may describe ontology?

Standards of interest to the BAA are described in the BAA.

For ontologies, this includes the referenced taxonomies/vocabularies/thesauri/ontologies, while the BAA allows for leeway in what specific ontologies are implemented based on the proposed solution domain(s), the personas that are being addressed, and the questions that are supported for each persona.

Does NOAA have APIS that we need to develop to?

There are a number of APIs to NOAA datasets that may be leveraged in solutions; there are a number of specifications for APIs that are desired in solution including OGC and W3C API standards; the NCCF mesh provides a set of APIs through a python library that the offerors will have access to.

Solutions are preferred to at least align with the provided mesh.

How long does the proposal need to be? 10 or 12 pages?

The technical part of the proposal should be 10 pages.

Exploiting environment can be quite expensive, can we define a subset of data for project?

Yes, solutions should identify what data they are attempting to address. Subsets are recommended, the BAA is not asking for NOAA's data to be entirely converted to ontology, but to study capabilities related to doing this.

Can proposals only target one of the 3 phases in the BAA

Yes, it is up to offerors to specify the configuration of phases being targeted in the context of what datasets are involved in what questions are being addressed to support which personas.

For past experiences, can personal experiences be used or does there need to be an explicit contract to point to?

Past performance should follow the template provided and provide specific examples of contracts the offeror has performed.

Is there a work instance envisioned that vendors/contractors/proposals can operate around? Visualization of what is out there and available prior to award?

Vendors may reference the provided documentation around the knowledge mesh, the information that is publicly available around the knowledge mesh, and the parallel and/or complimentary efforts that have been identified in terms of overall intent and motif (e.g., the proto-okn).

If responding to all 3 phases, if wanting to respond in one proposal, does it need to be contained in 10 pages?

Yes, a single proposal is limited to 10 pages no matter what configuration of phases are leveraged.

Is there additional scoring based on small business or is everyone scored the same?

All submitted and accepted proposals are evaluated based on the criteria given in the BAA. There is not a specific small business apportionment.

BAA mentions offer of AWS account, if using personal account vs NOAA account, do those costs needs to be accounted for in cost analysis?

Offerors are expected to define their infrastructure plans, which may or may not include use of AWS. The NCCF implementation of the knowledge mesh leverages AWS serverless components, and the mesh software will be provided to selected offerors; if the offerors wish to directly leverage the processing software aspect of the mesh, they would need an AWS account. Not all aspects of the mesh software require AWS (e.g. the parts of the mesh that create patterns and templates is library based). Offerors are free to propose leveraging all, part, or none of the knowledge mesh software.

What is the continuation period needed after contract ends if personal cloud accounts are used?

The continuation period is typically one year; however, the government is expecting to take over the account to continue to fund it and the price of the study should not factor that in. An agreement on the logistics to transfer the data from a personal account would be required.

Any guidance on how to determine persona?

Login/registration flow to determine appropriate persona for a given user?

The BAA provides guidance on persona development and intent.

Is there a way to determine the accuracy/applicability of the data/response for a given prompt?

The offeror solution(s) are intended to specify if/how they will address accuracy and applicability. During the Period of Performance, NOAA experts may validate the data and response based on expectations set by the offeror.

How will the multiple awardees be determined?

Are they dependent on each other or independent?

They are considered independent.

To clarify the intent, it's to integrate the digital twin with GenAI question-answering capabilities

One intent of the study is to demonstrate how potentially multiple digital twin technologies including whole digital to answer questions regarding the data within the NOAA datasets?

twins may be integrated into semantic mesh-managed processes. Each twin should itself ideally be process composed. One intent of the study is to demonstrate how this higher order integration would enable and/or be enabled by technologies like GenAI, answering questions based at least in part on NOAA datasets.

How is the enterprise data and the digital twin data and other NOAA datasets stored? For example, is the data stored within a data mesh or centralized in a data lake/lake house?

It is expected that multiple stores will be leveraged. Each of the Digital Twins developed in EO-DT developed their own digital record and digital replica components. It is expected that the mesh will enable interoperability across these diverse data stores, while also allowing native mesh storage as possible in the mesh-managed namespace of denormalized json-ld documents.

Can past performance come from subcontracting work? Additionally, can the subcontractor's past performance be used?

Yes, past performance can come from subcontract work. Yes, if a prime contractor is teaming with a subcontractor, it may use the subcontractors past performance as a reference.

However, of the up to 3 past performance reference requested in section 3.2.2 at lease one should come from the prime contractor's past performance. If a past performance reference is being submitted by a subcontractor, please ensure that the prime organization responding to this BAA is referenced on the form by the past performance submitter.

How do we establish budget for total project cost? By hours? By labor rate?

The study(ies) will be funded as firm fixed price. Offerors should provide the total cost with breakdown as appropriate.

Can we budget the project with student tuition included?

Any resulting contracts will be issued on a Firm Fixed Price basis. It is the offeror's responsibility to determine which costs are necessary to complete the proposal.

82 Is there travel required? No travel is required.

Are you expecting publications at the project end? The publication expectations from the government are described in the BAA. If the offerors develop other materials intended for publication that aren't described, the government would like to have review of them to make sure they are satisfactory to the government before they are released.

How about hardware? Will the government provide equipment to complete this project?

The government does not expect to provide any hardware or equipment (GFE) for execution of the project.

How is the mesh related to KGs? Will you integrate the proposed KGs into a mesh?

The mesh is itself a knowledge graph based on a specific paradigm called the archetype pattern with a specific method of horizontal binding that leverages processes and tasks described by the Open Archival Information System (OAIS). The mesh incorporates generally accepted principles of data mesh in terms of federation and decentralized governance. Through the OAIS constructs of SIP (submission information package) , AIU (archival information package), AIC (archival information collection), and DIP (dissemination information package), and primarily through the RDF-star oriented AIC, the mesh supports integration of open-ended property graph and semantic graph models.

In BAA shows questions about KM is it up and running now?

The mesh is 'up and running' with a TRL of 7. It is intended for full operationalization and feature completeness by the end of FY 24.

BAA describes various data sources, are they integrated into the KM?

Only a handful of data sources is currently integrated in the KM. The intent of the study is to demonstrate broader concurrent/parallel integration of other data sources and then demonstrate how this broad population of the mesh from heterogeneous data sources may be leveraged to support persona-driven uses and novel emergent capabilities.

Are we mocking the data into KM? Possibly. Data should either be mocked or otherwise made accessible within defined KM processes (e.g. virtualized R2RML directives, kerchunk queries, API calls held in template, some other similar mechanism).

Follow up, develop tools for user friendly access, elaborate, prioritize development of processes for whom?

The 'for whom' part is up to the offeror to define, several examples in terms of personas are provided. The tools for user access should support the identified users.

Domain access, open knowledge network is it attached to the KM?

The Proto-OKN connection via a unified NOAA node is an eventual goal of achieving mesh maturity.

Are there any vendors NOAA is already familiar with to leverage info from? Will that be provided prior to bid deadline?

No, this is a new study and there are no incumbents to provide additional support or information.

You had mentioned only one prime contractor; do you already have one assigned or will there only be one prime selected from the vendors submitting their proposals? If so, what qualifies who becomes prime? Is this due to the best overall concept?

The KM/NLP BAA is anticipated to be awarded for 3-5 vendors, not to exceed a collective total of $6M. Each award is to a "prime" and that prime may subcontract. However, we will only work with the prime. It is the responsibility of the prime (awarded vendor) to handle all details with respect to any subs contracted by the prime.

Do you have any preference for the NLS/NLP provider? We have integrated our KM solution with Algolia & AWS (Kendra) and can integrate with any, but was curious to any preference of

NOAA

The operational preference is AWS due to the S3 and neptune components of metadata management. However, as this is a study, there is no explicit preference and this is stated in the BAA.

The posting says Maryland. However, it's not required to physically be in Maryland, correct?

That is correct. Interaction with NOAA is expected to be mostly or completely virtual.

What's an example of a common scenario where you're using LLMs at NOAA? Or a couple scenarios please

There are currently no active approved uses of the LLM at NOAA. Current prototypes for the NCCF KM implementation have demonstrated SPARQL generation composed from user queries that help identify and return available process resources in a mesh marketplace context. This BAA is intended in part to help identify how Large Models including Language Models might be leveraged in a mesh.

Is the 1:1 required or an opportunity if we want it? You mentioned they may be full?

The 1:1 was just an opportunity for additional, and possibly proprietary, questions.

Can you repost the presentation? The presentation can be found on sam.gov for the KM/NLP

BAA.

How are BAA proposals scored? And how are partnerships with subcontractors or vendors scored?

Scoring is not changed specifically for partnerships with subcontractors or vendors, but it is likely that a good partnership will yield a better proposal. See section 3.1.2 and

3.2 for more information on proposal evaluations.

Will the infrastructure have to be FedRamp compliant?

No, there is no such requirement for the BAA. However, FedRamp compliance may make integration assessments cleaner and more straightforward for the government should the efforts be evaluated for integration into NCCF.

Do you anticipate vendors proposing to different phases (construction, grooming, exploitation) to integrate their solutions over the course of the project?

Individual vendor proposals should provide a clear description of any integration plans across the different parts of their proposal and how the parts are classified in terms of the three phases. Beyond that, there is no expectation that vendors would work across proposal boundaries to integrate.

Some of the veracity/transparency/interpretability capabilities conceived in the BAA are fairly challenging in LLM contexts; how big is your appetite for risk in researching novel solutions in these areas?

The appetite for risk on the government's part in terms of solution depends entirely on the proposed solution(s), the value proposition, the technologies selected for study, and the questions being addressed. It is expected that solutions leveraging high risk technologies should call this out and provide mitigation plans in terms of mesh qualification, uncertainty handling, trust management, CARE concern, etc.

which may itself be a major outcome of the proposed study.

Will you have any specific performance metrics (like accuracy, speed, etc.) that the system should meet?

The BAA describes desired performance metrics in terms of 'industry accepted' performance (i.e., quantitative and relative speed). Accuracy in result is called out as a study interest especially in terms of uncertainty qualification/quantification and communication.

Performance documentation in information model is also desired as it will help study and enable 'pre-flight cost estimation'. The BAA also describes the option for offerors to demonstrate specific performance capabilities in an operational profile, but then throttle during study time to save on cost.

Can you give a sense of the scale of language data you have (as opposed to graphical or numerical data) to work with in production and also that can be made available to us as sample data

There are significant amounts of language data available that either directly encodes NOAA data or contextualizes NOAA data. Some of the sources include the institutional repository, existing NOAA metadata and catalog systems, NOAA studies around economic or social impacts and assessments, NWS weather reporting data, and system specifications, at least. Some of these sources are linked in the BAA, and the BAA provides guidance that other similar sources are valid as long as the use case is well qualified.

Do you have target award amounts per awardee and an anticipated number of awardees?

The anticipated number of awardees is 3-5, but that can change pending a variety of factors. The budget constraint is a total cost of $6M for the total amount of contractor awards.

How would you generally quantify "goodness" of geophysical data? Often, we use statistical metrics for uncertainty but in some of the BAA us cases it's based on non-quantitative and sometimes derived measurements (i.e. how good was the fishing that day for tuna fisherman?)

Quantification or qualification of goodness is a complex issue and is one area called out by the BAA as very rich and valuable area of study. Semantic encoding or ontological development that contextualizes both statistical measurements and quantitative measurements is desired, as well as studies that demonstrate combination, expression, and root-cause-tracing of these uncertainties, along with related studies.

Are certain technologies or products already in use that vendors would need to integrate with?

Also, what is the preferred cloud network?

Alignment with the knowledge mesh reference architecture is the most important consideration in terms of technology/product alignment. Beyond this there are no explicit requirements in terms of technology although there are many preferences and preference-guided suggestions given throughout the BAA. AWS is named in the BAA as it is where the NCCF is being developed and the KM implementation relies primarily on AWS native serverless components (eventbridge, step functions, lambdas, s3, neptune). However, there is no explicit requirement to leverage the government provided software.

The 3 to 5 awards, is the ~$6 M allotted across all awards?

Yes, but $6M budget is divided across all vendor awards, which may be a limit factor for the number vendor awards.

Do we need to know about NSF's Prototype Open Knowledge Mesh in advance or interact with it in any way?

There is no explicit requirement to interact with the Proto- OKN. To support interaction with the Proto-OKN, following the recommendations around KM laid out in the BAA will naturally provide a pathway for Proto-OKN interoperability and integration.

Do we need to align knowledge mesh with the NCCF prototype?

Yes, the proposals will be evaluated in a weighted sense against architectural alignment.

Do the proposed knowledge meshes need to be an expanded mesh or a standalone KM?

Either of these configurations is fine in proposal, it is up to the offeror to determine the best fit to deliver the proposed solution.

Is there any specific persona that should be targeted or is preferred to be demonstrated?

Personas are given in the BAA as examples, but the offeror is free to choose what personas make sense for their proposed solutions. Keep in mind that the best proposals will support multiple personas and the personas will be clearly relevant to NOAA internal and/or external users. Persona selection may be driven by support for specific efforts, e.g. the coastal zone digital twin or the AI for public good digital twin effort.

Can a list of 'most utilized' datasets be provided?

Does NOAA know which datasets are most downloaded? Is that info available?

A holistic picture of the most utilized data at NOAA is not available. One of the purposes of working toward and studying a knowledge mesh architecture is to enable use driven discovery by interoperable encoding access processes with archive processes, and enabling holistic metric analysis in a 'single pane of glass' approach to records management.

What are mesh/persona interoperability requirements? Are multiple personas built into same knowledge mesh or multiple?

Personas should be describable in the shared mesh reference model as entities. Multiple personas should be supported by the same mesh.

Is there any requirement for front end and back end solutions being open source?

There is a very strong preference for open source front end (access aid) and back end (processing infrastructure) components, although this is not a strict requirement. More important is the use of an open and shared archetype-oriented semantic reference model for holistically describing entities, transformations, and deliveries. The NCCF implementation leverages an RDF encoding of the Open Archival Information System (OAIS) concepts of Information Package and Information Object, and the BAA provides guidance to align with this model.

If we can use existing tech, does that need to be accounted for in the budget? Cost for an fully open source vs open govt vs partial proprietary cost in same proposal?

Yes, decisions and associated costs around technology choices should be as transparent as possible to the government at the time of review.

What kind of AI compliance standards need to be addressed as part of this project?

AI Compliance requirement is largely driven by federal standards. See the references provided in the BAA around open data policy, FAIR, CARE, and TRUST frameworks, as well as White House memos around ethical AI.

Looking at use cases in Appendix B, what specific measures are currently in place, or what are you looking for to mitigate bias in data?

AI bias mitigation is not mature in NOAA in the operational sense, and proposals are welcome to suggest solutions or solution pathways for addressing bias identification, mitigation, and management.

Is the data needed to create the knowledge mesh already available? Do we need to create frameworks or data pipelines?

Creation of data pipelines ('processes') is assumed to be a part of the solution proposals. Data for use in these processes is widely and publicly available, although most available data/metadata is not already in semantic form so will either have to be mocked for use in a mesh, or accessed in semantic process via some API reference. Potential sources for data are indicated in the BAA, but which data sources are chosen are up to the offeror and will primarily depend on the solution proposed, what the scientific domain(s) are, and what the personas are.

What kind of infrastructure exists and how are we able to show a particular use case? Is there already existing machine learning and we need to build off of?

The selected offerors will be provided with existing NCCF mesh software that can be deployed on AWS, however use of this is not a requirement for proposals. As stated in the BAA, solutions will be developed on offeror-determined infrastructure, but regular validation by NOAA experts is expected, meaning mesh processes and capabilities will be expected to be available for testing by NOAA personnel. For processes, this means API exposure, and for Access Aids, this means internet-based access. This access can be fully open or session-qualified and role-based.

BAA is very broad, what specific milestones/phases are you looking at to tackle this approach?

The BAA is intentionally broad. The phases targeted for study are clearly outlined as construction, grooming, and/or exploitation (and any configuration of those) that supports the persona-driven use cases identified by the offeror solutions. It is up to the offeror to define specific milestones or time-based phases, and this plan will be evaluated during NOAA review of proposals.

Is NOAA looking to integrate legacy systems into overall solution? Are the any on-premise SLM approach we need to be aware of?

There is no requirement to integrate any legacy systems. No knowledge of on-premises software lifecycle management methods is assumed or required.

Do we need to include any operational objectives into proposal?

No, operational objectives are out of scope for this BAA.