RFI 6 M_S Questions and Answers 2024-10-25.pdf

PEO IEWS, Project Linchpin, PEO STRI, STE CFT - AI Modeling and Simulation (M + S) Capabilities - Request for Information (RFI) Questions Answers

10/25/2024

# Question Topic Area Answer

Can we highlight the security of data collection, sharing, and processing, while using a selected use case as an example

Security Yes, that information would be useful to the government.

What is the envisioned range of training environments? Is the entire training environment in a simulation environment? Are the Soldiers in a warehouse, or actually in the field with an AR HUD?

Training The training environments can be both fully simulated like in a classroom or immersive environment or a hybrid environment like an exercise on a base where portions of the scenario are simulated.

3 Is the goal to support something like XR trainers, or training in an instrumented environment? Training Both training environments are possible.

What is the purpose of asking about Counter – Counter AI (CCAI) capabilities? Is the goal a cybersecurity one, to ensure performers are aware of and prepared to protect and mitigate threats against AI models and systems?

Counter - Counter AI

Counter – Counter AI is within the realm of Cybersecurity. Its aim is to fortify AI systems against adversarial attacks by simulating attacks and mitigating vulnerabilities to the AI models though the knowledge gained. It is a continuous process used to find new vulnerabilities and mitigate them on and ongoing bases.

What use case scenarios do you envision or do you have a preference? Many are doable and plausible, e.g.

-Special Opera ons oArmy Special Forces conduc ng specialized missions oTraining and working with allied forces in the region -Land-Based Opera ons oSecuring and defending larger islands/landmasses a er ini al beachheads are established oOpera ng long-range missile systems and ar llery oConduc ng ground opera ons in jungle/tropical environments -Logis cs & Support oManaging supply lines and forward opera ng bases oOpera ng ports and maintaining supply chains oRunning field hospitals and support facili es

Scenario Examples Any of the three scenarios above would be acceptable.

What is the relationship between this RFI and the PEO IEW&S Fast Track MATOC opportunity (Notice ID: W56KGY-24-R-MATOC) that has been announced? Are they distinct opportunities or is this intended to be a task order on the MATOC?

MATOC

This RFI is a request for information and is not related to any competitive contract such as the MATOC. This is a request for information only.

Does Project Linchpin seek a particular range of technology maturity for this opportunity? Or are capabilities at any TRL relevant?

TRL

The government is not limiting responses to any specific Technology Readiness Levels (TRL), however, if specific solutions are provided, we would like to know what its TRL is.

Is the Intelligent Observer-Controller (IOC) use case described in the RFI provided as an illustrative example, or is there specific interest in AI M&S capabilities to support the IOC use case?

The IOC example refers to a squad-level training exercise in which an infantry squad reacts to direct fire. Is there particular interest in this domain and echelon? Or are other domains and/or echelons relevant

Use Case

The Intelligent Observer-Controller (IOC) is provided as an illustrative example. It can be used as a use case to answer the questions in section 5 of the RFI. However, if the vendor has an area of expertise related to M&S a use case for that area can be generated and used to answer the RFI.

Any domains related to M&S that the vendor has knowledge in are relevant to the RFI.

Is there a preference for AI capabilities that focus on enhancing M&S dimensions of the application (e.g., automated behaviors, OPFOR AI, etc.) vs training dimensions of the application (e.g., generating AAR materials to automatically assess squad performance, etc.)? Or both?

Capabilities Both capabilities listed would be applicable for a response to the RFI.

Is there a preference for AI capabilities focusing on scenario and training adaptation as opposed to data collection and analysis?

Use Case While responses that contain both types of information listed are welcome, the scenario based information is more aligned with this RFI.

Are questions about Counter – Counter AI (CCAI) seeking information about capabilities for fortifying AI systems that specifically support M&S for training?

Counter - Counter AI

Not exactly, Counter – Counter AI's aim is to fortify AI systems against adversarial attacks by simulating attacks and mitigating vulnerabilities to the AI models though the knowledge gained. It is a continuous process used to find new vulnerabilities and mitigate them on and ongoing bases.

The above is in the realm of simulation, not specifically for training.

What are the gov’t’s plans to verify/validate submitters’ data and data collection/generation techniques?

Verification of data - Not applicable This RFI is solely for informational purposes. Because of this, the government has no plan to verify/validate data, data collection, or data generation.

Should solutions be directed toward live and virtual training use cases, or are constructive use cases equally valid?

Use Case Both of those use cases would be acceptable for a response to the RFI.

PEO IEWS, Project Linchpin, PEO STRI, STE CFT - AI Modeling and Simulation (M + S) Capabilities - Request for Information (RFI)

10/25/2024

Can the sponsor please define what squad-level training consists of? Is it certain amount of ground troops only or is there a combination of soldiers and miliary vehicles? If so, what is the typical number of soldiers and vehicles in a squad?

Scenario question

A squad typically consists of 8 – 12 soldiers. Squad-level training focuses on preparing a small group of soldiers to operate effectively as a cohesive unit in various combat scenarios. The goal is to enhance teamwork, communication, and tactical proficiency. This type of training usually involves a combination of infantry troops and sometimes light military vehicles, depending on the specific objectives and the branch of the military. The initial capability will be for a single squad with at least 1 vehicle however the need will encompass squad on squad (Red vs Blue teams) and multiple squads up to Brigade level (5,000) as an ultimate goal.

Are there specific data sources or types (e.g., live training data, sensor inputs, operational data) that the government expects to be used for AI model training and validation in the proposed use cases?

Data Collection and Integration

There are no specific data sources identified for this RFI. However, any data sources provided should align with the use case submitted by the vendor to answer the questions in Section 5. If there is specific data that would significantly help the AI system for your use case it should be stated in your response as the government may have it or will develop a system to gather the information.

Could you clarify the preferred hosting environment for the AI models and M&S applications (e.g., cloud-based, on-premises, hybrid)? Are there any specific DoD-approved cloud environments required beyond what was mentioned (e.g., AWS GovCloud, Azure Gov)?

Hosting Environment

This RFI is for informational purposes. A specific solution architecture that is matched to existing M&S systems is not required to respond to the RFI. However, the vendor is free to do so. Project Linchpin has access to hybrid (on premises government owned and Commercial Cloud (AWS, Azure, Google Cloud Platform)) environments for model development, test and evaluation. It is assumed inference will be run within the M&S environments.

What is the Government's stance on the use of proprietary or internally developed AI engines compared to commercially available AI engines for this effort?

AI Engine

This RFI is for informational purposes. IP rights would vary depending on the individual requirement and the commerciality of the product. The Government would seek data rights consistent with DFARS part 227 and with special consideration for form, fit, or function data, or data required for repair or maintence. In the context of collaboration and integration, the prioprietary data is handled consistent with the established/negotiated data rights for the requirement and DFARS 227. This does not exclude proprietary AI engines if the cost and performance of the system is justified. Internally developed AI engines can be acceptable if the company developing the engine can meet the security and policy requirements to perform DoD contracts.

Should the solution be compatible with existing M&S software and infrastructure already used by PEO STRI and STE CFT? If so, can you specify which platforms, frameworks, or tools must be supported?

AI Model Deployment Responses to the RFI that include both existing M&S software and infrastructure or proposed new ones, are both welcome by the government.

What is the minimum TRL the government expects for AI models and M&S solutions? Should the proposed solution demonstrate readiness at a specific TRL before consideration?

Technology Readiness Level (TRL) The government is not limiting responses to any specific Technology Readiness Levels (TRL), however, if specific solutions are provided, we would like to know what its TRL is.

Could you elaborate on the key performance indicators (KPIs) for assessing the effectiveness of AI solutions in M&S? Are there specific metrics or thresholds the government will prioritize?

AI-Enabled Solutions for M&S This RFI is for informational purposes. Defining a specific solution architecture is not required to respond to the RFI. Therefore, no specific key performance indicators or metrics have been outlined.

Will the government provide detailed technical specifications or APIs for integrating proposed AI models with existing M&S environments, such as live, virtual, constructive, or gaming systems? Is interoperability with existing military systems a critical requirement?

Interoperability and Integration This RFI is for informational purposes. Defining a specific solution architecture is not required to respond to the RFI. Therefore, no APIs have been outlined. However, the vendor is welcome to submit a response that contains interoperability with existing military systems.

Regarding Counter-Counter AI (CCAI) requirements, what types of simulated threats (e.g., deepfakes, data poisoning) are most relevant to the government’s objectives? Are there any specific threats or adversarial attack models the government is prioritizing for M&S applications?

Simulated Threats Counter -Counter AI is a new topic for the government. Because of this, there is no specific area the Government is focusing on with Counter-Counter AI. The Government is open to any information the vendor has in this area.

How will the government evaluate the effectiveness of CCAI solutions in detecting, mitigating, and responding to adversarial attacks? What specific metrics or criteria will be used in this evaluation?

CCAI Metrics and Evaluation

This RFI is for informational purposes. Therefore, there are no specific metrics the Government will be using for evaluation. The government is seeking descriptions of solutions for Counter-Counter AI from vendors. The responses will be gathered and used for informational purposes by the Government.

Are there preferred licensing or cost models that the government prefers for AI and M&S software?

Should solutions propose perpetual licensing, subscription-based models, or something else?

Licensing Models This RFI is for informational purposes. The government does not have a preferred cost model at this time. The government would like to know what cost models the vendors are proposing.

PEO IEWS, Project Linchpin, PEO STRI, STE CFT - AI Modeling and Simulation (M + S) Capabilities - Request for Information (RFI)

10/25/2024

Can the government clarify expectations around IP ownership, particularly if the solution includes proprietary AI models or algorithms? How will the government handle proprietary vs. government-owned data rights in the context of collaboration and integration?

Intellectual Property (IP) Rights

This RFI is for informational purposes. IP rights would vary depending on the individual requirement and the commerciality of the product. The Government would seek data rights consistent with DFARS part 227 and with special consideration for form, fit, or function data, or data required for repair or maintence. In the context of collaboration and integration, the prioprietary data is handled consistent with the established/negotiated data rights for the requirement and DFARS 227. This does not exclude proprietary AI engines if the cost and performance of the system is justified. Internally developed AI engines can be acceptable if the company developing the engine can meet the security and policy requirements to perform DoD contracts.

Among the use cases outlined (e.g., Intelligent Observer Controller), is there a prioritized use case that the government sees as most urgent? Would the government prefer to focus on one high-impact solution initially or develop multiple AI use cases in parallel?

Use Case Prioritization This RFI is for informational purposes. The Government does not have a specific use case within the M&S realm it is focusing on. The vendor can choose to either provide a response with a single use case, or one with multiple use cases.

Are there existing testbeds or frameworks the government would prefer contractors to use, or will new T&E environments need to be developed?

Test and Evaluation (T&E) This RFI is for informational purposes. The Government does not have a specific test and evaluation framework. The vendor is welcome to provide information about test and evaluation frameworks.

What CI/CD tools are currently in use within your development teams, and what specific challenges do you face with these tools?

CI/CD Tools

This RFI is for informational purposes. Project Linchpin is currently in its planning stage. Tools such as Github, Slurm, Python Jupyter, R, SageMaker, AI Code Generator, AI Studio and others will be considered. The vendor is welcome to propose CI/CD tools for development, provided it is aligned with the questions asked in the RFI.

Can you describe your current testing processes? How do you manage test execution, and what types of tests (unit, integration, functional, etc.) are most critical to your workflow?

Testing Processes This RFI is for informational purposes. Project Linchpin is currently in its planning stage. Future testing will have a focus on Infrastructure, Data, Models, and Security. The vendor is welcome to provide information pertaining to testing processes, provided it is aligned with the questions asked in the RFI.

How do you currently determine which tests to run for each code commit? Are there specific criteria or methodologies you follow?

Test Selection Criteria This RFI is for informational purposes. Project Linchpin is currently in its planning stage. Testing strategies will be developed on a case-by-case basis in the future. The vendor is welcome to provide information pertaining to testing strategies, provided it is aligned with the questions asked in the RFI.

What are the main bottlenecks in your testing cycles, and how do they impact your overall development and deployment timelines?

Testing Efficiency This RFI is for informational purposes. Project Linchpin is currently in its planning stage. The vendor is welcome to provide information pertaining to testing efficiencies, provided it is aligned with the questions asked in the RFI.

32 How do you identify and manage flaky tests within your current testing frameworks? What impact do they have on your development process?

Variable Test Results This RFI is for informational purposes. Project Linchpin is currently in its planning stage. The vendor is welcome to provide information on how best to deal with variable test results, provided it is aligned with the questions asked in the RFI.

What metrics do you currently track regarding test performance and code quality? How do you utilize these metrics to inform decision-making in your development process?

Data and Metrics

This RFI is for informational purposes. Project Linchpin is currently in its planning stage. Being that the metrics for performance results are multi variate, and will be tailored by use case, metrics will be defined in a case-by-case basis in the future. The vendor is welcome to provide information on approaches to defining performance metrics, provided they are aligned with the questions asked in the RFI.

How do your development, QA, and security teams collaborate throughout the software development lifecycle? What tools facilitate this collaboration?

Collaboration Between Teams This RFI is for informational purposes. Project Linchpin is currently in its planning stage. Collaboration tools such as Jira, Confluence, Gitlab, NextCloud and others will be considered. The vendor is welcome to propose collaboration tools, provided it is aligned with the questions asked in the RFI.

How are security considerations integrated into your development and testing processes? Do you have specific tools or practices in place for AI/ML-based security measures?

Security Integration

This RFI is for informational purposes. Project Linchpin is currently in its planning stage. Security is an all-encompassing part of the planning for the AIOps+ pipeline. The vendor is welcome to provide information on how security can be integrated into all facets of the AIOps+ pipeline, provided it is aligned with the questions asked in the RFI.

How quickly can your teams respond to test failures or production issues? What processes do you have in place for tracking and resolving these issues?

Feedback Loops This RFI is for informational purposes. Project Linchpin is currently in its planning stage. The vendor is welcome to provide information on how best to respond to test failures or production issues and tracking for them, provided it is aligned with the questions asked in the RFI.

What are your future goals for improving your software development and deployment processes?

Are there specific capabilities you seek in new tools to support these goals?

Process Improvement This RFI is for informational purposes. Project Linchpin is currently in its planning stage. The vendor is welcome to provide information for improving software development and deployment processes and the tools to support these processes, provided it is aligned with the questions asked in the RFI.