QAs - 832016142.docx

JAIC T&E MAC RFI Q&A’s:

1. Can you confirm that it is a Total Small Business Set-Aside?

The intention is always to keep it open to large businesses and small businesses. Beta.sam requires you to choose the "set aside" and there isn't an option for open competition or even large business.

2. I see the classification is Total Small Business Set-Aside (FAR 19.5). Is that set in stone or the intended solicitation strategy?

The intention is always to keep it open to large businesses and small businesses. Beta.sam requires you to choose the "set aside" and there isn't an option for open competition or even large business.

3. Do you think this request will go SETI like the other one (JCF)?

We are still working on our market research report and the JAIC will determine the appropriate acquisition strategy based on our findings.

4. Is there any current contracted support to the JAIC related to the two specific areas called out in the RFI - “Test Technology and Tools” and “Test and Evaluation Services?” Just wanted to confirm that this is an all new requirement.

5. What tools or products, if any, are currently part of the Joint Common Foundation? In other words, are there incumbent technologies?

6. Is the JCF modeled after any existing DevSecOps environments deployed in DoD today?

7. If a product or tool is chosen to be part of the JCF platform, does that tool somewhat automatically (provided the C&A steps are performed) obtain Authority to Operate across DoD? Intel too? Does this piggy back off any existing data sharing policies?

8. The role of a data scientist is to not only build AI models but to test and evaluate them. It's sort of central to the job. The whole purpose of a model confusion matrix is to facilitate performance evaluation. Is the purpose of the T&E team to test for things like cyber security compliance or with the evaluation work... extend into the realm of data science? ie, Is it software testing or data science testing?

9. When you evaluate AI tools for explainability, will you be using any objective criteria that can be reviewed?

10. Is there a roadmap for the build out of the JCF in terms of which technologies will be selected for inclusion and in what rough order or is the idea to somewhat fully bake an entire solution and procure it all at once?

11. Can you say anything about the success criteria being advanced by the key stakeholders in Army, Navy, Air Force, etc?

12. Has any thought been given to the possibility of exposing toolsets to the stakeholders and then letting adoption metrics continuously inform what should be in the toolchest? Or has that been concluded as a non-optimal way to proceed? The question arises given that within the AI/ML technology community, there are so many tools in use across DoD....hard to pick the winners with a single bakeoff. Maybe there could be an environment set up where practitioners are exposed to the different toolsets--with demos/training and sample datasets to let practitioners get a deeper feel for how the tools work?

13. Can you please clarify if the JAIC expects any engineering development in the anticipated acquisition of AI Test Technology and Tools? Specifically, is the JAIC looking for off-the-shelf products or does the JAIC desire advanced technology with completed capabilities, however the capabilities do require some deployment development, documentation, and accreditation?

14. Is the goal of VR/AR Test Technology and Tools to evaluate how VR/AR extends humans capabilities/improves performance or is the intent to use VR/AR to test AI systems—or both?

15. We have expertise developing software that runs on VR/AR HMDs to benchmark the performance of that specific VR/AR hardware (e.g., suite of assessments to evaluate HMD’s perceived color, contrast, acuity capabilities) – is that type of software in scope as a T&E tool for VR/AR?

16. What is the expected Technology Readiness Level (TRL) of the Test Technology and Tools at the beginning and end of the acquisition?

17. What is the approximate duration of an anticipated acquisition for Test Technology and Tools and the T&E services contract?

The intent for the T&E services contract is that it will be a JAIC offering to the DoD in the form of a decentralized vehicle. The need for this sort of vehicle will exist as long as the DoD procures AI/ML.

18. What is the approximate budget for acquiring T&E technology and services?

The JAIC’s intent is to create a vehicle to absorb all of its current AI/ML T&E requirements. This vehicle will also be decentralized, meaning that all of the DoD will be able to leverage it for their respective T&E needs. The JAIC is still working on a market research report and environment scan that can forecast an approximate budget.

19. We have expertise developing software using relevant machine learning, and AI techniques to use behavioral, physiological, and neurophysiological data into indications of human cognitive or physical workload. Are wearable sensors in scope for measuring cognitive/physical workloads?

20. Is the JAIC seeking new tools for them to use themselves in their T&E lab or is their goal to fund T&E at an outside lab? If they are seeking new tools for their own T&E labs, could they provide an example of what they are testing and engineering in those labs?

21. The NAICS code that seems to fit this opportunity most accurately would be 541511 due to "discover, develop, and test solutions for their AI problems and serves as a host to the AI technologies developed by the Mission Initiatives." Could you please confirm this will be the NAICS code for the SBA Size Standard for Total Small Business Set Aside?

At this point in our market research process, this seems like the best fit for our NAICS code. This requirement will not be a small business set-aside. The intention is always to keep it open to large businesses and small businesses. Beta.sam requires you to choose the "set aside" and there isn't an option for open competition or even large business.

22. There are several references to voice-to-text and speech-enabled DoD applications, however no specific application is spelled out. Is it correct to assume that these applications, regardless of the AI technology at play, will require and include not only a capability for detecting speech as text but also to parse and understand that speech in the context of the mission or use case? Please clarify if this capability would require past performance and expertise over and above the understanding and application of commercial STT engines?

23. Regarding human-machine teaming is the assumption that the machines are sufficiently knowledge-rich and contextually aware that they can provide timely, accurate and useful information to the human at mission-time?

24. Is the government interested in testing and evaluating emerging explainable ML and DL techniques in addition to more classical approaches?

25. When do you expect to furnish the anticipated metrics that you wish to use when evaluating the quality of AI models?

26. Will data and environment it is stored be unclassified?

Data will range from unclassified to classified, based on effort.

27. Will historical ground truth data / validation sets be available to build upon or will training/test sets need to be curated from scratch?

28. Will analysis be performed on development pipelines running ad-hoc or in batch? Or will analysis need to consider real-time systems?

29. Will we have visibility to models and their configuration/metadata or will code and models be obfuscated/compiled?

30. Is the testing process prescribed for how the T&E is conducted?

31. What AI tools are currently are in place at JAIC?

32. Are there existing T&E processes in place at JAIC?

Yes, the JAIC currently has a T&E team that supports existing requirements.

33. Are AI testing tools currently being used? If so, what tools are currently in place?

34. The document mentions Mission Initiatives infrastructure needs AS i.e., cloud-based test tools that are containerized. Are there any tools/platforms already in place to manage containerized workloads and services?

35. It's mentioned in T&E services that for Data Curation, each MI has unique data needs. Is there test data currently available to address building products for the different MI. Do we have a list of these and is it available? What is the approximate size(s)?

36. Based on the above, are there data quality assessment tools currently being used at JAIC?

37. Is the attachment mentioned in Section 3(1), for test technology and tools, outside of the 10-page count mentioned in the response guidelines?

The attachment should be included in the 10-page count.

38. Can you provide indication of the Place of Performance?

National Capital Region.