BAA_Questions_Amendment_0003.docx

BAA Questions Expeditionary Intelligence, Surveillance, and Reconnaissance Science and Technology

BAA#: N00014-18-S-B006

Amendment A00003

1. Would this BAA include research of Computer Vision technologies that can provide input into the framework discussed in the solicitation?

A. The use of computer vision algorithms to interpret patterns in diverse data would be responsive.

2. How much user involvement would you anticipate when training an AI or machine learning system? Can a system work interactively with a user to transfer knowledge or generate training data, or should it be fully automated?

A. If by user you mean a warfighter, any machine learning from humans needs to be accomplished unobtrusively.

3. When discussing “patterns-of-life”, do you have specific data sources or representations in mind (e.g. features/categories, hierarchy, temporal, geospatial, etc.)?

A. Patterns of life for entities that represent an enemy order of battle are of interest. Those entities have properties as well as space/time movement patterns.

4. Several topics mention generation of training data or otherwise imply classification. What is the sensing domain of primary interest? Are we talking EO/IR or acoustic? Is aerial aerial/drone perception of interest or is this principally focused on underwater?

A. All modalities. Data from aerial sensors are of greater interest.

5. In general, are you ok with utilizing public datasets in support of ML training, or do you want us to focus on generating entirely new data from scratch?

A. Any available data set that that improves the performance of an agent should be used.

6. There is a lot of mention of agents, but a number of the areas of interest could be tackled using non-agent-based AI approaches. Is the TPOC only interested in agent-based approaches?

A. No. Any approach that uses AI/ML to enable a capability is potentially of interest.

7. What about approaches that use human authored models of adversary behaviors, with the longer-term goal of machine learning approaches to model creation? Or is the machine learning element critical in earlier phases of the research?

A. Offerings should have a strong ML component.

8. Regarding the topic related to battle management aids, do you have any particular success metrics in mind for evaluating battle management aids?

A. Any aids that enhance force protection, mission effectiveness or mission execution efficiency across collection and C2 are of interest.

9. Question on the application of GANs. Is ONR interested in data of a particular modality with regard to this focus area? What type of data may be most relevant for demonstrating the applicability of an approach to ONR’s concerns?

A. The use of GANS to generate labeled data to train any of the services described is of interest.

10. Many of the areas of interest involve augmenting, enhancing or predicting within a common tactical picture. Are there particular standards, existing schemas, repositories or APIs that ONR would like proposers to adhere to?

A. This is dependent on the service being provided and whether the Marine Corps or Navy would be the transition partner.

11. In terms of inputs to the tactical picture, what communications or sensor data is within scope (e.g. on-ship communications, Blue force audio/chat, SIGINT, ELINT)

A. All. Note improving data access is not within scope of the BAA.

12. Are there any envisioned size/weight/power constraints on a fieldable prototype at this stage in your development cycle?

A. No. The constraints would vary with level of command.

13. And one procedural question, are you looking for resumes of key personnel in the whitepaper? (My assumption is that is more an expectation for the proposal, but wanted to clarify)

A. Evaluators will consider the qualifications, capabilities and related experience of the offeror, proposed Principal Investigator (PI), team leader, and key personnel.

14. Can an offeror submit multiple white papers?

A. Yes.

15. What is the application domain? Is computer vision of interest? Are only EO image data types of interest or are the SAR and Radar inputs also considered?

A. Computer vision algorithms may be relevant. Analysis of embeddings is more important than EO images.

16. What is the exit criterion or metric of success? Detection Accuracy vs. number of synthetic training data samples (for augmentation)?

A. The more robust the service is the better. Different services may have different metrics for operational relevance.

17. What are the datasets and targets of interest? Are Battleships/Buildings/Port structures of interest?

A. Any object in a military order of battle is of interest.

18. What is the target classifier of interest? Is YOLO an acceptable classifier?

A. Classifier selection is up to offerors.

19. Does the target classifier detect targets, or classify target types, or label more refined features?

A. Threshold is battlespace situational awareness/understanding and decision support.

20. What is the size of available training data? What types of labeled datasets are available?

A. Data sets generated by military simulations are available.

21. Is the target AI system fully automated? Is it possible to have human in the loop?

A. The threshold system would be human in the loop.

22. Do you anticipate that domain knowledge could be available and be leveraged to generate synthetic data?

A. Potentially, dependent on what domain knowledge is desired.

23. To train AI with less data, do we have to use generative adversarial networks based augmentation? Are other few-shot learning methods also in scope?

A. Methods chosen are up to offerors.

24. Are you interested in only training data augmentation or also training data adaptation (domain adaptation, e.g., from day time images to night time images)?

A. Question is not relevant to the BAA.

25. Is there any time and computational complexity constraint that we need to take into account?

A. Most military decision support systems need to run in near real-time to be useful.

26. What simulation environments are available to train an agent? If the Navy can’t provide such an environment, can we use public environments?

A. JSAF/OneSAF can be made available.

27. Is there a target transition program or are there potential Navy end-products for integration at the end of the project?

A. Navy and USMC command and control programs of record.

28. Who will be the end-user? Will it be the researcher or analyst using the Docker to augment a dataset or a system (target recognition system) that is trained with augmented dataset?

A. Different services will have different end users.

29. Should the ROM include the entire period of performance (potentially 3 years) or just the first year?

A. Entire period of performance.

30. In the white paper do we specify that we are applying for BA2 or BA3

A. Not required.

31. Should the white paper provide a preliminary project schedule (e.g., Gantt Chart)?

A. Potentially useful but not required.

32. Given existing relationships with a number of individual organizations and academic institutions that may also be submitting white papers, is there the potential for teams to be formed after white paper submission?

A. The government will not form such teams.

33. Will there be preference for products with existing ML/AI capabilities, over products with potential future integration?

A. Both are responsive.

34. Are there preferred qualifications for a successful PI? Are experiences and capabilities valued over educational qualifications?

A. Both experience and education qualifications that support the researcher’s ability to be successful are important

35. Is novel visualization of data retrieved from agents a preferred capability?

A. That is considered to be in-scope

36. Given existing capabilities for developing training environments and identifying training points, how may we evaluate whether said capabilities satisfy criteria for success, or where improvements can be made to meet said criteria? What are desirable metrics for successful solutions?

A. That is largely the responsibility of an offeror to determine. Any developed capability should have operational or potential operational relevance.

37. In the last bullet of the Naval areas of interest, can you clarify if the “little or no communications” limits the communication between the commander/operator and the agents, between the agents themselves, or both?

A. Between the platforms/sensors.

38. BAA, Section II, B.4, Instrument Type, page 9 of 38: Please confirm that a Large Business is eligible for award of both a Procurement Contract and/or Other Transaction agreement (for Prototype) under the subject Announcement.

A. Large businesses are eligible.

39. BAA, Section II.E. 1, Application Review Information, Criteria, page 16 of 38: Please confirm that proposals that do not include research involving human beings to not require a Principal Investigator.

A. All research efforts require a principal investigator

40. BAA, Appendix 1, A.1.a, Fixed Fee on ONR Contracts, page 32 of 38: Please confirm that offerors should propose fee in accordance with the weighted guidelines on page 32, but does not need to provide the completed weighted guidelines table supporting the proposed fee.

A. Will be clarified if a full proposal is requested.

41. Cost Proposal Template (excel spreadsheet): What is the recommended start date to use for the proposed pricing model? Especially if offeror’s quote is CPFF?

A. Also a full proposal question.

42. Cost Proposal Template (excel spreadsheet): Does the Government require offerors to provide onsite and off-site rates in the pricing template?

A. Answer not relevant to white papers.

43. In Section II D b of the BAA (p.9) of the section titled “Submission of Unclassified and Classified Proposals”, the BAA states:

“White Papers and Full Proposals submitted under this BAA are expected to be unclassified; however, classified proposals are permitted. If a classified proposal is submitted and selected for award, the resultant contract will be unclassified.” Does this mean that no DD-254 will be associated with the resulting contract (i.e. no classified work will be sponsored by the contract)? We understand the desire that any proposed development of models or algorithms be unclassified; however, we have a few proposals that may require training with classified data. Would this be allowed?

A. A DD254 can be attached to the contract. The referenced comment means that an unclassified SOW will be required. The work can be classified. Training on classified data is ok.

44. Does ONR have any interest on this BAA in machine learning applied to CBRNE signatures?

A. No.

45. Which symbology standards (e.g. MIL 2525) should a prototype capability to update symbol placement support?

A. MIL 2525 would be acceptable.

46. Would a proposal that assumes use of a 3rd party audio-to-text tool be considered responsive? (In other words, would research that focuses exclusively on inferring tactical information from text, and not on the problem of converting audio to text, be responsive?)

A. Yes although license fees can be a barrier to transition.

47. Will the government make data available that contains representative text/audio information with ground truth location information?

A. Potentially. We are working with one unit that may enable that data.

48. What metrics will the government use to measure and assess the performance of a capability that responds to the symbol-placement focus area?

A. Timeliness, accuracy.

49. How is symbol placement and tactical picture updating from audio/chat data done today?

A. Manually.

50. Can a single company be awarded multiple proposals?

A. Yes. Companies may receive multiple awards depending on the technical merits of their proposals.