OpML BAA INDUSTRY DAY ANNOUNCEMENT

OpML BAA Questions and Answers
Aug 02, 2019

Presentation Questions
• Will the presentation materials (all Distribution D) be release via FBO, AMERDEC or other means? Distribution D material will be distributed via AMRDEC upon request to Distribution D qualified entities. Overview slides will be posted with Q & A on FBO.

• Can you tell us a bit about Flyleaf: Is it a "multidomain env." in a MLS sense, or air-cyber-space, or both? Additional info on Flyleaf will be available soon, most likely via AMRDEC upon request. In the meantime, Flyleaf is a program with a goal to create a capability within AFRL/RI to assess, in a multi-domain command and control (MDC2) context, the maturity and operational applicability of various C2 applications and micro-applications - all with a mind toward their transition to the Air Force ShadowOC infrastructure and architecture. Flyleaf is an unclassified program. Flyleaf's focus is on the multi-domains of air, space, and cyber.

• From Tom Clark's Brief: To date, every effort socialized through Kessel Run has required the industry partners to be from the greater Boston area. Is AFRL inferring that any further effort (beyond the scope of the BAA) may mean participation by a non-Bostonian company? Else, what was the purpose of specifically mentioning this? Kessel Run was only mentioned for context and to show that the OPML program is aware of Kessel Run and its mission to replace AOC WS 10.2.

• Where is CRC ID # generated? (Aircraft vs Geese) - from D. Marks presentation. Does the ID# have structure? What is the structure/key? Due to answer specificity, please contact OPML TPOC for Distribution D based answer to this question.

• Has UCI been applied to operational - level workflows? Yes

• How do AFRL AI/ML MDC2 efforts align with efforts currently underway or sponsored by the Office of the Chief Architect and those of AFLCMC C3I & N? AC2 CTC has briefed AI/ML based MDC2 efforts in its portfolio and coordinated with AFLCMC C3I & N and PEO Digital

• Where in the USAF, DoD, or OSD is the integration of CONOPS, IT, + Workforce Development being worked? Not sure to what portion of AF (C2, MDC2, ISR, etc.) this question refers to.
Technical Questions
• Are there any preferred tools or programming environment for the software prototype development and delivery in Task 1? Programing environments and tools are up to performer. Open-standards and industry best practices are encouraged. Lock-in and dependence of proprietary software is discouraged.

• The BAA shows the sources of some possible input data. Will the government provide some of such data to the project teams for testing and validation of the developed software prototype when executing the projects? We are not providing that data. Each performer is responsible for identifying specific data requirements and we will do our best to help with the acquisition.

• Will the program consider the uncertainty quantification in the contingency planning? It seems that this factor is not emphasized in some topic areas, although briefly mentioned in the program goal. Uncertainty quantification is of relevance, and is likely to be use-case specific.

• Related question; how much effort to put into building interfaces to other systems (e.g., to get air crew information for TACC missions)? The level of effort is dependent on the approach and domain. The focus is on the ML-based function. Other systems can be stubbed in.

• Is there an objective to push OpML processing out to the combat "edge", to the CRC's and to potentially account for disconnected operations? OPML does not have disconnected operations as an objective. The use of CRC's as it relates to Battle Management functions is considered within scope.

• What are the criteria upon which the AI/ML algorithms will be judged? The BAA provides criteria for source selection with respect to AI/ML application in task 1. The performer develops evaluation criteria following task-dependent templates in BAA. For task 2, we want simulation executions on the order of seconds.

• Technical Task 3 in the "Task 1 Technical Tasks" involves "with SME input, identify[ing] and develop[ing] input training data." Does AFRL envision a generative data approach driven by SME-provided heuristics? All approaches will be considered, "identifying and developing input training data" implies that the performer might develop the data using generative means. This is not necessarily the case. Finding a relevant dataset is sufficient. Generative approaches will be considered but are not required.

• Technical Task 7 in the "Task 1 Technical Tasks" describes developing a multi-domain use case or workflow for the proposed approach. Does AFRL envision this analysis to be restricted to varying capabilities/assets between domains, or should performers also consider different command chain hierarchies, operational languages, etc. between various services? No restrictions on developing a use case or workflow. The more detailed and realistic the workflow is to current operations, the better.

Programmatic Questions
• Are you open to follow up discussions in the next week of two? Yes, please contact the POC for availability.

• Can a company submit multiple proposals to Task 1? Yes, no limit to the number of white paper submissions from a single company.

• Are multiple related AF Threads acceptable as a single proposal? Yes, to be clear we are requesting whitepapers first.

• Are we restricted in classification of data (is this unclass only)? Yes, there are restrictions. Allowed classification levels are up-to-and-including SECRET, as mentioned in the BAA.

• P7, Under 'Task 2 Technical Tasks, 2. (Phase 2a) "Identify SMEs with relevant experience," - Can the SMEs be from, be asked to participate from the active Air Force, for example those serving on the Joint Staff or the Air Staff? Yes, it is up to the proposers to acquire SMEs, either as part of their internal team, or as a consultant.

• In the "potential evaluation criteria" tables of the operational threads attachment, there is no mention of evaluating ML approaches based on their required dataset sizes, training time requirements, or execution speeds. Are these metrics out of scope for this effort? Potential evaluation criteria, literally meant as "potential"; therefore, you are welcome to pick the one that makes most sense for your approach.

• The BAA mentions awards would be for 24 months, but the schedule including options is 27 months. Should we assume a 24-month schedule? Assuming this refers to Task 1. Task 1 schedule is a 9 Month prototype development + 6 Month refinement. Task 1 option is an additional 12 Month priced separately.

• Is on-site presence required/preferred? No, an onsite presences is not required.

• Is building on StreamlinedML a hard requirement? No, this is not a hard requirement.

• Can it be combined with other framework? You can combine with other frameworks, locked in and proprietary systems are discouraged. No end-to-end systems.

• Different/larger award range for Task 2? Please refer to the BAA.

• Will AFRL be integrator for Capstone? Yes

• In Table 2, what is meant by an "Air Thread"? Operational Workflows for those functions.

• How much weight of effort towards M & S dev is acceptable? Depends on your approach to the problem.

• Is there any preference on the teaming? (small business of large business, alone, or combinations with academia?) No, there is no preference.

• If the white papers we submit only answer partial portions of your requirement will you suggest possible partnerships? (Not everyone has or can provide a full end to end solution.) We are not asking for End-to-End solutions. Forming partnerships is the responsibility of the performers. The Government will not be recommending partnerships.

• In what format will the offeror deliver the design of the simulation and evaluation environment? And does Task 2 involve an evaluation of the environment itself by the USAF? We are open to any format for the simulation. Open-standards and industry best practices are encouraged. The USAF would incrementally evaluate the environment during the life of the contract.

• For this BAA, how much of focus is specificity vs. generality? I.e. engineering solution for a specific application vs. novelty of algorithmic approach? We are looking for a specific solution to an Operational process. Algorithmic approaches need not be novel.

Scope Questions
• Are there some threads more important than others (e.g. MOG Planning or Tactical Chat)? No, there is no preference of one thread over another.

• Is there interest in applications of AI/ML to operational C2 outside of the 8 threads in the BAA? Yes, the threads were meant as examples of potential areas. We are open to all applications to any operational C2 process.

• Will the government accept proposals that focus on future systems, e.g. UAVs vs. existing manned fighters? No, the focus of the BAA is on existing C2 processes and systems.

• Does AFRL have any interest in an anytime ML algorithmic approach, particularly for in-mission operational threads? All ML algorithmic approaches will be considered.

• Are there specific CONOPS of interest, e.g. A2AD, SEAD, COIN etc.? No. Choose the one that makes most sense (availability of data, well defined con-ops) for your proposed approach.

• Preferred domains for multi-domain? Air/Space/Cyber but we are open to others as long as they are within scope as described in the BAA.

• It was stated that algorithms for ISR are out of scope. What about algorithms/frameworks for ISR C2? Out of scope.

• Is it relevant to the BAA: The AI/ML mission planning in the presence of sections of Autonomous UAVs? Yes, mission planning is within scope as described in the BAA.

• Is it relevant to the BAA: The planning of new operations that include new Man Unmanned Teaming (MUM-T) in the battlespace? Yes, planning is within scope as described in the BAA.

• ISR not in scope, but how about multi-agent target tracking and engagement? This is application dependent, multi-agent tracking and engagement is within scope as long as the solution is applied to a non-ISR operational C2 problem.

• What is the level of interest in Nonlinear Independent Component Analysis? This area is of less interest unless data-driven or accurately mapped to a current process or workflow.

• What is level of interest in these machine learning attributes/capabilities: - Explainability (of autonomous decisions)? - Casual Reasoning? Both are of interest as long as they are applied to an operationally relevant problem within scope of the BAA.

• What is level of interest in these Technologies/Techniques: - Evolutionary Algorithms? - Information Theory? Both are of equal interest as long as they are applied to an operationally relevant problem.

• Is the Battle Management being considered only at the CRC level or also as applicable to JSTARS/AWACS/ASOC? Yes, applicable to all platforms listed in your question.

• What kind of Machine Learning (ML) algorithms will be considered? Any one that can solve the problems? There are no restrictions in algorithm consideration. The algorithm chosen should solve the problem in question.

• What is the scope for Battle Management (CRC) that ML should be applied to? I.E. Track on Radar, Discussion Support, ...? Currently there is no limit with respect to the CRC function.

• Are you looking for overall solutions or solutions for individual problems? The OPML BAA is looking for ML-based solutions to individual functions.

Repeat Questions
• Do you have preferred data sources? No, there are no preferred data sources. However, data should be representative of what is expected in the real world.

• Are their curated data sources? If so, what are they? No. Already answered.

• It is stated on p3 task 1 "The use of ML must leverage a repository of training data in order to improve the predictive assessment of future data points." Does this mean that you have a repository of data that we are required to use? No. Already answered.

• Do you have any repository of data? No. Already answered.

• Will the government provide any data sets to support machine learning/training? No. Already answered.

• Multiple whitepapers by a proposer? Yes

• Can offerors propose against multiple operational threads? Yes

• Is historical readiness data from AOC's available: - How many missions of varying types were scheduled vs. flown? - Maintenance of aircraft availability? - Would a DD254 be necessary, possible for this type of data? We are not providing this data.

• Where/how can we obtain more information about the Flyleaf environment? Already answered above.

• Would you consider AI/ML applications that support other functions of the AOC beyond those defined in the operational threads, such as spectrum management? Yes

• Since data cannot be provided, can AFRL provide detailed process map of the data types, structures API that will go into task 1 so industry has an accurate representation of the process flow needed to be improved. That is characterization of the data if not the real data. No, the government will not provide process maps, API's, etc. Process map, data types, etc., should come from proposer team as described in BAA.

.