ILMP Industry Day QnA 2020_06_08.docx

Interactive Learning for Mission Planning Industry Day Q&A

FA8750-20-S-7001

1. Machine Learning Questions

1.1. Do I have to use a learning algorithm that falls into what's traditionally considered active learning?

Not necessarily. We like that interactive learners pose easy to answer queries, and not too many of them. We will seriously consider any learning paradigm satisfying those criteria.

1.2. Are there any AI/ML techniques that must be used?

No. We don't want to limit creativity, so any approach that efficiently poses easy to answer questions to real humans and learns effectively from the responses will be considered.

1.3. Are only supervised approaches allowed?

ILMP's primary motivation is the disconnect between existing autoplanners and operational SMEs. Therefore, some SME feedback or labeling should be done. However, unsupervised approaches may play a role in the overall process (e.g. to "warm start" an IL algorithm).

1.4. How does the government intend to measure the utility that the autoplanner plus learner provides, as measured against the baseline (autoplanner only)?

We expect all efforts to propose baselines for human only planning, machine only planning, and, if sensible, human+machine planning without learning. Such baselines will be necessary to evaluate progress made by introducing learning. The Government expects performers to propose planning measurement specifics, because they are autoplanner and domain dependent. That said, the goal of the program is to reduce time and manpower costs of planning, and improving plan quality if possible. Time improvements of human+planner+learner can be measured over human, planner, and human+planner configurations. Quality is slightly more difficult, but, for example, given a generated plan one could measure how much change to the plan is required for human acceptance (edit distance). We assume stronger learning would reduce the change required to reach a good plan. Qualitative evaluation from SMEs is also encouraged.

1.5. Is there interest in exploring if a single learning technique applies across autoplanners from multiple domains and/or echelons?

Yes, as well as multiple learning techniques for a single planner.

1.6. Is there interest in augmented reality to facilitate interactive learning?

No.

1.7. Is there interest in performing the interactive learning during live flight?

No.

1.8. Is it within scope for SME requirements to include considerations of plan flexibility and robustness in response to new information and events that may arise during plan execution?

Yes, this is within scope. Note that human SMEs contributing to efforts under this program may consider any criteria they believe to be relevant to the given planning task.

2. Autoplanner Questions

2.1. Is the need to identify and access an existing autoplanner tool required?

Yes.

2.2. Will you provide access to the mission planning software or do you expect the successful respondent to come with that?

Come with your autoplanner.

2.3. Is there any consideration of making the autoplanner(s) GFE?

No.

2.4. My autoplanner does not have a user interface. Should the learning system provide one for human SME use?

Yes. Ideally the learning system, autoplanner, and user interface are loosely coupled.

2.5. Before learning, can plans be exhaustively “pre-generated” for all input configurations? This might allow for efficient experimentation with learning algorithms as the planner would not need to be re-run?

This approach of pre-generating data is often used in the active learning community for evaluation and is well within scope of this program This approach for evaluation provides a baseline of autoplanner data that can be used to evaluate and test different learning approaches. It is related to BAA criteria (a).

2.6. Would it be acceptable to utilize a military-relevant autoplanner that is currently being developed on another effort if no autoplanner development is done on ILMP?

Yes. Be clear in your proposal how the separation of tasks will be enforced.

2.7. We are currently developing an automated autoplanner that solves some Air Force planning problem. This system enables users to edit the plan manually to improve the plan. For the ILPM program, we would enhance this planning system to learn from user edits to improve its own performance. Would this planning system be in scope for this program?

Yes. However, autoplanner development must be separated from ILMP. Be clear in your proposal how the separation of tasks will be enforced.

2.8. What mission planning software do you envision the solution to be integrated with?

No particular autoplanning software is required. Choose your autoplanner that you think most suits this approach to prove the utility of interactive learning in planning.

2.9. Are there recommended autoplanners, perhaps as part of systems of record, that we should be considering? If so, are there POCs associated with those autoplanners?

No specific recommendations. Propose a planning domain and autoplanner in which you have expertise and can measure the utility of interactive learning.

2.10. Any particular restrictions on domains for autoplanners?

Autoplanners for this program must have direct relevance to an Air Force planning problem. Purely academic or commercial planning problems are not acceptable.

2.11. Is there any interest in closing the loop between tactical and operational planning?

No.

2.12. Is there a preference for a tactical or operational autoplanner solution? Or are they deemed equal?

No preference.

2.13. Can I propose to use multiple autoplanners?

Multiple autoplanners may be proposed so long as the scope of experiments and deliverables is properly accounted for in proposed cost and time.

2.14. What if I have a planning domain with a hard to use existing autoplanner, or no autoplanner at all?

In this situation we question whether or not the domain is of sufficient complexity to warrant study. Or it may be too new/underdeveloped, and not at the correct stage in its lifestyle for this program.

2.15. What changes to an autoplanner are allowed?

· Configuration: We expect that autoplanner configurations will be a valuable asset allowing for a learner to influence the planning artifacts generated.

· Code changes: Generally no. We would like to strictly limit changes to autoplanner code. Exceptions include changes that make available otherwise unavailable autoplanner configurations (e.g. changing a private member variable to public in a class). The core optimization and modeling should remain unchanged.

· Code certification: As this is a 6.2 effort, code certification and deployment to operational networks is not a main focus. Choose a deployment strategy that is easy for your organization to execute and deliver, and for the Government to reproduce your results.

· Other changes: Must be clearly motivated and identified in proposal submission

3. Experiments Questions

3.1. Are synthetic human preferences ok?

For getting things up and running, yes, but the primary goal of the program is to develop something that works well with input from real SMEs. Proposals must include a centerpiece evaluation component that involves real people. However to quantitatively measure and evaluate the merit of the technical approach and establish baselines the creation or use of synthetic preference data may be preferred as a foundation for more select human evaluation.

3.2. Should the proposed solution be able to quantitatively demonstrate utility with respect to autoplanner performance, rate of convergence, and generalizability?

Yes

3.3. Should the proposal identify evaluation data sets for baseline evaluation?

Yes. A clear technical baseline should be defined and occur early in the effort with complete quantitative results. Baseline data should be part of your proposed deliverables. If, for some reason, this is not possible, state so clearly with an explanation of why in your whitepaper.

3.4. Should the results be reproducible by the government team?

Yes. This should be reflected in proposed deliverables. All data and all software (including the autoplanner itself) required to rerun experiments must be part of your proposed deliverables.

3.5. Should the proposal indicate early in the effort the establishment of an appropriate baseline and how the baseline will be measured?

Yes.

3.6. Is there an emphasis on (1) shaping a plan in advance or (2) providing real-time re-planning and SME input during plan execution?

Emphasis at this early stage of the program is on advance or deliberate planning. However, the Government has considered dynamic planning as well, and will entertain proposals in this area.

3.7. Should systems assume that mission teams have both humans and autonomous systems (forward looking) ?

Systems need not assume this to be the case.

3.8. Am I to focus on an IL technique that will work for a single scenario, and/or with a single SME?

Two notions of generalization:

1. The learned model will work across scenarios/SMEs (e.g. parameters transfer).

2. The general approach is likely to be successful in another scenario, even if, e.g., the parameters of a learned model don't transfer.

We would like to see experiments showing how an approach generalizes, since this is 6.2 work and not 6.3. So while the focus can be on one choice of scenario/SME, applicability to other cases must be addressed somehow.

4. Program Questions

4.1. Has the budget been approved?

Yes.

4.2. Will multiple awarded efforts be completely independent of one another, or do you anticipate that there will be some degree of synchronization (e.g., common events) among them?

Proposers may submit multiple white papers, or team with other firms, but no contractor interaction will be required. The Government anticipates multiple awards largely independent of each other. For example, interactive learning conducted on offensive tactical air route planning might not transfer to cyber defense planning. The emphasis is on measuring the utility of interactive learning on AF planning. The Government seeks insight into an array of autoplanners to include both operational and tactical autoplanners in the air, space, and cyber domains. The Government does not expect an effort with an autoplanner to solve all learning problems for all domains, but to measure the utility of interactive learning within the domain of your autoplanner.

4.3. Is there any preference toward either shorter (e.g., 12 month) or longer (e.g., 24 month) efforts?

No. The Government will evaluate your proposed schedule in accordance with the criteria set out in the parent BAA under criteria (a) and criteria (d).

4.4. What is the anticipated relationship between the 24-month maximum duration of individual awards and the 36-month duration of the BAA?

No explicit relationship. Efforts will be complete by the end of Q3 of FY 23.

4.5. Is there a preference for either unclassified or classified scenarios?

No particular preference. Efforts awarded under the parent BAA may be or incorporate data up to and including SECRET//NOFORN.

4.6. Would starting with unclassified scenarios, possibly progressing later to classified scenarios be acceptable?

Yes.

4.7. How does this BAA relate, if at all, to other BAAs in AFRL/RI’s Autonomy Command and Control (AC2) CTC that involve machine learning, such as OpML?

This BAA is independent of other BAAs in the AC2 portfolio. Other BAAs within the AC2 portfolio may provide context that offerors find helpful and all offerors are encouraged to read those other BAAs.

4.8. Is the Flyleaf environment relevant to this BAA, and if so, how is it relevant?

Deployment to Flyleaf is not required. External control interfaces (e.g. API, messaging layer) to the learner or autoplanner are desirable. The Flyleaf BAA is number FA8750-19-S-7013.

4.9. One of the references is to the DARPA RSPACE program, which developed automated operational-level autoplanners; how is the DARPA RSPACE program relevant to this BAA?

The DARPA RSPACE program built some of the autoplanners and forged some of the lessons learned that inspired the ILMP program.

4.10. Are you open to additional communication following the publication of the question responses? In particular, are you able and willing to review and provide feedback on short (i.e., 1 page) abstracts of potential white paper submissions next week?

No.

4.11. Given the COVID-19 situation, and the short proposal turn-around time, the normal method of sending out white paper evaluation results by US mail is even more problematic than usual, as many people are still working from home. Will you send out white paper evaluation notifications by email for this BAA?

Yes. Email and regular mail.

4.12. Is a list of the virtual industry day “attendees” available to facilitate teaming discussions?

Yes. Company name are provided with this Q&A document.

5. List of Attending Companies Black River Systems Company

Neya Systems

Infoscitex

Data Intelligence Technologies, Inc.

Neya Systems, LLC

Agile Technology Solutions, LLC

Knowledge Based Systems Inc.

Discovery Machine Inc.

Siege Technologies

Siege Technologies

Global Infotek, Inc.

Draper

Raytheon Technologies Research Center

SRI International www.trek10.com

CACI Inc. LGS Labs

Soar Technology, Inc.

Booz Allen Hamilton

Booz Allen Hamilton

Honeywell International

Attollo LLC arcarithm

Polaris Alpha

Stratagem Group

Lockheed Martin

Spectrum

Vanderbilt Univ

DCS Corporation

Forrester

AFRL/RISC

Deloitte

Booz Allen Hamilton

Raytheon BBN Technologies

BAE Systems

HRL Laboratories, LLC

HRL Laboratories

Adaptive Immersion Technologies

Intelligent Automation, Inc

Evans & Chambers Technology ebase

JHU/APL

ANDRO Computational Solutions, LLC

GE Aviation

Dynatrace LLC

Monterery Technologies Inc.

Draper

Problem Solutions

Northrop Grumman Systems Corporation

Parsons

Aptima, Inc.

AIS

Systems & Technology Research

Radiance Technologies

Trideum Corporation

Riverside Research

C5T Corporation

Northrop Grumman

Integration Innovation Inc

The Boeing Company

Boeing/Jeppesen

Lockheed Martin Missiles and Fire Control

Tapestry Solutions, Inc.

Progeny Systems Corporation

CACI

Lockheed Martin ADP

Lockheed Martin

MongoDB

USAF

Boeing Phantom Works

RADIANCE TECHNOLOGIES

Triumph Enterprises

GE Aviation

Parsons

CACI

Design Interactive Inc.

Securboration, Inc.

Peraton, Inc

Lockheed Martin

The Boeing Company

Booz Allen cubrc, inc

SRI International

Lockheed Maritn

Pacific Northwest National Laboratory

Lockheed Martin

Polaris Alpha

Stottler Henke Associates, Inc.

Lockheed Martin Advanced Technology Labs

Starke Solutions

Siemens

Boston Fusion Corp.

SRI International

Design Interactive, Inc.

BAE Systems FAST Labs

RADIANCE TECHNOLOGIES, INC.

Lockheed Martin RMS

Southeastern Computer Consultants, Inc.

PARC (Palo Alto Research Center)

Toyon Research Corporation

Perspecta Labs

Palo Alto Research Center Inc. (PARC)

PARC

PARC

Aurora Flight Sciences

Siemens Corporate Technology

Quanterion Solutions Inc. (QSI)

Lockheed Martin

Sikorsky, A Lockheed Martin Company

Physical Optics Corporation

Booz Allen Hamilton

SRI International