ATAK IVAS_Commercial Solutions Opening.pdf

Commercial Solutions Opening ATAK & IVAS plug-in integration

SUMMARY

The Army Artificial Intelligence Integration Center (AI2C), in collaboration with the Army Contracting Command – Rock Island (ACC-RI) is seeking commercial technologies to integrate custom plug-ins for Android Tactical Assault Kit (ATAK) and the Integrated Visual Augmentation System (IVAS) in order to interface with a team of autonomous air and ground sensors.

Project Description and Requirements:

This project involves integrating custom plug-ins for Android Tactical Assault Kit (ATAK) and the Integrated Visual Augmentation System (IVAS). Both ATAK and IVAS will be used to interface with a team of autonomous air and ground sensors. The autonomous systems are designed to perform zone reconnaissance missions and use onboard AI processing to identify, localize, and report all threats within the designated search area.

The goals of this work are to: 1) ease the ability to communicate with systems in terms of relaying intent and receiving feedback in return (e.g., target details, recommendations, etc)., and 2) shorten the sensor-to-shooter timeline through intuitive interfaces that facilitate the rapid validation of targets and the relay of targeting info to effector systems.

Top Priorities for ATAK Plug-in:

Zone recon functionality

1) Ability to easily select zone to recon on map via finger touches to build polygon (or touch and drag to build a rectangle). Zone should be highlighted with map light/shaded overlay.

2) As systems search the zone, overlay should change colors or lighten in color so that the user can distinguish which areas have been searched thus far.

3) Display percentage of zone search completion inside the zone as an overlay.

4) Provide “field of view” fan of where sensor is currently “looking” as shaded sector from vehicles’ marker/symbol.

Threat alert functionality

1) Add features for “Follow target”, request “new thumbnail”, request “FMV” – full motion video, “call for fire”

a. “Call for fire” should be integrated with Firestorm and is essentially a quick way to automate the normal FIRESTORM workflow

2) Show classification confidence of detection

BDA (Battle Damage Assessment) functionality

1) Incorporate ability to click on a “red threat”, then select “BDA”

a. This would send a command to a system to take snapshots of the target location and append those as new thumbnails

ATAK Desired Features:

• Mission Planner Features o User can define the priority (rank-order) of threats they’re concerned about Prior to mission, user can select which targets they are primarily interested in searching for during recon (targets will vary between ATAK-CIV and ATAK-MIL version)

Perhaps the user is given a list in which they can reorder based on their preferences

Perhaps user is given option for fine-grain vs coarse labels (military vs civilian vehicle, friend vs. foe, or specific-vehicle classes.)

• At a minimum, the user should select what targets they are interested in searching for and rank them according to priority

– which is the highest threat and so on)

• Given edge compute constraints, this priority information may be used to in the future to determine which trained model to use during inference (tradeoff - as number of classes goes up, computation/performance goes down, etc.)

Ranking by user can be used to rank order threats found so far during mission (perhaps a side window shows a running list of threats detected – highest threat at top and so on) o User can select mission task & subsequently provide mission guidance Route recon

• waypoint navigation along a user-defined route Zone recon

• User can select zone via finger touch sketch of shape

• User should be able to select multiple, disjointed zones

OP (Observation Post)

• Sensors move to a waypoint or area where it will stay and scan for threats

• Modify field of view when stationary – requires message publishing back to PTZ module Stationary Screen (future work)

• Sensors autonomously maneuver to areas around manned formation, looking for main avenues of likely enemy approach (e.g., roads) and then establish appropriate OPs to maintain over-watch

Moving Screen (future work)

• While manned formation moves, the sensors autonomously move alongside or in rear of the formation protecting flanks Decoy (future work)

• Sensor maneuvers to an area away from manned elements and deceives threats through movement and/or emitters

NOTE: At any point, the user should have the ability to re-task an asset to a new mission (e.g., a UAS performing zone recon could be given the mission to perform a specific waypoint navigation (route recon) mission.

Note: BDA mission is embedded as a sub-function after a threat detection occurs and once a marker is on the COP (e.g., user can select a marker, then select “perform BDA” or recommendation presented to the user to perform BDA after forwarding a threat to

FIRESTORM)

• Mission Execution Features o After planning mission, user should be given options “Start Mission” “Halt in Place” “Return to Base” o During mission, user should be able to select a UGV/UAS elect to see its full motion video (FMV) see sensor health (battery life/time remaining) estimated time to destination (or mission/recon complete) o Sensors should trigger warning when battery life is low & will soon need to return for recharging o Once a sensor reaches its final waypoint or OP, user should be able to change the sensors orientation (sensor’s field of view – direct sensor is facing) o ATAK should show sensors approx. field of view on map as shaded or subdued fan overlay – to give user an idea of where sensor is currently scanning This will allow the user to see where the sensor is currently scanning If a system is given a waypoint mission, then after reaching its destination, the user should be able to “control” the PTZ or where the sensor is scanning o User should be able to scroll list of threats “found so far” Perhaps a side window can slide over from right-to-left when prompted by user, and in this window, it provides a rank order lists of the threats found thus far

• Rank-ordering should be based on preferences chosen by user during mission planning phase o COP should have an overlay (shaded area) that indicates the regions within the zone that have been searched so far Additionally, it would be helpful to show on the COP a percentage

(%) of the zone searched so far

• Threat Presentation o Threat alert popup notification Should include the following information:

• Thumbnail image of threat

• Class or type of threat

• Confidence level

• Geolocation of threat o Distance of threat from you – the user

• Orientation of threat (facing you? True or False)

• Moving or Stationary o Moving toward you? True or False If True, approximate speed of threat

• Timestamp (calculate “last seen” value based on current time)

• Unit size (grouped near other threats? – approximate unit type)

User should be asked whether threat is valid (I.e., a true threat and not a fragmented track)

• Threats should populate map as the appropriate marker/class type (e.g., if T-72 is detected, then it should be a RED diamond of the right type, etc.)

• If user is unsure from thumbnail, then there should be an option to request more thumbnail images (sensor should infer to get closer, different perspective)

• User should get option to reclassify threat if class ID is incorrect or label as a false positive o Sample images should be stored for future retraining of classifier (via communication to sensor) o False positive marking – at marker level user should be able to designate a detection as a FP, then that propagates to all thumbnails. Additionally, the user should be able to mark individual thumbnails as FPs

• User should be able associate multiple detections/tracks as the same (or single) track along w/ associated symbols (and thumbnails) o In theory, this feedback could be used to teach or optimize the tracker algorithm

Upon confirmation of a valid threat, user should be presented options

• Track threat (i.e., follow threat by keeping sensor “locked” onto threat) or continue recon o If “track threat” selected, ask user whether FMV is desired or periodic thumbnails to be sent for BDA (battle damage assessment) purposes o Tracking the threat (following in field of view) will allow for auto-adjust fire capability

• Nominate for effect (I.e., forward to FIRESTORM) o Optimize this handoff to FIRESTORM (e.g., attach thumbnails directly from marker to FIRESTORM VMF message without having to manually do this from the image gallery) o Immediately after forwarding threat to FIRESTORM, ATAK should prompt user whether BDA mission should be performed ATAK recommends optimal asset(s) to perform

BDA (single button touch approves recommended asset for BDA mission) - (e.g., UAS 1 is closer than UAS 2)

• BDA images (snapshots from UGV/UAS) should be appended to the same threat marker (not create a new one)

At any point, user should be able to select a detection (marker) on the COP, then have the option to “Follow”, perform “BDA” o User should have option to define sensitivity of threat alerts (e.g. more or less false positive alerts based on changing of detection threshold using slider bar) o Uncertainty of threat geolocation can be represented with underlying “circular probability of error” via color shading or some other means. As the tracker continues to update its estimate for the geolocation of a threat that circular (ellipse) error should eventually narrow.

IVAS Desired Features:

• Mission Planner Features o Leverage voice recognition, gestures, and/or augmentation to relay intent for where systems should conduct their reconnaissance mission o Leverage marker overlays (IVAS object detector) to convey direction to systems via voice (e.g., “recon 1 km beyond building 3” or “recon beyond that building” as shown through hand gestor, etc.)

• Mission Execution Features o System locations (movement) should be provided (or “seen”) through the augmented environment using marker overlays o Support query of system state information through voice (e.g., “how much flight time is left with UAS 3”)

• Threat Presentation o Show detected threats from autonomous sensors through IVAS’s augmented lens – threat alerts should pop-up on display As part of the alert message, a thumbnail image should be presented to the Solider. At that point, the Soldier should be able to relay voice commands such as “engage”, “another thumbnail”, “track target”, etc.

Under the Commercial Solutions Opening (CSO) as authorized by Section 879 of the National Defense Authorization Act (NDAA) for Fiscal Year (FY 2017 (Pub. L.

114-328) this is considered to be a competitive procedure for the purposes of 10 U.S.C. chapter 127 and Federal Acquisition Regulation (FAR) 6.102. This Announcement is considered a general solicitation; therefore, it is considered a competitive process and includes evaluation criteria for selection for potential award. As a result of the competitive processes used under this CSO Announcement multiple awards of various demonstrations may be executed. Award(s) executed from this CSO competitive process may include, but are not limited to, a FAR-based contract, an Other Transaction Agreement (OTA), and/or a Test Services Agreement.

Under the authority of 10 U.S.C. 2371b, the AI2C may desire to award an OTA to vendors to carry out prototype project(s) that are directly relevant to enhancing the mission effectiveness pursuant to 10 U.S.C. § 2731b through an Other Transaction (OT) agreement. The information provided in this CSO announcement is intended to ensure that to the maximum extent practicable, competitive procedures are used by the Government to enter into agreements to carry out a prototype project(s) and subsequently the potential for follow-on transaction agreements/contracts.

The potential for follow-on agreements and production contracts for projects awarded from this announcement will be in accordance with 10.U.S.C. 2371b (f).

Upon a determination that the competitively awarded prototype project(s) has been successfully completed in accordance with the criteria set forth in the respective agreement, and subject to the availability of funds, the prototype project(s) may result in the award of a follow-on production contract or transaction without the use of competitive procedures.

PERIOD OF PERFORMANCE

ATAK Proposed solutions shall be delivered by July 2021 with at least monthly deliverables provided for testing and feedback for improvement towards final solution.

IVAS Proposed solutions shall be delivered by July 2022 with at least monthly deliverables for testing and feedback for improvement towards final solutions; these shall start being provided after final ATAK solutions have been received.

PROPOSAL ACCEPTANCE PERIOD

Proposals for this CSO will be accepted by the Government two weeks from the date of this announcement being placed on https://beta.sam.gov.

EVALUATION CRITERIA

Proposals will be evaluated to determine if the proposed innovative solution can demonstrate the following. These criteria are listed in descending order of importance, with 1 being the most important and 6 being the least important.

1.) A solution to interface ATAK and IVAS with a team of autonomous air and ground sensors outlined above.

2.) Ability to communicate with systems in terms of relaying intent and receiving feedback in return

3.) Shorten the sensor-to-shooter timeline through intuitive interfaces that facilitate the rapid validation of targets and the relay of targeting info to effector systems.

4.) Ability to demonstrate an understanding of ATAK and/or IVAS interfaces and plugin development using a Software Development Kit (SDK).

5.) Terms of consideration, whether monetary or non-monetary (ex: provision of data), with all terms to be determined as fair and reasonable by the U.S.

Government in accordance with the regulations.

6.) The potential impact of intellectual property and data rights assertions.

INSTRUCTIONS FOR PROPOSAL SUBMISSION

Proposals shall be submitted to the following: Amanda.k.schanze.civ@mail.mil and Kaci.m.christensen.civ@mail.mil.

Proposals must be in a format such as Word or Excel and be written in clear, concise, layman style statements. Proposals shall not include any proprietary information and all formulas used for cost information shall be visible. All proposals shall be commercially priced and come with any and all relevant supporting documentation associated with the pricing. All submissions shall be unclassified.

Submissions containing data that is not to be disclosed to the public for any purpose or used by the Government except for evaluation purposes shall indicate as such on the cover page. If, however, an agreement is awarded as a result of, or in connection with the submission, the Government shall have the right to duplicate, use, or disclose the data to the extent agreed upon by both parties in the resulting agreement. This restriction does not limit the Government's right to use information contained in this data if it is obtained from another source without restriction

Proposals shall not exceed five (5) pages, not including the cover page. Proposals shall include a cover page with the following information:

1.) Name of Company

2.) Company Address

3.) Company POC to include name, phone number, and email

4.) Company Cage Code and DUNS

Interested vendors for this CSO Announcement shall address the following questions in the proposal submissions:

1.) Describe your demonstration solution. Please detail any distinguishing features, characteristics, or constraints of the solution as it relates to this announcement.

2.) Describe how your demonstration solution will integrate easily, for the purposes of quick demonstration, with existing IT infrastructure and artificial intelligence platforms within DoD, use of existing models from government, industry, academia, and open source.

3.) Describe the consideration requested in exchange for the Army’s testing of the innovative commercial solution (e.g. provision of Government owned data, payment of money for services required to assist the DoD in testing the product, cloud consumption usage needed for the demonstration, etc.)

4.) Identify and discuss any Intellectual Property and/or data rights involved in the mailto:Amanda.k.schanze.civ@mail.mil mailto:Kaci.m.christensen.civ@mail.mil mailto:jason.e.caulkins.civ@mail.mil effort and associated restrictions on Government use; identify and discuss what rights are anticipated to be made available to the Government.

QUESTIONS

Questions regarding this Announcement should be sent to:

amanda.k.schanze.civ@mail.mil and Kaci.m.christensen.civ@mail.mil.

DISCLAIMERS

Use of Contractor Support

Non‐Government advisors may be used in the evaluation of the proposals and already have signed Non‐Disclosure Agreements (NDAs) with the Government. The Government understands that information provided in this Announcement is presented in confidence and may contain commercial or financial information and agrees to protect such information from unauthorized disclosure to the maximum extent permitted and as required by law. A respondent’s participation under this announcement indicates concurrence with the aforementioned use of contractor support personnel.

• Booz Allen Hamilton

• Frontier Technology, Inc.

• Carnegie Mellon University mailto:amanda.k.schanze.civ@mail.mil mailto:Kaci.m.christensen.civ@mail.mil