18-03008_REVISED_-_NFFS7300-18-03008_SOW_-_01AUG18_final.pdf

VIAME Modifications and Image System Integration

Statement of Work (SOW)

Background The Polar Ecosystems Program at the Alaska Fisheries Science Center’s Marine Mammal Laboratory (MML) is responsible for estimating the abundance and distribution of ice-associated seals and coastal harbor seals of Alaska.

We conduct large scale fixed-wing surveys of the pack ice in the Bering, Chukchi, and Beaufort seas in a collaborative effort with international researchers in order to accurately assess transboundary population estimates. We have also tested the feasibility of using unmanned aerial systems (UAS) to reach areas of the Bering Sea still inaccessible by manned survey aircraft. Both manned and unmanned surveys result in the collection of large amounts of imagery which, historically, have been evaluated manually or, more recently, using a semi-automated approach.

An autonomous payload is required to integrate UAS into surveys of ice-associated mammals, in order to improve the efficacy and human safety in gathering essential data for NOAA stewardship. Additionally, post-processing methods are needed to extract animal sightings from the imagery in a timely manner. A human-operated system has been developed to collect and process thermal and color imagery for the detection and classification of animals on the sea ice. This system was used during surveys of the Bering and Chukchi seas using occupied aircraft. A key objective is to use thermal data in real time to trigger paired color and thermal image collection, minimizing the collection of extraneous imagery that requires burdensome data storage, management, and processing. In its present state, the system requires continuous data recording, supervision, and extensive post processing. Further development is needed to improve the detection algorithm.

Objectives A firm, fixed price contract is needed to develop an environment in which to test advanced algorithms focused on detecting seals and polar bears on the sea ice. This test environment should be developed within NOAA’s VIAME framework (see viametoolkit.org for details) to quickly process imagery, identify and classify detections, and record data in a format compatible with our database. Ultimately, this work will allow us to develop detection algorithms in house, which will then be integrated with the in-flight control software for real-time detection. The color and thermal cameras have different frame sizes and ground resolutions, so image registration is challenging yet necessary to allow detection and classification algorithms to access all available wavelength bands (RGB and LWIR). The contractor shall have previous experience with implementation of advanced algorithms using deep learning techniques (such as R- CNN or YOLO) for wildlife image processing including identification of species. To be clear, this work does not include annotation of imagery or algorithm development, but rather creating/modifying the VIAME framework to accommodate these images and specific goals so that these deep learning approaches can be applied.

Technical Specifications

1. Provide framework for training and testing detection and classification algorithms to meet image processing goals utilizing imagery from multiple wavelength bands (RGB and LWIR) with in the NOAA VIAME image processing framework.

2. Provide simultaneous viewing of paired color and thermal imagery.

3. Provide environment in which algorithms are able to track detected objects in subsequent frames.

4. Ingest pre-registered and unregistered imagery (20 MB color and 170 KB thermal) and data files (csv).

5. VIAME shall run detection and classification algorithm(s) on both thermal and color imagery, either in line

(I.e., process thermal and then color), or simultaneously (allowing access to all bands) depending on selected approach.

6. Register images of different size and frame rate to allow simultaneous processing and identification of detected objects in both thermal and color images, providing image chips of objects of interest.

7. Ability to confirm/correct/test algorithm results.

Additional elements

1. Software must be able to run on US government computers and servers

2. Provide specifications for image processing machines to meet power requirements of algorithms

3. Written manual/instructions

4. Additional 50 hours of support

5. Work may include one trip to Seattle, WA, for implementation and testing. This trip is not to exceed one week.

Project Tasks/Timeline: We expect the time of award to be no later than (TO BE DETERMINED) THE DATES BELOW ARE ALSO SUBJECT TO CHANGE BASED ON THE AWARD DATE. We will defer to the expertise of the contractor, however, we do envision these general tasks to be completed in this approximate timeline:

Task # Description of Task Projected Completion

1 Coordination meeting August 2018 2 Image registration complete Oct 2018 3 Algorithm testing Nov 2018 5 Final product delivered July 2019

Option # Description of Option Exercise Deadline

1 Integrate UV spectrum (or other imagery) into display and detection algorithm June 1 2019

Task 1 – Coordination meeting Upon award of the contract, expected by (TO BE DETERMINED), the contractor and MML biologists will have a coordination meeting to clarify goals and objectives so there is an understanding of steps and procedures from both sides. Also, the contractor will be connected to the team developing the in-flight system to ensure compatibility.

Task 2 – Image registration complete The contractor shall receive a package of images from MML at an agreed upon date decided in Task 1. This package will include images that have already been processed and a csv file of hot spot detections and classifications. The contractor is expected to build code compatible with VIAME to register the overlapping images within the image set (i.e., identify the pixel coordination of paired color and thermal images).

Task 4 – Algorithm testing The contractor shall work with biologists and detection code developers to test algorithms within the image processing framework. Modifications/corrections to the framework will be identified and implemented.

Task 5 – Final product delivery MML is willing to negotiate the Project Task timeline as long as it does not significantly diverge from the one listed in this Statement of Work and assuming the final products described below are delivered no later than July 1, 2019. The contractor will ensure functionality in VIAME and the in-flight system. Once operational onsite, the contractor shall ensure the system is performing at the level defined in this statement of work.

Option 1 – Integrate UV data for improved polar bear detection In the case that UV (or other) imagery is available to improve polar bear detection, this option will be exercised no later than June 1, 2019, contingent on funding. UV imagery will be collected from target survey altitude of 1000ft. UV imagery can ease animal detection on ice because mammals absorb UV rays that are reflected by the sea ice. These images will show polar bears (and other species) as black objects on the white background of monochromatic imagery. UV imagery will be registered with color and thermal imagery and analysis integrated into the workflow.

Deliverables Each Project Task listed in the Statement of Work will result in a deliverable provided to MML by the Projected Completion date detailed in the Timeline or otherwise agreed upon by the Vendor and MML.

1. The Coordination Meeting will result in a document providing any modifications to the Timeline.

2. The Image Registration code/module shall be delivered to MML electronically, as well as integrated into

VIAME.

3. The final product shall be delivered to MML electronically, as well as integrated into VIAME in person at

MML in Seattle, Washington, to ensure functionality.

Government-Furnished Property, Material, Equipment, or Information (GFP, GFM, GFE, or GFI) MML will provide a package of processed images from previous years’ surveys along with an excel sheet of hot spot data corresponding to the images. This is public property and government furnished materials. These are not sensitive materials and are public information. The contractor understands that this package and the final products are the property of the government and are public information. These images are not to be shared with anyone outside of the vendor until the completion of the crowdsource event. If Option 1 is exercised, the Vendor shall establish an avenue to develop the detection algorithm(s) and is free to share images as necessary to complete this task.

Security There are no unique security requirements associated with this contract performance. However, it should be understood by the contractor that all images and data are public property collected and managed by MML, NOAA.

Place of Performance The contractor will be responsible for processing materials, and developing the final algorithm(s). The final product delivery will be performed at MML.

Travel The contractor shall travel to Barrow, Alaska, for one week to test algorithm functionality in flight.

Period of Performance

The period of performance will begin upon awarding of the contract. The contractor may negotiate the Project Task timeline with MML. The contract shall expire September 30, 2019.