Automated Image Analysis Software - Problem Statement.pdf

Problem Statement for the Automated Image Analysis Pipeline for the Optics Strategic Initiative Working Group

September 2024

Summary:

In marine ecosystems, accurate and timely identification of taxa are critical across a range of natural resource management activities, biodiversity conservation, and ecosystem monitoring. Traditional methods of identification rely heavily on manual observation by trained experts, which are labor-intensive, time-consuming, and prone to errors. Furthermore, with the increasing demand and use of optical sampling methods, there is a need for efficient and reliable automated methods to detect and identify targets from imagery. To that end NOAA Fisheries has funded a suite of Strategic Initiatives with the intent to accelerate the implementation of advanced technology directly into a wide variety of surveys. The Optics Strategic Initiative (OSI) focuses on operationalizing large-scale image collection, processing, and storage; complement, optimize, and replace existing ship- and aircraft-based surveys;

automate data processing pipelines; and provide novel, mission-improving metrics. Specifically, OSI will fund projects to increase the accessibility and functionality of machine learning assisted image processing pipelines, and hybrid cloud processing capacity. Upgraded tools will be deployed on a portfolio of surveys. We envision developing a suite of transformational technologies that enable end-to-end automation of optical sampling methods. Those technologies will be implemented in demonstration surveys (i.e. fish, plankton, benthos, aerial/mammals) selected for their topical relevance to the agency and perceived differences in the image analysis problem being addressed.

Internal polling across NOAA Fisheries identified some common barriers to entry and implementation of automated image analysis including: insufficient expertise/personnel, lack of labeled imagery, access to high-speed computing, computational complexity, interpretability of output, poor software interfaces, low programming flexibility, and inadequate administrative and technical support (see attached figures). Results represent a wide spectrum of uses and thus can have varying requirements (e.g. software interfaces). Thus we provide general software requests as well as more domain specific issues for software functionality. Software investments should consider scalable solutions that empower full implementation of automated image analysis into post-processing workflows across domains. Solutions should seek to provide end-to-end solutions across a range of user levels, improve usability and functionality, provide relevant diagnostics for model tuning and evaluation, and generally seek to lower the barrier of entry for this critical toolset.

Desired functionality across modalities.

● Cloud enabled end to end automated detection and classification

● Improved user interfaces and documentation

● Improved and simpler workflows to label images

● Estimates of body size (e.g. using stereo image methods) using multiple measurements of individual animals enabling calculation of precision metrics for individuals and groups (e.g. mean, sd)

● Ability of algorithms to be operable with camera platform upgrades/changes

● Generation of model metadata for tracking deployment in post-processing workflows

● Batch processing for large image/video collections

● Ability to process data to evaluate behavioral response or label behaviors observed

State of the Tech: Fish There is a wide range of software in use across NOAA Fisheries to automate post-processing of optical imagery for in situ fish detection and classification that by-and-large reflect previous NOAA Fisheries investments. We observe highly variable levels of user knowledge/expertise and therefore have a need to be able to provide solutions across a spectrum of user entry points thus software usability is important for this group. Experienced users requested increased flexibility for data inputs and model control, while novice users focused more attention on user interface and accessibility. Common issues expressed for end-users in this group include the lack of labeled imagery, access to high-speed computing, and lack of expertise to create and maintain models. Other issues center on problems associated with varying conditions experienced in underwater photography such as light availability, turbidity, and image quality.

Requests include incorporation of stereo-calibrations, and accurate tracking of individuals for measuring fish and estimating vital length data used in stock assessment. Specifically:

● Frame level species specific count data, including classification uncertainties

● Ability to annotate using tracks across multiple frames

● Incorporation of stereo-video calibrations to automate ranging, length measurement, and calculation of precision metrics using multiple measurements of an individual

● Behavioral data such as swimming speed, trajectories, proximity to other individuals

● Solutions for easier creation of labels

● Iterative query/response GUI for rapid model generation and tuning

● Ensuring applications work with varying camera configurations

State of the Tech: Aerial/Mammals Aerial surveys for marine mammals involve continuous imagery collections for low density populations such as ice seals, circling over animals of interest for individual identification or group size for cetaceans or harbor seals, and collecting clean image sets of complex haulouts of sea lions where animals are piled upon one another. Recent efforts to analyze satellite imagery for marine mammals offers yet another modality requiring machine learning solutions to reduce the image analysis burden. Additionally, underwater imagery is also used to study foraging behavior with the attachment of animal borne camera systems. Many of these approaches have made inroads into machine learning models that have supported the development of semi-automated processing approaches. To realize the true benefits of fine-tuned and applied machine learning models, additional investment in software is required to streamline processing pipelines and reduce the manual components of image processing.

● Species and age-class specific count data

● Bulk processing of image sets from different flights

● Multispectral and satellite imagery support

● Ability to annotate using tracks across multiple frames and multiple passes over the same animals (duplicate identification based on location and habitat - ice floe, reef, etc).

● Use altitude and camera calibration yaml files to automate body size measurements.

● Event detection for animal borne imagery to evaluate diets

● Identification of species from underwater cameras

State of the Tech: Benthic Existing, image-based survey applications for surveying benthics habitats (e.g. coral), will greatly benefit from platform-agnostic sensor development and the accelerated segmentation based annotation pipeline. For instance, the National Coral Reef Monitoring Program’s (NCRMP) Pacific benthic survey is now entirely imagery based, but still dependent on NOAA White Ships, open circuit scuba diver image collection, and human annotation. This application has high potential for rapid application and tangible benefits of new sensor platforms, AI classifiers, and edge bearing imaging systems. Much of this work will focus on classifying sponge and coral but solutions are also sought for demersal epifaunal communities (e.g. Sea Scallops, Conchs, urchins, giant clams, etc). For example, current optical surveys for Atlantic sea scallops require manual annotations of number and size of scallops. Manual annotations are labor intensive and do not allow for processing all viable data collected. This application would allow for increased productivity, streamlining data processes, increase data use for sea scallops, and expand the survey data to other stock and ecosystem applications.

● Species specific spatial distribution and percent coverage data

● Colony/Organism specific segmentation for size, growth, mortality, and recruitment

● Bulk processing of image sets from transects and mosaics.

● API to allow easy and automated deployment of a trained classifier on new images.

● Repository of trained classifiers for different applications to select from.

● Photogrammetric Modeling and Scaling to support 2.5D Annotations

● Enumerating and measuring demersal fish and invertebrates for deriving standardized abundance estimates and size classes

● Automated substrate classification

State of the Tech: Plankton Plankton are microscopic organisms that are incredibly patchy in space and time, and are often sampled with traditional methods such as nets. However, these methods are time consuming and provide a snapshot in time and space. Further, sample analysis involves time consuming taxonomic analysis with a microscope. Advanced imagery methods are quickly becoming more commonplace for quantifying plankton. These instruments include, but are not limited to, in situ flow cytobots (IFCB) that image phytoplankton, shadowgraph zooplankton imaging on gliders, towed vehicles (ISIIS, Plankton Scope as examples), autonomous vehicles, and benchtop imagers (Zooscan, Plaktoscope). Each of these methods captures images of plankton across a wide array of sizes- 10 microns for smaller phytoplankton to 13 mm for larger zooplankton.

There is a need for automated image analysis across these various size ranges to drastically reduce the time from image acquisition to data in hand. While plankton image classification pipelines exist, non-data scientist user interfaces are needed along with the ability to use shared image libraries.

Desired functionality:

● Post-processing of plankton imagery across size classes (10 microns - 13 mm).

● Accessible and discoverable labeled image libraries.

● Target level species specific count and size data, including uncertainties and detritus.

● A custom built SparseConvNet for plankton imagery was built that is very good at IDing plankton but is quite heavy, computationally.

● Ideal system strikes a balance between accuracy and computational cost, and is integrated into a platform that can train as well as test in Cloud environments.