Statement_of_Work_Aerial_Survey_(1)_1.pdf

PERFORMANCE-BASED WORK STATEMENT

I. GENERAL INFORMATION

A. Introduction Observer-based aerial surveys have been an important tool for evaluating coastal distributions and estimating population abundances to understand sea duck responses to their changing environment. However, safety, expense, observer bias and lack of methodological consistency are rising concerns associated with observer-based surveys, making it imperative to transition to more sustainable methods. Digital aerial surveys that automate counts from aerial imagery using convolutional neural network (CNN) models are one way to improve survey safety and count accuracy. Here we propose to develop a standardized digital aerial survey for the lower Pacific Flyway to help maximize safety, while improving data consistency and model accuracy among important regions within the Flyway.

B. Background

Determining the distribution of wintering sea ducks is a fundamental need for evaluating population abundances and determining how sea ducks respond to threats such as coastal development, vessel traffic, and climate change.

The Pacific Flyway is a vital region for sea ducks that has proven challenging to survey due to logistics, cost, and safety. The SDJV Continental Technical Team (2018), recognized both the importance and limitations of a Pacific Flyway survey and recommended prioritizing areas in relation to habitat threats, with more emphasis on southern regions including British Columbia, Washington, Oregon, and California, where coastal development is increasing.

Due to the challenges associated with low altitude aerial surveys, including a dwindling number of qualified pilots and observers, surveys at these Sea Duck Key Habitat Sites are in jeopardy and it is imperative to transition to more sustainable methods. Furthermore, this need to transition creates an opportunity to take a coordinated approach and develop sustainable, regionwide methodology that will allow comparable results and the capacity to expand to important outer coastal regions of the Flyway that support large concentrations of wintering sea ducks. Developing digital aerial surveys that automate counts from aerial imagery is one way to improve survey safety and count accuracy.

Important efforts (e.g. Prebyl and Knoedler 2022; Fara et al. 2022) are underway to build imagery libraries, estimate classification precision and recall, and train convolutional neural network (CNN) machine learning models (Hong et al. 2019) to identify sea ducks in aerial images.

However, challenges, including the difficulty in obtaining a sufficient number of annotated images for model training, variable image resolution among sites and projects, and environmental differences that affect imagery across wide-ranging study areas, remain impediments to transferability among sea duck survey efforts (Weinstein et al. 2022; Miao et al. 2023; Shandilya et al. 2023). To date, these problems have prevented the development of a CNN model that can be used to automate image-based sea duck survey results with regional consistency.

C. Definitions/Applicable Documents Not applicable.

D. Scope

General Purpose: The long-term, overarching goal of our proposed project is to develop standardized, imagery-based digital aerial survey (DAS) methodology to help resolve safety issues, while improving data consistency and model accuracy among important regions within the Pacific Flyway. To accomplish this goal, we will build on methods developed during the ongoing USGS Pacific Coast Seabird Survey (Adams et al. 2019). This includes retraining a CNN model developed for seabirds to classify sea ducks to the lowest taxonomic level.

Vendor objective 1: Identify Bird Presence: In years 1 and 2, the vendor will process images collected by USGS and collaborators (Washington Department of Fish and Wildlife WDFW, Environment Canada) using a pre-existing seabird detection CNN model developed in YOLO (Redmon et al. 2016) through collaboration between USGS WERC (Adams) and Conservation Metrics, Inc. (https://conservationmetrics.com). The vendor will identify images that contain birds and place bounding boxes around individual birds and return these images to USGS. Staff from USGS and WDFW will then label bounding boxes to the lowest identifiable taxa using image annotation software (e.g., CVAT, LabelStudio). During the annotation process, staff will also assess the imagery for false positive and false negative bounding box results and will remove/add bounding boxes as appropriate so that all birds are appropriately labeled in each image. Annotated images will be used to develop a sea duck library and will be stored in a format appropriate for YOLO CNN detection models.

Vendor Objective 2: Modify (retrain) existing sea bird model to identify sea ducks. Annotated sea duck images acquired in years 1 and 2 of the project will then be provided to the vendor to retrain the sea bird model to create a sea duck-specific CNN model. Retraining will be iterated in years 2 and 3 using new sea duck imagery collected to refine the new sea duck model. In year 4 we will release the annotated image library to the public through a joint USGS-WDFW data release. This comprehensive image library can be used to train future models, as well as be used in domain adaptation exercises to modify models trained at one study site to recognize sea ducks from images taken at another study site, ultimately aiding in the development of a universal CNN model that can be used to automate image-based sea duck surveys across multiple flyways.

Vendor Objective 3: Evaluate and compare model performance.

The vendor will train several models on the revised training dataset, then work with USGS to help evaluate and compare model performance of the full model dataset in order to conduct detailed evaluation of the best model. It is not possible to customize existing preliminary models specific to data in this project until the training dataset is complete. Furthermore, understanding model performance requires a complete test dataset for evaluation. The workflows of this project are additive, which is compounded by additional data and survey delays. Specifics about the range of conditions encountered during surveys, which are summarized and incorporated as image quality metrics, must be incorporated into training data to adequately train models to identify objects across the range of ocean and estuarine surface backgrounds imaged during aerial surveys. These factors have the potential to impact model performance for both object detection and classification. Therefore, additional evaluation is necessary to compare final models, and fully evaluate the best model. Only through more robust evaluation will we be able understand the output performance, used to create estimates of abundance, density and distribution for trust sea duck species.

The final stage of our project will implement the new sea duck model and quantify sea duck abundances in Puget Sound, San Francisco Bay, and at one or more selected coastal nearshore locations.

II. WORK REQUIREMENTS

A. Technical Requirements.

To build upon previous efforts and existing expertise, Conservation Metrics Inc, has developed seabird models that will need to be modified and trained to incorporate sea ducks.

Vendor will need to provide a scaled-up prediction process to match the appropriate computer infrastructure to run the complete image dataset through a final detection-classification model.

Because the high-resolution images collected by USGS are so large, these must be broken into smaller images called “tiles” for ingestion into the machine learning pipeline. This equates to 90 tiles per original image, or ~86 million jpgs. Running many millions of images through prediction algorithms is time and computer intensive and requires a substantial reworking of existing CMI code to fit appropriate infrastructure to complete analyses in a reasonable timeframe (days vs.

months).

B. Deliverables.

1. The vendor will receive tens of thousands of digital aerial images from USGS and project collaborators WDFW annually. Images will be processed through the vendor’s existing seabird CNN model to identify those that contain birds and bounding boxes will be placed around individual birds. All images with bounding boxes around birds will be returned to USGS for evaluation of false positives and negatives, expert identification to lowest taxa and annotation in appropriate software.

2.Vendor will use annotated image library of sea ducks in a wide range of environmental conditions (created in #1) to retrain the existing seabird model to identify sea ducks.

3. Vendor will provide the final model workflow, code, and model evaluation metrics to identify sea ducks from digital aerial images.

Data will be reviewed by the USGS Point of Contact within 60 days of receipt of the data.

Remaining sample material will be archived in USGS facilities. Please notify the USGS Point of Contact of any remaining sample material. Remaining sample material may be disposed of only after USGS Point of Contact confirms they are not needed for other analyses and after data review shows no further analyses are needed.

III. SUPPORTING INFORMATION

A. Place of Performance.

Work will be conducted in an external independent lab facility. The Vendor has the appropriate laboratory space, equipment, personnel, and expertise to complete and deliver the laboratory analyses.

B. Period of Performance

09/16/2024 - 12/31/2027 (estimated).

C. Government Furnished Property (GFP): list property and its value.

None.

D. Special Considerations.

USGS will retain all rights to the digital aerial images collected, image library, and the sea duck model produced. We are requesting a sole source agreement with Conservation Metrics, Inc.