Statement of Work.docx

DEPARTMENT OF COMMERCE (DOC), NATIONAL OCEANIC AND ATMOSPHERIC ADMINISTRATION (NOAA), NATIONAL MARINE FISHERIES SERVICE (NMFS)

NORTHEAST FISHERIES SCIENCE CENTER (NEFSC)

Accessing NOAA’s Observer Species Image Library To Automate Species Verification In the Northeast Region

STATEMENT OF WORK

C.1.0 GENERAL

The Department of Commerce (DOC), National Oceanic and Atmospheric Administration (NOAA), National Marine Fisheries Service (NMFS), Northeast Fisheries Science Center (NEFSC) is seeking a contractor to develop software solutions that closely align the image review process currently underway as part of the Species Verification Program (SVP) with machine learning/deep neural network based analysis. Specifically, we seek a contractor to develop software with detailed documentation that applies machine learning (ML) algorithms to identify 45 common fish species and incorporate continuous learning to improve models as new images and verification data become available. The goal of this work is to produce software that can maintain or improve accuracy and increase efficiency of the SVP data processing, and be shared with others working to identify similar species. The Contractor personnel shall be working under the guidance of NMFS NEFSC staff.

The base period of performance will be twelve (12) months and will include project scoping, algorithm development, testing and evaluation, production of ML algorithms, development of a database or software platform to incorporate results into SVP data acquisition and notation, and the creation of accompanying documentation. The project is expected to begin in late spring of 2020 with meetings to coordinate the planning and specification of technical requirements. Details of the software including specific algorithms and output, where and how the algorithms and output will be stored and accessed (e.g., via cloud based or in house NOAAGPU computers), specific requirements for integration with and/or development of a front end user interface will be finalized in the initial scoping period. The scoping period shall entail the development of a project plan as well as a project timeline and governance structure. The Contractor shall be responsible for developing algorithms for identifying species including potentially annotating images with bounding boxes for locating the target specimens. In addition, the Contractor shall create a database platform that would, at minimum, load output into tables matching existing templates (e.g., output converted from soft-max layer probabilities) to facilitate integration with ongoing SVP data processing (Oracle/SQL or compatible). The algorithm(s) shall allow for continuous model improvement/retraining by automatically incorporating new SVP verified images and data. In addition, the Contractor shall submit a draft report for comments and then final report describing the models’ development, developed code, detailed directions for use, and a summary of top-1 accuracy by species based upon 20% of the data withheld from algorithm testing species.

The Contractor will be able to access NOAA FSB’s Species Verification library of images to develop models, produce and test the deep learning algorithm. The development of software solutions that allow application of machine learning models to existing and additional accumulating images and data, and the creation of a database or platform to incorporate output into the SVP data stream shall be primarily the responsibility of the Contractor. However, NMFS will assist and provide feedback on development and comments on incorporating results at regular intervals. Algorithms, data processing code (scripts including those to produce image annotations), and software for this project are encouraged to be made available as open source code with associated documentation published on GitHub. However, images and data are to remain the exclusive property of the U.S. Government. We also require the Contractor to maintain images, ML models and code, and associated software for a period of up to 3 years from the contract start date, but all supplied photos and data must be returned to the NEFSC Fisheries Sampling Branch (FSB) prior to the end of this period. Our intention is to make the algorithms, associated software, and documentation available to the public, and so the Contractor is encourage to produce these in a form that is open source where that can be shared in a public domain, but certain privacy issues still require resolution prior to any publication. The Contractor shall work with NMFS staff to develop software solutions that can be of use to the broader scientific community.

This development aligns with NMFS’s longer-term goal of implementing machine learning in fisheries monitoring to aid in the development of resources with which to enhance the efficiency and accuracy of automated species identification.

C.2.0 BACKGROUND

C.2.1 Overview

DOC/NOAA/NMFS is the Government agency responsible for monitoring and managing Federal fisheries. Within the NEFSC, the Fishery Monitoring and Research Division’s (FMRD) Fisheries Sampling Branch (FSB) is responsible for monitoring and observing fishing in the Northeast and mid-Atlantic U.S. The FSB collects, processes, manages, and provides fishery dependent data and biological samples obtained from fishing vessels via the Northeast Fisheries Observer Program (NEFOP), Industry Funded Scallop Program (IFS), and the At-Sea Monitoring Program (ASM). These programs collectively referenced here as NEFOP, collect, process, and manage data and biological samples obtained during commercial fishing trips from Maine to Cape Hatteras, North Carolina. The branch staff and observers under contract perform at-sea operations aboard commercial vessels to collect information on fishing operations, fishing effort, catch, including bycatch and discard information, economic data and vessel efficiency, and biological samples of landed catch and discard. Collection of these data is mandated or otherwise needed for management under the Magnuson-Stevens Fishery Conservation and Management Act, the Marine Mammal Protection Act, the Atlantic Tuna Convention Act, and the Endangered Species Act.

FSB Observer programs strive to provide accurate and precise near real-time data for fisheries management. An essential part of data collection is the proper identification of species encountered. Reliable species verification is critical for the observer program as the catch data are used for quota monitoring in the Northeast multi-species, herring, scallop, and longfin squid fisheries. In addition, mammal, sea turtle, seabird, sturgeon and shark (species of concern) data are used to evaluate the population status of these animals with respect to regulations and provisions of the acts listed above.

The SVP program has been in existence in some form for over 25 years. Historically NEFOP Observers were issued film/disposable cameras to record records of Incidental Takes and IALs (defined below). In 2008 NEFOP and IFS observers were asked to submit frozen whole samples from a list of species as encountered. In 2010 all NEFOP and IFS Observers and At-Sea Monitors (collectively ‘observers’) were issued digital cameras and required to submit images of all species of concern (incidental take and IAL) and requested to submit images of encountered finfish for verification with their trips.

In January 2011 observers were required to submit images of fish species encountered from a list of priority species (30 species for NEFOP and IFS and subset of 20 species for ASM) each quarter and high volume trips required submission of most herring species encountered. In January 2014, all observers were required to correctly submit 85% of the same 30 species (shown in Appendix 1) as encountered each calendar quarter. In addition, species including Atlantic Herring, American Shad, Hickory Shad, Blueback Herring, and Alewife had to be photographed and both images and whole fish submitted for verification on each high volume trip (defined below). More recently additional species on each high volume trip have been added to the list including Atlantic Menhaden, Atlantic Mackerel, Chub Mackerel, and Atlantic Thread Herring. Required fish or squid species include commercially or ecologically important or regulated species commonly seen in the Northeast fisheries. The intent of these requirements are to ensure accurate identification of these species and to maintain records to regularly evaluate identification accuracy, inform observers promptly of issues, improve training methods, and provide end users information about the accuracy of observer identification.

During initial observer training classes, observers receive instruction on camera use and on how to take the species specific images most useful for identification. These images focus on the unique characteristics that allow differentiation of each species, such as whole body, fins, or gill-rakers of fish. Observers receive a number of resource materials during training including training presentations (PDF), On Deck Reference Guide (hard copy and PDF), Photo Guide to Common Species (PDF), and several commercially available field guides (hard copy).

Goals of the Species Verification Program (SVP) include:

· Evaluate observer identification abilities including compliance with the SVP.

· Quickly provide information about the accuracy of species identification to observers and FSB staff so any data issues can be quickly improved or properly addressed.

· Regularly provide information to training staff about observer ID issues in order to improve training or verification methods.

· Use positions of verified records to audit species records in non-verified observer data.

· Provide support for observers through a SVP program that is simple and fair but allows us to maintain the highest level of data accuracy.

Observers are required to submit these images:

· Every Incidental Take Log (Inc Take) species (mammals, birds, sea turtles caught as bycatch or impacted by vessel or fishing operations).

· Every Individual Animal Log (IAL) species (bycatch of sharks, certain rays, sturgeon, terrapins, pelagic and certain other fish such as tunas, swordfish, ocean sunfish, wreckfish, etc., plus all tagged animals) except where conditions such as safety prevent it.

· Species Log (SPP) Required Species, quarterly (30 species), if encountered.

· On High Volume Fishery trips (7 herring species plus Atlantic and Chub mackerel on every trip, if encountered).

Observers are asked to submit these images:

· Unusual or ‘NK’ (not known) species (submit as images and frozen specimens).

· Species encountered for the first time.

· All encountered fish and invertebrate species (suggest at least every year).

Independent Verification:

· Two verifiers evaluate each submission (can be photo or specimens, but only images referenced here) and assign them to identification categories as ‘correct’, ‘inconclusive but correct based upon observer comments’ (INC Take species only), ‘inconclusive (unclear)’, ‘incorrect’, or ‘no submission but correct based upon observer comments’ (INC Take species only).

· Discrepancies between verifiers are discussed with reference to characteristics listed in keys, guides or other resources in order to come to a consensus.

· If verifiers cannot agree, a third verifier may be consulted which can be an outside expert on the species or group of species.

· Generally, the consensus view is accepted. In cases where a consensus at the species level is not obtained, the submission is recorded as inconclusive with the agreed upon more general taxonomic group entered as the correct identification.

All specimens are normally submitted by observers along with trip data (but may be submitted separately) with strict time deadlines (48 hours after disembarking from the vessel). At present, images are uploaded as ‘BLOB’ files and loaded into Oracle relational database tables. These BLOBs are organized and presented within a web based user interface “Fish House” that provides a user friendly interface for staff viewing and identification. Images[footnoteRef:1] of all non-Incidental Take and non-IAL fish or invertebrate species are typically verified within 2 business days. Incidental Take and some IAL species may take longer (up to 30 days) based upon the number of records received in the period. [1: Process for species not target of this proposal are included for context and many additional species are likely to be included in a future contract so, where possible, efficiencies to incorporate additional species into future ML algorithms and output should be incorporated into the current contract (see Appendix 1 for listing of intended future species).]

At present, Fish House is the web based user interface used for species verification (changes in near future are planned). Fish House is designed to facilitate rapid access to images based upon trip, haul, photo type (Inc Take, IAL, SPP). This interface provides a stepped series of pages to broadly view images by trip, by records that have not yet been verified, by single submissions (may include a series of images and more than one individual), and a single photo (see section C.2.4). Fish house is in part designed to facilitate the verification process that includes an independent first verification by 2 or more reviewers and final resolution of verification with input for additional notes. Submissions are identified by specific codes that are used to identify a single species or group of individuals, and may include additional species codes that are used to identify species that are less commonly encountered or that observers are not required to know. Verifications and associated comments (which may include descriptions of the specimen, comments about images including quality, issues that influenced verification, suggestions for future submissions of that species, etc.) are included in order to provide observers results of their species identification and information about how to improve future photo submissions.

Since January 1, 2010, the number of records (total number of images) submitted to the NEFOP in the following categories were: SPP records 106,237 (202,996); IAL records 20,341 (59,372); Inc Take 6,116 (47,262) (as of 12/19/2019). In 2018 (last full year of submissions) we received 11,482 SPP records based upon 22,701 images, 2,048 IAL species records based upon 5,438 images, and 379 Incidental Take records based upon 3,441 images. A significantly smaller proportion of these images may be suitable for species level machine learning identification (see Appendix 1 for number of submissions by species required for this project and others commonly submitted). However, given the volume of images within the SVP library and the database of specimen identifications this regional resource is unmatched.

An initial trial to develop algorithms from a subset of images with generic deep learning models to classify 30 common species resulted in correct assignment of 96% of records (Figure 1). This simplified trial suggests that accurate identification of a variety of marine species is achievable.

Figure 1: A confusion matrix from preliminary experiments with a deep-learning model. Darker colors along the diagonal indicate better performance. Model accuracy was ~96% for ~30 species. Training data set was ~15,000 images and test data set was ~15,000 images. Categories other than skates were assigned correctly near 100% of the time.

C.2.2 Priority Species for SVP - SPP, IAL, Inc Take Images of an individual specimen are typically linked to a trip, haul, subject code (submission category based upon log where species was historically recorded, either Inc Take, IAL, or SVP), and also linked by sequence number for IAL specimens or protected species ID number (PSID) for Inc Take specimens.

At present, slightly different rules apply for positive ID if SPP versus IAL or Inc Take species. For SPP images, it must be clear which individual in the images is the subject (e.g., clear target subject) and specific identification characteristics be visible in the images in order to be verified as correct. Typically, when no obvious target individual can be identified in an SPP photo then no ‘credit’ is given (e.g., ID unclear) but, for IAL or Inc Take species, if the specified animal can be identified in photo, then it is generally verified as correct. Within a submission, multiple specimens of the same species and/or other species may occasionally occur in the frame or series of images. These may be separated into individual specimen records post submission (e.g., using visual clues to verify same individual such as scrapes or tattered fins). In addition, occasionally a series of images that are submitted as 1 or more records may be combined or separated as in case when the haul number is not correct on one or more images (e.g., based time stamp, inconsistent labeling, or debriefer or observer requests).

Observers operate at sea under highly variable lighting and sea conditions possibly with little photographic experience and are required to submit specific images for each species (but not limited to these). As such, many issues with images may occur that present challenges to machine learning algorithms. A partial list of common issues include:

· Each single animal submission may include multiple images.

· More than one of same and/or different animals may be in a photo or photo series.

· For certain species, observers are required to take specific images to include whole animal and parts like gill rakers, fins, mouth characteristics, damage, etc.

· Whole body images are often not ‘whole’ and not necessarily oriented similarly. Sometimes the only images may be of part of a body part like the head, tail only, or possibly damage parts.

· Multiple individuals of a species or multiple species may be in a single photo, so a machine-learning algorithm should identify the location of the subject under review (e.g., Grad-CAM, Grad-CAM++ and associated technology).

· Subject may be damaged (e.g., missing parts due to bite wounds) or degraded (e.g., due to long net soak times).

· Some species require size-reference to ascertain maturity status (e.g., Little versus Winter Skates). Many different objects may serve as size references. In addition, size estimates may be skewed by perspective distortion: subjects and size references may not be similarly orientated or be a similar distance from the camera lens (focal point).

· Images may be blurry due to camera shake or poorly resolved due to light conditions.

· Low-resolution: may occur due to observers not setting camera correctly or trying to import images from non-approved sources, or due to distance of subject from camera.

· Prior to 2018, the filenames of some images stored outside of the database may be incorrectly labeled. These records, in almost all cases, can be corrected by reference to SVP data in the Oracle Species Verification tables.

As discussed above, verifiers also provide comments that are available to observer via a web based viewer (FSB Web Portal) that is intended to notify observers of issues with images (often even when verification is correct) to improve future submissions. These comments are also available to debriefers and used as basis of In-House Debriefs or other FSB reviews. These include comments like “please include photo of ‘X’”, “view of ‘X’ required for ID”, “please submit specimen when next encountered”, “photo blurry”, “please include size reference in photo”, “please show close-up of vent area”, “please include whole animal in photo”, etc.

C.2.3 Image Library - Camera Specifications, All Sources Beginning in late 2008 observers were issued Olympus Toughbook 8000 digital cameras which were capable of 12 megapixel (MP) but observers were asked to set to 3 MP to reduce File Transfer Protocol times and digital image management and storage. Cameras transitioned to Nikon S32 (13 MP but set to 4 MP). Later camera model also included Nikon S33, which was issued as a fix/upgrade, but all settings and photo specifications were identical. Some observers were issued Panasonic Toughpads (many models, electronic devices for data upload but also typically included a 5 MP rear camera). It should be noted that, although there have been relatively few digital camera models issued to observers, a wide number of camera models have been used (Table 1) and so images may occur in a large number of pixel dimensions but all images have been converted to or are in original .JPG or .JPEG formats.

Table 1. Manufacturer, model and frame dimensions of images that serve as source records for species verifications. The typical pixel dimension of images for that maker/model is shown.

*Commonly occurring photo source.

C.2.4 SVP Photo Library, Oracle tables, Record Review

Data and images for SVP records are currently stored and organized in Oracle relational database tables. Images are stored as ‘BLOB’ files in a media table and identified by unique random ‘checksum’ value. Each specimen or record submission is identified by a unique ‘IDNUM’ (e.g., value same for all images or videos submitted for a single record). Other types of images are also stored in this table (see Table 2 for a list of fields in the media table used to store images).

Table 2. Description of NEFOP_IMS.MEDIA table where recent images (about 2017 to present) are stored.

For record verification, BLOB files are associated with checksum values and made available on an FSB server with images accessible via a web interface called ‘The Fish House’. Verification Staff can access and view all recent submitted records (2018 to present) by navigating individual html pages. Individual pages list species not yet verified (SVP Records Needing Verification, Fig. 2) which allows staff to view images by trip (View Uploaded Images, Fig. 3), by individual record to verify (Verify Species, Fig. 4), by verification records that need resolution (Resolve Species Verification, Fig. 5), or by individual images (Fig. 6).

A goal of this project is to produce software and model algorithms that will efficiently support confident identification of certain usually common species by using existing library of previously identified photo records to initially train models and produce algorithms that will permit improvement based upon future submissions. Species proposed to be included for identification by ML algorithm are listed in Appendix 1. Ideally, the model output should provide results in a tabular format that can be incorporated into an Oracle database table) to seamlessly integrate results into the ongoing SVP data stream. For example, if an algorithm provides a list of identifications and associated probabilities (e.g., a probability matrix), filtering and including only species with a probability above some specified value or listing the most probable species could provide one of two initial reviews. This might require manual acceptance until the algorithm shows a level of consistent and documented valid identifications.

Assuming some or most species can be reliably identified this might allow partial or full automation for these identifications. For example, for species consistently identified by the ML algorithms, it might be possible to set up an efficient viewer to scroll through all recent records of that species (with images for each submission as icons in a row) and approve or audit batches of records. Additionally, this step might be where comments or descriptions could be added to a record (see above). Any ML identified records that are not consistent with the observer submitted species should be marked to go back to human reviewers and inconsistencies automatically resubmitted to the algorithm for retraining. Similarly, if a historic identification record is changed, there should be process to re-submit/retrain the ML algorithm with the newly modified record.

Figure 2. Screen capture of FSB ‘The Fish House’ web interface for ‘SVP Records Needing Verification’. This page shows records not yet verified.

Figure 3. Screen capture of FSB ‘The Fish House’ web interface for ‘View Uploaded Photos’. This page shows images listed by trip and include the variables haul, subject, ID correct?, ID (of specimen), ID comments, and ID’d by.

Figure 4. Screen capture of FSB ‘The Fish House’ web interface for ‘Verify Species’. This page shows an individual record and to provide an input for the initial identifications for each record. Two reviewers provide an independent identification for each record submitted.

Figure 5. Screen capture of FSB ‘The Fish House’ web interface for ‘Resolve Species Verification’. This page shows initial identifications for each record and allows entering final resolution. In cases where initial identifications from reviewers are not the same reviewers discuss and agree on the final ‘resolved’ identification.

Figure 6. Screen capture of FSB ‘The Fish House’ web interface for individual images. Clicking on an icon of a photo (when visible) from any screen will provide an expanded view of the individual photo.

C.2.5 Examples of open source software related to ML image analysis for species identification

This project seeks to work with a company that specializes in computer vision ML to help develop tools that leverage an extensive library of mammal, bird, turtle, fish and invertebrate images collected by observers to begin to automate some aspects of FSB’s labor intensive image review. By collaborating with a company that specializes in this technology, we hope to begin integration of these technologies into the existing work stream to make data acquisition more efficient.

Computer vision as a technique has the potential to be extremely useful in enhancing accuracy and timeliness, while reducing the cost of fisheries data collection. This technology has shown promise in other regions[footnoteRef:2] and there is a high likelihood that it could be successful in this region as well. To achieve these long-term goals though we need continued investment to develop data sets and proof of concept examples. These types of examples can help to further develop techniques and infrastructure to accept, house, analyze and integrate this data into the species verification process and to allow more efficient capture of image data. [2: Richards BL, Beijbom O, Campbell MD, Clarke ME, Cutter G, Dawkins M, Edgington D, Hart DR, Hill MC, Hoogs A, Kriegman D, Moreland EE, Oliver TA, Michaels WL, Piacentino M, Rollo AK, Thompson C, Wallace F, Williams ID, Williams K. 2019. Automated analysis of underwater imagery: Accomplishments, products, and vision. U.S. Dept. of Commerce, NOAA Technical Memorandum NOAA-TM-NMFS-PIFSC-83. 59 p. https://doi.org/10.25923/0cwf-4714.

Automated image classification using machine learning and other algorithms are being developed for use in a broad set of fisheries applications[footnoteRef:3][footnoteRef:4][footnoteRef:5][footnoteRef:6]. These types of algorithms have been the focus of many recent advances and have been the focus of many machine learning competitions (e.g., ImageNet Challenge Competition[footnoteRef:7]). In the northeast region a number of stakeholders have a great interest in developing these techniques, including fishers who are seeking to decrease the cost of catch monitoring, as well as non-governmental organizations hoping to improve catch accounting. Moreover, scientists and managers see these techniques as an opportunity to streamline and enhance survey and monitoring methods. [3: Matai, J., Kastner, R., Cutter Jr, G.R. and Demer, D.A., 2010, September. Automated techniques for detection and recognition of fishes using computer vision algorithms. In NOAA Technical Memorandum NMFS-F/SPO-121, Report of the National Marine Fisheries Service Automated Image Processing Workshop, Williams K., Rooper C., Harms J., Eds., Seattle, Washington (September 4–7 2010).

] [4: National Research Council, 2015. Robust Methods for the Analysis of Images and Videos for Fisheries Stock Assessment: Summary of a Workshop. National Academies Press.

] [5: Allken, V., Handegard, N.O., Rosen, S., Schreyeck, T., Mahiout, T., Malde, K. and Handling editor: Richard O’Driscoll, 2018. Fish species identification using a convolutional neural network trained on synthetic data. ICES Journal of Marine Science, 76(1), pp.342-349.] [6: Malde, K., Handegard, N.O., Eikvil, L. and Salberg, A.B., 2019. Machine intelligence and the data-driven future of marine science. ICES Journal of Marine Science.

] [7: Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M. and Berg, A.C., 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3), pp.211-252.

Pursuing work that focuses on forms of automation, machine learning, computer vision, and artificial intelligence is a both regional and national priority. Nationally, Executive Order 13859[footnoteRef:8] sets out a number of ambitious goals for developing products and training staff. Products from this project align with these priorities and could have both direct and indirect positive impacts on similar work in other regions. Direct positive impacts could include the use of image library and algorithms developed for this project by adjacent regions with similar species complements (or in other applications). Indirect positive impacts of this project include but are not limited to the technical advances and insights gained through this project, which could serve as a valuable resource for other regions. [8: https://www.federalregister.gov/documents/2019/02/14/2019-02544/maintaining-american-leadership-in-artificial-intelligence]

There are a growing number of open source software options for machine learning image analysis. These include software for recording and reviewing of EM video and sensor data. Perhaps the most notable example comes from the combined efforts of Chordata LLC, Saltwater Inc, and Sea State Inc. (see: https://pt.chrdta.com/em/). Open EM is another similar EM review software that is open source (see: https://github.com/openem-team/openem). These tool sets include software solutions that allow for the full lifecycle of EM data collection (from shipboard data acquisition to a review user interface). Additionally, open source software for image and video annotation is also becoming more common. These are generally distinct software packages that aim to generate data sets on single video streams (e.g., https://github.com/cvisionai/Tator, https://github.com/opencv/cvat).

The aim of this project is to incorporate the functionalities into these types of software and machine learning technologies into the current work and data stream within the SVP in a straightforward manner. The ideal product of this work would be software that could ‘plug-in’ with existing SVP review software.

C.2.6 Machine Learning and Image Analysis in the Northeast to Date

There are a growing number of groups interested in applying a machine learning approach to automate species identification in images related to fisheries data acquisition. Much of the innovation is being driven by collaborations between NOAA staff and researchers from the University of Washington. In New England there have been a several projects that have tried this using data collected as part of regional EM development. These include, but are not limited to: a recent Kaggle competition organized by TNC[footnoteRef:9]; second Kaggle competition aimed at identifying individual right whales organized by the NEFSC[footnoteRef:10]. Additional projects underway utilize data currently being collected as a part of EM pre-implementation monitoring (collaborations between TNC, UMass Dartmouth's School for Marine Science & Technology (SMAST), and a private company CVision Consulting). [9: Please see the Kaggle winners: http://drivendata.co/blog/fish-winners/ and general competition description: https://www.drivendata.org/competitions/48/identify-fish-challenge/ ] [10: Please see this right whale competition: https://www.kaggle.com/c/noaa-right-whale-recognition

Currently, GARFO and NEFSC are building the database infrastructure and processing tools for data collected from EM video footage, conducting comparative analyses to augment the existing catch monitoring systems in the fishery, and addressing the ongoing legal and logistical hurdles inherent in developing and integrating new data streams. These efforts aim to implement EM as a means to both enhance the accuracy and efficiency of fisheries independent data collection within the region.

In addition, the NEFSC has developed a large image library for species encountered in the northeast by using fixed cameras aboard the R.V. Bigelow. This work capitalized on specimens processed by the bottom trawl survey as well as the large number of annotations currently entered into FSCS. The large quantities of sample images are an essential part of training machine learning algorithms, and thus are an essential ingredient in developing the ability to automatically classify images. The image collection for this project has recently finished and this library is available as a resource for model development for this project.

C.3.0 OBJECTIVES

NEFSC is proposing to develop algorithms and software solutions to help ease the burden of the Northeast Fisheries Observer Programs Species Verification review. The machine learning components will utilize the existing image photo image library together with the identifications listed in the Species Verification database, which provides a listing of the correct identification for each record. The software solutions produced by this work are encouraged to be open source.

Through this project (as well as other ongoing regional projects), NMFS intends to contribute to the following strategic goals:

· Enhance the efficiency and accuracy of species identification within the FSB’s Species Verification Program

· Integrate results as seamlessly as possible into the work flow of the existing program.

· To the extent practicable, allow flexibility into the training and output results to allow retraining the algorithm and to allow thresholds for incorporating results to be set by FSB staff.

· Provide a process to incorporate comments intended to further describe submissions (records), include information about the identification process, or suggest steps that can be taken by the observers to improve future submissions.

· Reduce the overall cost and time required for documenting and validating species verification records in the FSB.

· Provide a resource for managers within the regionthat may facilitate their involvement in similar projects.

· Potentially provide a mechanism to apply SVP ML process to EM programs for the verification of species identified through EM.

This software development project aims to develop a software and algorithm solutions for identification of species typically encountered by commercial fishers within the Northeast U.S. Federal waters. The goal is to leverage existing images and data sets for algorithm development, and develop an infrastructure that can effectively integrate output with existing workflow in a manner that allows automatic model improvement. An important measure of the success of this project is whether the algorithms and data platform can support accurate identification while reducing the time required by FSB staff for SVP image verification review. Assuming it can, the next measure is can it be incorporated in a straightforward manner into similar projects (such as ongoing Electronic Monitoring). To achieve these objectives, NMFS has identified the following Contractor specific project deliverables:

· Through an iterative process, must develop a plan for training ML algorithms to identify 45 priority species (Appendix 1) with the existing photo library of identified specimens. We intend to adapt any model developed to a greater number of species at a later point (expected additional species also listed in Appendix 1) and constructed algorithms should allow for future additions.

· Through an iterative process, must develop a plan for generating output from machine learning algorithms in a tabular format that permits interpretation (e.g., probability matrix). This must include records of image annotations in a form that can be recreated (e.g., point positions of bounding boxes that that provide position of identified individuals). We also encourage the use of technology that provides visual explanation of features that supported identification (e.g., gradient based Grad-CAM or similar technology[footnoteRef:11]). [11: Slvaraju et al. 2019. Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization. arxiv.org/abs/1610.02391v4]

· Through discussions with FSB staff, must develop a straightforward process to incorporate output from algorithms into the existing SVP identification, notation, and storage process.

· Through an iterative process, shall develop a plan for automating including new images from final verifications into the ML training process including or purposefully excluding some images (e.g., anomalous specimens) from the training library.

· Must outline a process whereby ML algorithms can be applied to other image libraries to minimize the process required for others to implement these products into their similar species identification ML projects.

· Must provide models/algorithms on site to FSB staff (encouraged to be published as open source) and detail in a final report instruction on its intended usage, accuracy of identification, discussion of strengths and weaknesses (where data are available), clearly document requirements for monitoring and maintenance of the model, and other important aspects of its use. This report should include specific guidance for others that may be interested in utilizing similar approaches to solving species identification or other similar natural resource projects.

Overall, the project evaluation will be based upon:

· Consistent and reliable identification of 45 priority species as defined in Appendix 1.

· Efficiency/time savings related to incorporation ML results into the SVP review. Time saved will be estimated/interpreted as function of the time required to operate and maintain and ML software/algorithm compared to the average staff time required for existing verification process (e.g., time for independent verification by reviewers and resolution thereof not including comments to observers), versus reduction in staff (e.g., reviewer time) and resources required when ML is incorporated into review process.

· Software transparency and the ease (flexibility) with which this software can be adapted to additional species (see Appendix 1) or other applications within the region.

· Effective training of FSB SVP and IT staff in use of ML software.

· Responsiveness to FSB staff guidance, questions, or issues.

· Comprehensiveness and clarity of the final report.

C.4.0 SERVICE REQUIREMENTS

C.4.1 Project Phases

C.4.1.1 Project Setup: a period of up to two (2) months (dependent on the date of the contract award) involving consultation and development of algorithms and software plan requirements described in Sections 4.2 (Project Initiation). Additional tasks related to establishing project roles and pre-emptive work needed to complete sections 4.3 (Algorithm Development), 4.4 (Developing Machine Learning to automate Species Verification in the NEFOP), 4.5 (Using SVP Library to Develop Machine Learning Training Data), and 4.6 (Final Report and Other Documentation).

C.4.1.2 An eight (8) month algorithm and software development period (initiated approximately three (3) months post contract award) involving the development of algorithms in Sections 4.3 (Algorithm Development), 4.4 (Developing Machine Learning to automate Species Verification in the NEFOP), and 4.5 (Using SVP Library to Develop Machine Learning Training Data).

C.4.1.3 Analysis and reporting: two (2) month period (initiated ten (10) months post contract award) involving the service requirements described in Section 4.6 (Final Report and Other Documentation).

C.4.2 Project Initiation

C.4.2.1 The Contractor shall participate in regular contact with appropriate NMFS staff during algorithm and software planning phases. Frequent and consistent communication with NMFS staff will help build a good working relationship, reinforcement commitment (NMFS staff and Contractor), and will serve to address any outstanding questions or concerns either party may have. The Contractor and NMFS shall also use these opportunities to describe project goals, evaluate the progress towards project objectives since the last meeting, and provide feedback on the state of development. Contacts will be conducted in a manner that ensures that clear software requirements are developed and to ensures that the ultimate goal of this project are clear to the Contractor. The Contractor will track all contacts and document the purpose of each event in a project progress tracking sheet which will be accessible to NMFS.

C.4.2.2 The Contractor shall also participate in occasional coordination meetings with NMFS staff and other stakeholders throughout the course of the contract. Meetings will be led, organized, and scheduled by NMFS. Outreach may include software demos and detailed discussions about algorithm development.

C.4.2.3 The Contractor shall engage in regular communication (every week or more frequently as required) with NMFS staff (in person or by conference call) to review development objectives, software requirements, development plans, project deliverables, project spending and budget plans (quarterly), and staffing/resources.

C.4.2.4 The Contractor shall lead the planning phases of algorithm and software development (with the assistance of NMFS). Specifically, the Contractor shall perform an iterative process of coordination with NMFS staff to ensure that the algorithms and software being developed will have the greatest utility feasible given the existing constraints (data, funding, and time available). Algorithms must be developed and implemented such that they complement and are easily integrated into the ongoing verification data produced by SVP staff. The Contractor shall provide a detailed description of their algorithm and software development plan and its components in their quote and as part of their reference sheet. The Contractor must also include a detailed description of how the existing image data sets and software will be used to train and update software. Additionally, the Contractor shall lay layout the general plan for software development that specifies key attributes of the software including where the software will run (in the cloud or locally) and how the software will interface with existing SVP data and the ongoing SVP program. Finally, the Contractor should explicitly outline how the accuracy of algorithms will be assessed by users in applications where they are applied.

C.4.3 Access SVP Library[footnoteRef:12] for Algorithm Development [12: Bigelow image library is also available as a resource and incorporation/use of these images to improve the ML algorithms produced by this project is encouraged.]

The Contractor shall develop algorithms that meet the goals identified through consultation with NMFS during the project initiation (4.2). Examples of tasks that are to be required include (but are not limited to): 1) identification of required species (listed in Appendix 1) including identifying images within each submission appropriate for ML, 2) identify position of identified individual using bounding boxes and database coordinates), 3) estimate length of certain species (e.g., in order to determine maturity status). Other tasks that that are encouraged are to use methods such as Grad-CAM technology or similar to identify important regions in images important for identification. The Contractor is encouraged to use open source software development protocols including modern version control. The algorithms will be made such that they can be updated as simply as possible as new data becomes available and should incorporate efficiencies for the intended goal of adding additional species in future (see Appendix 1). Utilization of modern machine learning techniques are essential for this task.

C.4.4 Develop Machine Learning Training Data and Model

Following the development of machine learning algorithms, the Contractor shall develop mechanisms to import results from these algorithms into the ongoing SVP review process. This process will follow the plan developed in coordination with NMFS. The Contractor shall work to the greatest extent practicable to make a tool that is effective in identifying species and efficiently integrates into the SVP identification process. This is to include a software framework that allows access at multiple levels and that is clearly and precisely documented.

The Contractor shall create and maintain a detailed plan for progressing through the stages of development for this software. This includes developing the required project management framework and clearly establishing team roles. This information shall be made available to NMFS on request NMFS. As soon as possible, the Contractor shall provide NMFS access to software to allow for feedback and experts within FSB and FSB’s SVP to provide feedback. Clear documentation of these interactions and software requests shall be maintained even if feature requests fall outside the scope of this project. The Contractor must provide instruction on the use of the software product to NMFS staff. This is to include instructions for basic software installation and its use and include detailed descriptions of software features. The Contractor will assist NMFS in integrating the software and results into the SVP’s ongoing review process. A complete inventory of all code libraries, third party software tools, algorithms, models, scripts, etc., version histories and licenses required to run the ML algorithms and data delivery platform should be regularly maintained by the Contractor and a complete copy of this inventory must be provided to NMFS at the end of the contract period. This should also include all necessary details of software requirements necessary for future SVP ML contracts.

C.4.5 Integrating Machine Learning output to automate Species Verification in the NEFOP

The Contractor will work to leverage existing image annotation tools to create a novel software tool that allows for the ongoing annotation of training data during the course of regular SVP review. This software may be stand alone or integrated with other tools developed during the course of this project.