RFQ 1305M221QNFFM0003.pdf

U.S. Department of Commerce

National Oceanic and Atmospheric Administration Eastern Acquisition Division 200 Granby Street, Suite 815

Norfolk, VA 23510

REQUEST FOR QUOTATION

Building a Groundfish Image Library, NOAA Ship Henry B. Bigelow

1305M2-21-Q-NFFM-0003

October 2020

1305M221QNFFM0003 Page 2 of 66

NOTICE DATE: October 7, 2020

Response Due: Monday, November 2, 2020 at 12:00 PM EST

This is a combined synopsis/solicitation for commercial items prepared in accordance with the format at FAR Subpart 12.6, as supplemented with additional information included in this notice, and is being conducted pursuant to FAR Part 12- Acquisition of Commercial Items and Subpart 13.5-Simplified Procedures for Certain Commercial Items. The synopsis/solicitation constitutes a RFQ and incorporates provisions and clauses in effect through Federal Acquisition Circular No. 2020-09 dated 07/02/20. Should the Government issue an order resulting from this RFQ, it will be in the form of a written offer to the supplier and shall require the supplier’s acceptance by notification to the Government, in writing, as defined at FAR 2.101. THIS ANNOUNCEMENT

CONSTITUTES THE ONLY SOLICITATION; QUOTES ARE BEING REQUESTED AND A

WRITTEN SOLICITATION WILL NOT BE ISSUED.

The National Oceanic and Atmospheric Administration (NOAA), Eastern Region Acquisition Division, 200 Granby Street, Suite 815, Norfolk, Virginia 23510, on behalf of the National Marine Fisheries Service (NMFS), Northeast Fisheries Science Center (NEFSC) is soliciting quotes for Building a Groundfish Image Library, from NEFSC Survey Collection, NOAA Ship Henry B. Bigelow.

This RFQ is Total Small Business Set-Aside. The associated North American Industry Classification System (NAICS) code is 541511, Custom Computer Programming Services, which has a corresponding size standard of $30M.

The Government anticipates awarding a Firm-Fixed-Price contract with a Base Period of one-year and three (3) one-year option periods, Period of performance from March 1, 2021 through February 29, 2025.

The Contractor shall supply all necessary personnel, supervision, services, materials, travel and other items as required in accordance with this RFQ.

Attachment 1-Pricing Schedule shall be completed by the Contractor for pricing purposes.

1305M221QNFFM0003 Page 3 of 66

DEPARTMENT OF COMMERCE (DOC)

NATIONAL OCEANIC AND ATMOSPHERIC ADMINISTRATION (NOAA)

NATIONAL MARINE FISHERIES SERVICE (NMFS)

NORTHEAST FISHERIES SCIENCE CENTER (NEFSC)

Image Collection, Processing, and Machine Learning to Support Northeast Electronic Monitoring Programs

STATEMENT OF WORK

C.1.0 GENERAL

The Department of Commerce (DOC), National Oceanic and Atmospheric Administration (NOAA), National Marine Fisheries Service (NMFS), Northeast Fisheries Science Center (NEFSC) is seeking a contractor to provide and install an image collection system capable of recording from four stations (three sampling and one belt sorting) on the survey vessel NOAA ship Henry B. Bigelow. The Contractor shall be responsible for the installation of equipment, removal of equipment, equipment maintenance, field support, coordination and meeting with NOAA staff, image review, data analysis, image data set curation, and report writing. The image collection and processing will be the first step in this project. The Contractor shall also develop software solutions that more closely align the video review process currently occurring as part of electronic monitoring (hereafter EM) programs with machine learning based video analysis.

Specifically, we seek a contractor to develop software and documentation in an open manner that both allows for the application of machine learning algorithms to video footage and allows for the further development of annotated training data sets. The goal of this work is to produce tools that can be shared with companies working to process the review footage. The Contractor personnel will be working under the guidance of NMFS staff, and with project partners in the northeast region.

The base period will include project start-up, project scoping, data collection, data analysis, algorithm development, the production of software, and the creation of accompanying documentation, and report writing periods. The project is expected to begin in early 2021 with the coordination and installation of image collection equipment on the NOAA ship Henry B. Bigelow in order to collect high quality images of the fish specimens processed by the bottom trawl survey. In addition to follow-up meetings to coordinate the planning and specification of technical requirements. This will entail the development of a project plan as well as a project timeline and governance structure. The Contractor will be responsible for developing image collection methods which minimize the impact of image data collection on NMFS survey staff and shipboard resources. The Contractor will be responsible for pairing the collected image data with species records from the ship’s data collection system and producing a dataset appropriate for use in machine learning applications. The processing and data annotation of the electronic data will primarily be the responsibility of the Contractor; however, NMFS may assist or review the data as needed. The Contractor will be responsible for developing algorithm based tools for detecting catch items, estimating the length of these items, and performing species identification via machine learning that are suitable for processing video data from fisheries in the northeast region. All data collected will be the property of the Government. The Contractor shall work with NMFS staff to develop an image library that can be of use to the broader scientific community.

The Contractor will be expected to leverage existing image libraries for machine learning algorithm and model development. Additionally, the Contractor will be encouraged to build upon existing open source software tools. The

1305M221QNFFM0003 Page 4 of 66 development of software solutions that both implements machine learning models and allow for the generation of additional training data will primarily be the responsibility of the Contractor; however, NMFS will assist and provide feedback on development at regular intervals. All software, models, and data (including annotations) that are generated by the project will be the property of the Government. The Contractor shall work with NMFS staff to develop software solutions that can be of use to the broader scientific community.

This development aligns with NMFS’s longer term goal of implementing machine learning in fisheries monitoring to aid in the development of resources with which to enhance the efficiency and accuracy of automated species identification.

The information gained through this study will help ultimately enhance EM in New England and Mid-Atlantic by providing the means to make EM video review more efficient and thereby a more viable option for collecting fisheries data.

C.2.0 BACKGROUND

C.2.1 Overview DOC/NOAA/NMFS is the Government agency responsible for monitoring and managing Federal fisheries. Within the NEFSC, the Fishery Monitoring and Research Division’s (FMRD) Fisheries Monitoring and Operations Branch (FMO) is responsible for monitoring and observing fishing in the Northeast and mid-Atlantic U.S. The FMO collects, processes, manages, and provides fishery dependent data and biological samples obtained from fishing vessels.

Collection of these data is mandated or otherwise needed for management under the Magnuson-Stevens Fishery Conservation and Management Act, the Marine Mammal Protection Act, the Atlantic Tuna Convention Act, and the Endangered Species Act. The branch staff and observers under contract perform at-sea operations aboard commercial vessels to collect information on fishing operations, fishing effort, catch, including bycatch and discard information, economic data and vessel efficiency, and biological samples of landed catch and discard. The Greater Atlantic Regional Fisheries Office (GARFO) receives and reviews EM summary data to conduct data analysis and comparisons with other data sources where applicable.

EM is being considered as a tool for catch monitoring and reporting compliance in fisheries around the world. Nationally there are several EM initiatives and programs, and in the northeast EM implementation is scheduled for May 2021. As part of the Greater Atlantic Region’s Electronic Technology (ET) Implementation Plan, the Councils and NMFS have approved EM in the mid-water trawl Atlantic herring and northeast multispecies groundfish fisheries to improve catch monitoring and accountability. This project builds off of the recommendations and action items from the following completed projects and current initiatives:

● National EM Workshop, January 2014, GARFO and the NEFSC, in collaboration with The Nature Conservancy (TNC), hosted a two (2) day workshop in May 2014 for the purpose of developing a shared strategy of implementing EM.

● The NEFSC completed the final phase of its New England Electronic Monitoring Project in 2013 and published the final report in August 2014. This project yielded initial technical specifications and onboard configurations for EM systems, catch handling protocols, methods for weight estimation, and other best practices for a groundfish EM program.

● TNC, the Gulf of Maine Research Institute, the Maine Coast Community Sector, and Ecotrust Canada, have collaborated with GARFO and NEFSC, to operationalize an EM program in the groundfish fishery. The final

1305M221QNFFM0003 Page 5 of 66 project goal is to develop standards, process, and analysis to support the use of EM in lieu of At-Sea Monitors (ASMs) in the New England multispecies fishery and implement an operational program in 2018.

● GARFO, the NEFSC, and Saltwater Inc. worked together to pilot the use of EM on the Atlantic herring and mackerel fleet in the Northeast and Mid-Atlantic. The goal of the study were to: 1) evaluate the utility of EM for monitoring catch retentions and identifying discard events in the Atlantic herring and mackerel MWT fisheries (detection and classification of discard events) and 2) to inform EM cost responsibilities for the fishing industry and NMFS. EM (plus portside sampling) coverage was approved by the NEFMC at its April 2018 meeting as an adequate substitute for at-sea monitors (ASM) aboard midwater trawl (MWT) vessels. Rulemaking and implementation will continue through the remainder of 2018.

● NEFSC and GARFO have partnered in 2016, 2017, 2018, 2019, and 2020 to test the applicability of EM to provide specific monitoring services in the Atlantic herring and mackerel fisheries.

● In 2018, 2019, and 2020 NEFSC and GARFO expanded EM to a true audit model program (100% video and partial review).

● In 2018 and 2019, NEFSC collaborated with CVision, AI and deployed an EM system on the Henry B. Bigelow to build an groundfish image library.

● In 2019 and 2020 NEFSC collaborated with CVision, AI and developed an algorithm to automate species identification and length estimation in support of EM in the region.

● NOAA’s strategic initiative to promote machine learning in support of EM programs with the development of a national image library in 2020.

This study will deploy a camera system on a NOAA research vessel with the goal of producing an image library that can be used to develop tools that will advance catch accounting methods. This project is consistent with the objectives of the Councils to improve monitoring and to address bycatch in the northeast fisheries.

Since 1963 the NEFSC has conducted a scientific bottom trawl survey spanning the Northeast Continental Shelf Large Marine Ecosystem. This expansive design-based survey typically carries out ~370 tows (per season) at random stratified stations throughout the region. Surveys are currently conducted in the spring and fall and typically represent ~120 seadays. The fall survey typically runs from September through mid-November and the spring survey runs from March through mid-May. Each spring and fall survey is divided into legs, between 2-3 weeks duration. Survey legs typically depart from and return to Newport, RI, however, Norfolk, VA and Boston, MA may be used depending on cruise logistics. The NOAA Ship Henry B. Bigelow is the the primary vessel used for these surveys. The research vessel has a total length of 206ft and is staffed by 24 officers and crew and as many as 16 scientific personnel. The gear used for this survey is a standardized, multispecies bottom trawl that features an 89’ rock-hopper sweep and a 69’ headrope. Nominal fishing characteristics are 105’ door spread, 42’ wing spread, and 13’ headrope height. The on-bottom tow duration is standardized to 20 minutes. Total catch weight per tow are typically between 100 and 1000 lbs, however, tows as large as 30,000 lbs are occasionally encountered. The entire catch of each tow is dumped into a hopper (also called a checker pen) and brought inside a wet-lab via conveyor system. The catch is sorted by species along the conveyor and then weighed in at a primary weigh station by the watch chief. Individual species are then passed along a secondary conveyor for biological sampling and processing at one of three workstations. While dependent upon the status of the species examined, data elements collected per individual finfish species typically include: length, weight, sex, and analysis of stomach contents. These data are captured using the Fisheries Scientific Computer System (FSCS) and stored in an Oracle database. For additional information on the research vessel, the FSCS system, and survey protocols please see relevant publications1.

1 PJ Politis, JK Galbraith, P Kostovick, RW Brown. 2014. Northeast Fisheries Science Center bottom trawl survey protocols for the NOAA Ship

1305M221QNFFM0003 Page 6 of 66

The data generated by each survey varies, but for the most commonly encountered species ~10,000 individuals are processed for length measurement per cruise (Figure 1) and closer to 1,000 individuals are processed for additional sampling like weight estimation (Figure 2). For a breakdown of some of the data elements included with the length data, see the InPort entries for the materialized view UNION_FSCS_SVLEN (https://inport.nmfs.noaa.gov/inport/item/33602).

Figure 1. The distributions of length samples taken per survey season (the number of individual specimens processed).

Data comes from cruises 2000 - 2018. Species selected are the most common species sampled in this manner across all surveys. Dashed vertical line represents ten thousand individuals with length estimates. Note the x axis is a log scale.

Henry B. Bigelow. US Dept Commer, Northeast Fish Sci Cent Ref Doc. 14-06; 138 p. Available from: National Marine Fisheries Service, 166 Water Street, Woods Hole, MA 02543-1026, or online at http://nefsc.noaa.gov/publications/ https://inport.nmfs.noaa.gov/inport/item/33602 http://nefsc.noaa.gov/publications/

1305M221QNFFM0003 Page 7 of 66

Figure 2. The distributions of weight samples taken per survey season (the number of individual specimens processed).

Data comes from cruises 2000 - 2018. Species selected are the most common species sampled in this manner across all surveys. Dashed vertical line represents one thousand individuals with weight estimates. Note the x axis is a log scale.

Additional cruise information is available through the InPort system at these locations (https://inport.nmfs.noaa.gov/inport/item/22560 for the fall survey and https://inport.nmfs.noaa.gov/inport/item/22561 for the spring survey). These inPort files represent presentation data models and do not encompass the full scope of data collected at sea. Individual fish data elements that may be relevant to linking images include: location of the vessel, sampling site identifier, sampling location where individual fish parameters are collected, species identification, unit of measure where applicable, and date and time stamp for each parameter collected (please see Appendix 1).

This project seeks to leverage the existing image data sets developed under previous work and open source EM video review software with the goal of producing an integrated software solution that can be used to advance catch accounting methods. This project is consistent with the objectives of the New England Fisheries Management Council to improve monitoring and to address bycatch in the multispecies groundfish fishery.

A summary of how EM works in the groundfish fishery follows: A typical EM system (please see Attachment B) includes three (3) to four (4) cameras as well as other sensors to capture fishing activity (i.e., GPS or hydraulic pressure).

Generally, video is collected from the time a trip starts to the time the vessel returns to the dock. During each trip captains and crew follow highly specified catch handling, and are only allowed to discard fish from specific locations (control points). Camera views and catch handling is laid out in documents called vessel monitoring plans that are approved by NMFS for each vessel to ensure that a reviewer is able to estimate portions of catch that are most relevant to management and monitoring.

All of the data from cameras and sensors are recorded on hard drives that are delivered to an EM service provider. The EM service providers use software to annotate footage and create EM data summaries the outline the fishing activity relevant to managers (https://apps-nefsc.fisheries.noaa.gov/FSBEM/index.php). Video review is conducted in two parts by the current provider. First, an initial review where reviewers inspect the video and sensor data for issues that prevent review (camera failures, malfunctioning sensors, etc.). A more thorough review of the video is then conducted where the https://inport.nmfs.noaa.gov/inport/item/22560 https://inport.nmfs.noaa.gov/inport/item/22561

1305M221QNFFM0003 Page 8 of 66 bycatch is estimated from individual fish (lateral images on a marked board) and non-compliant fish handling behavior is noted. These annotations are turned into a summarized JSON file and submitted to the GARFO for in season catch accounting (https://apps-nefsc.fisheries.noaa.gov/FSBEM/index.php).

Currently EM is used as a monitoring option on ~25 vessels in New England and these vessels use a range of gear types (e.g., trawl, handline, longline) but all have specified handling for catch estimation (see above). The number of trips taken by these vessels has recently increased to ~400 trips per fishing year. Trips generally last one (1) to three (3) days.

Currently EM review is conducted for all of these trips and video review is time consuming. Typically, each review takes between two (2) and ten (10) hours. The time and resources required to annotate video footage is not efficient or effective and currently limit scaling up to fleet-wide participation (which could be as many as 200 vessels and reviewing footage from longer multiday trips).

Speeding up video review with some automation (such as machine learning based video processing) is essential for moving electronic monitoring forward. Automating key aspects of quality control and catch estimation (i.e., detection, identification, and length estimation) with modern machine learning techniques is the goal we are working towards.

Additionally, incorporating modern tools in EM review software that improve video review, but also allow for the continued development of image data sets suitable for furthering machine learning model training. Specifically, because more data is generally better for training machine learning algorithms, leveraging the existing human review to build annotated training data sets would be an ideal use of EM review software. Implementing these features in an open source software package is the key to taking EM to the next level in New England and Nationally

C.2.2 Image Library Development

C.2.2.1 Specifics of fish handling and data acquisition

Fish are processed in a streamlined and efficient manner on the research vessel2 (see also the the wet lab schematic and processing description Appendix 2). Typically, catch moves from a checker pen to sorting where individual species are grouped into baskets. From there, baskets are passed to two person teams (a ‘recorder’ and a ‘cutter’). These teams prep specimens for biological sampling and take the appropriate measurements and samples. Specimens which are only measured (only a length is recorded) are processed quickly often only with a few seconds between each specimen. This rapid processing allow the vessel to collect information from a large number of individuals, but also poses a challenge for those trying to collect images of the specimens. Specifically, the hands of the cutter will often remain on the fish while a length measurement is taken. This is especially true for the most experienced crew that are processing samples quickly. Generally, crew are working at capacity and it is unlikely that major changes (such as delayed sampling protocols or cutters taking their hands off the specimens can be accommodated). For an example of this challenge please see the images on slides 16 and 18 of the noted processing presentation2 (again, also see the attached wet lab description and fish processing document). The goal initially will be to determine if adequate images can be collected without altering catch handling.

C.2.3 Examples of open source software related to EM

There are a growing number of open source software options in the EM sphere. These include software that is designed for recording and reviewing of EM video and sensor data. Perhaps the most notable example comes from the combined

2 Please see this presentation for rough details of onboard fish processing (note some details have changes with the implementation of FSCS 2.0): https://www.nefsc.noaa.gov/Bigelow/pdfs/fish-handling-data.pdf https://www.nefsc.noaa.gov/Bigelow/pdfs/fish-handling-data.pdf

1305M221QNFFM0003 Page 9 of 66 efforts of Chordata LLC, Saltwater Inc, and Sea State Inc (see: https://pt.chrdta.com/em/). Open EM is another similar EM review software that is open source (see: https://github.com/openem-team/openem). These tool sets include software solutions that allow for the full lifecycle of EM data collection (from shipboard data acquisition to a review user interface). Additionally, open source software for image and video annotation is also being produced. These are generally distinct software packages that aim to generate data sets on single video streams (e.g., https://github.com/cvisionai/Tator, https://github.com/opencv/cvat). This project aims to merge the functionalities into these types of software to applying machine learning to EM in a straightforward manner.

The ideal product of this work would be software that could ‘plug-in’ with existing EM review software. The NMFS data model requires that a 3rd party provider is responsible for the initial review of video data, therefore the end product of this contract needs to be a product that could interface with the current software used by EM service providers. These providers generally stream video and allow for review of these data via web portals.

C.2.4 Machine Learning and Image Analysis in the Northeast

There are a growing number of groups interested in applying a machine learning approach to EM nationally. Much of the innovation in this realm is being driven by collaborations between NOAA staff and researchers from the University of Washington. In New England there have been a couple of projects that have tried this using data collected as part of regional EM development. These include, but are not limited to: a recent Kaggle competition organized by the Nature Conservancy (TNC)3; second Kaggle competition aimed at identifying individual right whales organized by the NEFSC4. Additional projects underway utilize data currently being collected as a part of EM pre-implementation monitoring (collaborations between TNC, UMass Dartmouth's School for Marine Science & Technology (SMAST), and a private company CVision Consulting).

In addition to these ongoing projects, a large library of images to train machine learning models was identified as a regional priority and NOAA strategic initiative. Because existing data sets were seen as not sufficient to train models to distinguish the species found in the region, a project was funded to do just this using the NMFS bottom trawl survey.

The NMFS bottom trawl survey provides perhaps the best venue to develop a large data set of images for model training if images and data can be collected without impacting existing sampling practices by pairing images of specimens processed by the regular operations of the NMFS research vessel with the data currently collected by the FSCS. This work capitalized on the large number of specimens processed by the bottom trawl survey as well as the large number of annotations currently entered into FSCS. These large quantities of sample images are an essential part of training machine learning algorithms, and thus are an essential ingredient in developing the ability to automatically classify images. The image collection for this project has recently finished. Therefore, it is expected that the funded proposal will utilize this resource to advance machine learning and EM review.

C.3.0 OBJECTIVES

C.3.1 Image Library

3 Please see the Kaggle winnners: http://drivendata.co/blog/fish-winners/ and general competition description:

https://www.drivendata.org/competitions/48/identify-fish-challenge/ 4 Please see this right whale competition: https://www.kaggle.com/c/noaa-right-whale-recognition https://pt.chrdta.com/em/ https://github.com/openem-team/openem https://github.com/cvisionai/Tator https://github.com/opencv/cvat http://drivendata.co/blog/fish-winners/ https://www.drivendata.org/competitions/48/identify-fish-challenge/ https://www.kaggle.com/c/noaa-right-whale-recognition

1305M221QNFFM0003 Page 10 of 66

NEFSC is proposing to collect images on two (2) sequential bottom trawl surveys (Spring 2021 and then Fall 2021).

The Fall survey typically runs from September through mid-November and the Spring survey runs from March through April. Through this project (as well as other ongoing regional projects), NMFS intends to eventually meet the following strategic goals:

● Enhance the efficiency and accuracy of fisheries data collection

● Reduce the reporting burden on fishermen

● Reduce the overall cost of fisheries monitoring

This research project aims to develop a library of individual fish specimen images with co-registered specimen information. The aim is to collect as much of this data as possible (as many images and paired metadata as possible), while impacting the survey operations as little as possible. To achieve these objectives, NMFS has identified the following Contractor specific project deliverables:

● Through an iterative process develop the means for image data collection with NMFS amenable to NMFS survey staff.

● Through an iterative process develop the means for specimen data collection with NMFS amenable to NMFS survey staff.

● Test camera image collection systems to ensure functioning and image quality standards for machine learning (in collaboration with machine learning experts).

● Develop protocols for camera maintenance and failure (including necessary redundancies to ensure data generation during project period).

● Perform installation of the camera system capable of collecting data at all four locations within the vessel’s wet lab.

● Collect high quality images and associated data for as large of a proportion of the specimens encountered by the trawl survey.

● Pilot a stereo camera over the ship’s conveyor sorting belt and possibly over one of the sampling stations.

● Develop local infrastructure to support data collection between cruise legs.

● Develop a process with which to pair collected image data with specimen data.

● Work with NMFS to package data for image analysis.

● Develop necessary licensing documentation to make images and specimen data widely available.

● Expand upon the current image library with the inclusion of non-groundfish species.

● Develop an image library suitable for inclusion in NOAA’s national image library.

C.3.2 Algorithm Development

NEFSC is proposing to develop algorithms and software solutions to help ease the burden of EM video review. The machine learning components are expected to utilize the existing image data sets and if feasible work with existing open source EM review software (e.g., https://pt.chrdta.com/em/). These software solutions produced by this work are expected to be open source. Through this project (as well as other ongoing regional projects), NMFS intends to eventually meet the following strategic goals:

● Enhance the efficiency and accuracy of fisheries data collection

● Reduce the reporting burden on fishermen

● Reduce the overall cost of fisheries monitoring https://pt.chrdta.com/em/

1305M221QNFFM0003 Page 11 of 66

● Expedite video review

● Expand upon current image library and algorithm development

This software development project aims to develop a novel software and algorithm solutions for EM. The goal is to leverage existing data sets as much as possible for algorithm development, while also highlighting future image training set needs. An additional focus is to utilize open source software to bring together EM video review and machine learning data set annotation. To achieve these objectives, NMFS has identified the following Contractor specific project deliverables:

● Through an iterative process develop a plan for training machine learning algorithms.

● Through an iterative process develop a plan for implementing machine learning algorithms in EM review software.

● Through an iterative process develop a plan for implementing training data collection via EM review software.

● Develop machine learning algorithms that help to automate EM review following the plan developed with

NMFS.

● Develop a means of integrating machine learning algorithms in open source EM software.

● Develop a means of using EM review efforts to develop novel training data sets for further training of machine learning algorithms.

● Document products produced in a final report that allows for further development.

C.4.0 SERVICE REQUIREMENTS

C.4.1 Project Phases C.4.1.1 Image Library Project Setup: a period of up to two (2) months (depending on the date of the contract award and the dates that the bottom-trawl cruise is scheduled) involving the installation requirements described in Sections 4.2 (Project Initiation), 4.3 (Camera and Equipment Installation), 4.4 (Servicing of Equipment Aboard Vessels, and Field and Project Support) and 4.5 (Data Curation and Library Development). The activities provided in Section 4.4 relevant to this phase are specifically related to establishing local capacity and infrastructure to provide and support field services.

The activities in Section 4.5 relevant to this phase include development of a project plan. Other activities include clarifying project roles and responsibilities and formulating the project team.

C.4.1.2 Algorithm Development Project Setup: a period of up to three months (depending on the date of the contract award) involving project scoping, consultation and development of algorithms and software plan requirements described in Sections 4.2 (Project Initiation). Additional tasks related to establishing project roles and pre-emptive work needed to complete sections 4.3 (Algorithm Development), 4.4 (Bringing Machine Learning to EM Review), 4.5 (Using EM Review to Develop Machine Learning Training Data), and 4.6 (Final Report and Other Documentation).

C.4.1.3 Image Library Data Collection Period: (initiated approximately two (2) months post contract award and coordinated with spring and fall surveys) involving the service requirements described in Sections 4.4 (Servicing of Equipment Aboard Vessels, and Field and Project Support), 4.5 (Data Curation and Library Development), and 4.6 (Removal of equipment from the research vessels).

1305M221QNFFM0003 Page 12 of 66

C.4.1.4 Algorithm Development: Algorithm and software development period (initiated approximately three (3) months post survey legs) involving the development of algorithms in Sections 4.3 (Algorithm Development), 4.4 (Bringing Machine Learning to EM Review), and 4.5 (Using EM Review to Develop Machine Learning Training Data).

C.4.1.5 Analysis and Reporting: two (2) month period (initiated twelve (12) months post contract award) involving the service requirements described in Section 4.7 (Final Report and Data Set Finalization).

C.4.2 Project Initiation

C.4.2.1 The Contractor shall participate in regular contact with appropriate NMFS staff during equipment installations, data retrievals, equipment service events, algorithm and software planning phases, or as required. Frequent and consistent communication with NMFS staff will help build a good working relationship, reinforcement commitment (NMFS staff and Contractor), and will serve to address any outstanding questions or concerns the vessel and NMFS survey staff may have. The Contractor and NMFS will also use these opportunities to describe project goals, provide feedback on equipment performance, evaluate the progress towards project objectives since the last meeting, provide feedback on the state of software and algorithm development, and convey study results. Contact will be conducted in this manner to ensure that clear software requirements are developed and to ensure that the ultimate goal of this project are clear to the Contractor. The Contractor will track all contact and document the purpose of each event in a study service tracking sheet, which will be accessible to NMFS.

C.4.2.2 The Contractor shall also participate in occasional study coordination and communication meetings with NMFS staff, EM service providers, and other stakeholders throughout the course of the contract. Meetings will be lead, organized, and scheduled by NMFS. Outreach may include an equipment testing/demonstration components of the image collection system, software demos, sharing algorithm source code, and detailed discussions about algorithm development and improvements. The Contractor may be working directly with EM service providers under the agency’s guidance to provide guidance on best methods to integrate algorithms into video review software.

C.4.2.3 The Contractor shall engage in regular communication (every week or more frequently as required) with NMFS staff (in person or by conference call) to review monitoring objectives, data requirements, project plans, project deliverables, project spending and budget plans (quarterly), and staffing and resources.

C.4.2.4 The Contractor shall lead equipment development (with the assistance of NMFS). Specifically, the Contractor shall perform an iterative process of equipment testing and coordination with NMFS staff. Equipment must be developed and installed such that the current sampling processes are maintained. Equipment must be developed such that redundancies ensure adequate data collection. The Contractor shall provide a detailed description of their image collection system design and its components in their quote and as part of their reference sheet. The Contractor must also include detailed description of how the images will be paired with FSCS specimen data. The Contractor shall provide a protocol for troubleshooting common system problems.

The Contractor shall lead the planning phases of algorithm and software development (with the assistance of NMFS).

Specifically, the Contractor shall perform an iterative process of coordination with NMFS staff to ensure that the algorithms and software being developed will have the greatest utility feasible given the existing constraints (data, funding, and time available). Algorithms must be developed and implemented such that they provide data comparable to that currently recorded by humans doing EM review. The Contractor shall provide a detailed description of their algorithm and software development plan and its components in their quote and as part of their reference sheet. The

1305M221QNFFM0003 Page 13 of 66

Contractor must also include a detailed description of how the existing image data sets and existing open source software will be capitalized upon. Additionally, the Contractor shall lay layout the general plan for software development that specifies key attributes of the software including where the software will run (in the cloud or locally) and how the software will interface with existing open source EM review software. Finally, the Contractor should explicitly outline how the accuracy of algorithms will be assessed by users in applications where they are applied.

C.4.3 Camera and Equipment Installation

The Contractor shall provide a camera system necessary to capture a large specimen images at-sea from four stations in the vessels wet lab. These consist of three sampling stations as well as one station at the sorting belt (please see the wet lab schematic Appendix 2). The Contractor shall also have adequate spare or backup equipment available locally in case of a system failure or equipment malfunction. The vessel installation locations shall be determined through consultation with NMFS staff. Preparation, development, and deployment of camera systems may need to be done with limited access to the research vessel as the vessel's schedule and research commitments may prohibit extensive access prior to installation (for example, there may only be access to the vessel for one to two days prior to and between deployments).

The camera system supplied by the Contractor shall be approved by NMFS staff as appropriate quality for the development of image library. Suggested guidelines include a raw image format or a jpeg compression that avoids the loss of image quality (not H.264) as well as at least 2/3 size image sensors (2.3-5 mega pixels [MP]) to enhance overall data density. Specimen lighting must also be approved by NMFS and interested parties as appropriate for the ultimate goals of the study. Maintaining as consistent lighting as possible is desirable as variable lighting introduces complexities into the machine learning process. Because specimens are handled in the ship’s wet lab, equipment used for specimen imaging and illumination should be constructed in such a manner that they are robust and able to withstand spray (e.g., at least IP65 rated or equivalent). Data storage must also be approved by NMFS staff and interested parties as appropriate for the study and compliant with IT infrastructure. All equipment must be ruggedized such as to withstand the conditions present in catch processing locations.

In considering camera systems for this project the Contractor must pay close attention to the above fish processing section of the background (section C.2.2). Specifically, the positioning of the cameras and the frequency of image capture must be tuned so as to capture images at the rate that specimens are processed (which can occur rapidly). Also, there is a high likelihood that individuals processing specimens will have their hands occluding fish images for a significant portion of the time the fish is handled, and therefore the image collection scheme be able to collect images quickly (some fish are processed in a matter of three to five seconds). FMO strongly suggests that to capture the largest number of high quality images that the field of view from the camera system capture the entire length of the Scantrol measuring board at each sampling station (roughly 110 cm). Additionally, it is suggested that capturing images of the finer scale may help to increase counts of images (please see the wet lab schematic Appendix 2). Further, the scheme for pairing of images with additional data must take these issues into account.

C.4.4 Servicing of Equipment Aboard Vessels, and Field and Project Support

Prior to commencement of the active data collection component of the project, the Contractor shall set up local capacity to install equipment and provide field support to NMFS research vessel. The field support shall include; equipment installation, equipment or system troubleshooting, data retrievals (by means of in-person retrieval), and equipment removal. The Contractor shall provide a local service technician who is capable of providing all servicing events and needs on the short time frames associated with the beginning and end of NMFS survey legs. The data retrievals and equipment assessment shall occur after each survey leg no more than twenty-four (24) hours after trip landing. The

1305M221QNFFM0003 Page 14 of 66

Contractor shall present a formal plan on servicing, local capacity, and response time to address any equipment issues in their quote.

The Contractor shall create and maintain a study service tracking log to include all development, installation, service events, data retrievals, equipment or system troubleshooting and equipment removals. If equipment servicing or troubleshooting occurs the Contractor shall note the issue, length of time to fix the issue, and how the issue was rectified in the service tracking log. The tracking sheet shall be updated daily and the Contractor shall provide full access to NMFS. The Contractor shall also track all system failures (any instance where the equipment is not functional) and report all system failures to NMFS as soon as they are discovered.

The Contractor shall install, test, and initialize all the study equipment on the NMFS research vessel. The field service technician may be required to go out to sea on a trip with the vessel to understand sample processing operations and evaluate vessel configuration to meet project goals. Should the field service technician be required to go to sea they may be required to assist with fish processing and other cruise duties. In the rare event that the Bigelow be unable to operate such that the cruise is delayed or potentially conducted on another fishing vessel the Contractor must be prepared to deploy after a delay or potentially on a similarly sized fishing vessel.

It is important to note that the deployment of the image collection equipment will likely have to occur over a very short period of time (potentially as little as a period of hours immediately prior to the cruise). This is because very limited time is available for even the survey crew to put out equipment (likely between one and two days). These logistic challenges need to be met by the Contractor.

The Contractor shall provide instruction on the use and general care of the EM equipment to the vessel’s crews. The Contractor shall provide a detailed description of what is involved in the setting up and removal of equipment to NMFS.

The Contractor shall develop a vessel reference sheet which provides the description of the equipment, outlines NMFS crew responsibilities, and includes instruction on the use and care of the equipment. The vessel reference sheet is required prior to the first deployment of the equipment.

The Contractor shall develop a clear protocol for image and specimen data collection and pairing. The protocol shall outline suggested specimen handling protocols and camera equipment configurations being used throughout the project.

The study protocol document is a communication tool used to ensure that NMFS staff, Contractors, field technicians, and project coordination staff know what their roles are for a successful project implementation. The protocol document will include at a minimum; general image collection procedures, camera system configuration, operator responsibilities, protocols for data retrieval, specimen handling protocols, general equipment information, project objectives, basic camera and data system troubleshooting, project contacts, and any other pertinent information.

The Contractor’s quote shall include suggested solutions or arrangements to address power issues and data storage issues. Because the purpose of this study is to collect a large number of high quality images, data storage must be sufficient to ensure data collection over the full course of a survey leg. Data storage decision must be approved by NMFS staff and not impact the IT infrastructure of the research vessel. The Contractor shall coordinate and determine a schedule for equipment installation on each vessel and will take into account NMFS staff and the research vessels’ schedules and concerns. Again, the installation and maintenance of the study equipment shall not impact the operations of the survey crew or vessel in any substantive way.

C.4.5 Data Curation and Library Development

1305M221QNFFM0003 Page 15 of 66

The aim of this study is to collect as many annotated images as possible, thus the Contractor shall ensure data collection is maximized during the study period. This includes maximizing the number of images of unique specimens as well as maximizing the connection to relevant collected specimen information.

The Contractor shall curate the image and specimen data sets to remove poor quality images, or images for which specimen data are extracted from the final data set. The process for this curation will be documented so as to communicate the methods to NMFS and potential data end users. A key component in this data curation will be pairing the collected image data with specimen information collected. Matching may be achieved via a number of means, however image data must be collected such that it is time stamped and could be paired with specimen data via the matching of timestamps (should other methods fail).

During the project period NMFS will provide the Contractor with exact specifications for the final data set, but generally it will be a set of images and data labels suitable for machine learning applications. The Contractor shall provide NMFS with the highest quality digital images appropriate for later manipulation. The Contractor shall provide NMFS with data summaries from captured data and provide complete access to all data (both specimen and image data) for NMFS review. It is preferred equipment developers and data processing technicians have experience with machine learning and automated image processing or plan to partner with a vendor with this experience. Summary data provided to NMFS and images submitted for NMFS review purposes under this contract will be the property of NMFS.

The Contractor will work with vendors to determine the best licensing for this data set to encourage use by parties interested in developing algorithms for automated image analysis. Additionally, the Contractor shall determine an appropriate and effective means for distributing the large data set. NMFS must approve all licensing decisions and dissemination means.

C.4.6 Removal of equipment from the research vessel Contractor shall be responsible for removing the study equipment from the vessel at the end of the project. The Contractor shall remove all evidence of the equipment installation and care must be taken to return the vessel to the state it was in prior to the installation of the study equipment. The removal shall not unduly delay the research vessel from departing for other cruises.

C.4.7 Algorithm Development

The Contractor shall develop algorithms using industry standard open source tools to meet the needs of EM video review. These algorithms will meet the goals identified through consultation with NMFS during the project initiation (4.2). Examples of tasks that may be required include (but are not limited to): 1) species identification from New England’s waters, 2) the estimation of length of an individual catch items passed over a measuring board, and 3) the tracking of individual fish so as to develop piece counts of the fish processed on a vessel. The Contractor shall follow the best open source software development protocols including modern version control. The algorithms will be made such that they can be updated as simply as possible as new data becomes available. Utilization of modern machine learning techniques is essential for this task.

C.4.8 Bringing Machine Learning to EM Review

Following the development of machine learning algorithms, the Contractor shall being developing mechanisms to import these algorithms into EM review software. This process will follow the plan developed in coordination with NMFS.

1305M221QNFFM0003 Page 16 of 66

The Contractor shall work to the greatest extent practicable to make a tool that can be widely used. This is includes a software framework that allows access at multiple levels and that is impeccably documented. To the extent that it is practical this software should utilize existing open source software resources.

The Contractor shall create and maintain a detailed plan for progressing through the stages of development for this software. This includes developing the required project management framework and clearly establishing team roles.

This information shall be made available to NMFS upon request. As soon as possible the Contractor shall provide NMFS access to software to allow for feedback and experts in EM review to provide feedback. Clear documentation of these interactions and software request shall be maintained even if feature requests fall outside the scope of this project.

The Contractor shall provide instruction on the use of the software product to NMFS. This includes instructions for basic software installation and use. Also, detailed descriptions of software features. The Contractor will assist NMFS in performing pilot trials of the software on EM footage.

C.4.9 Using EM Review to Develop Machine Learning Training Data The Contractor will work to leverage existing image or video annotation tools to create a novel software tool that allows for the development of annotated training data during the course of regular EM review. This software may be stand alone or integrated with other tools developed in this project.