Attachment_1-EM_Summary_Data_Technical_Specifications.pdf

NOAA Fisheries Greater Atlantic Regional Fisheries Office (GARFO) Electronic Monitoring Summary Data File

Technical Requirements

Policy Date 04/22/2016

Introduction

The NOAA Fisheries Greater Atlantic Regional Fisheries Office (GARFO) has begun implementation of Electronic Monitoring (EM) in select fisheries. EM is being implemented in the groundfish fishery to identify and enumerate groundfish discards by bottom trawl and gillnet vessels. EM is also being considered for implementation in the midwater trawl fisheries for herring and mackerel to verify the retention of catch. Both of these EM programs, as well as other variations of future programs, will involve deploying cameras onboard fishing vessels to collect video that would be reviewed and synthesized in summary reports. The content of the summary reports will vary between the different EM programs, but all summary reports will be comprised of data that would be used by GARFO for fisheries management purposes.

This document specifies such information as file format, security, submission protocol, and file structure for EM summary reports. The information contained in each summary report is subject to change as the needs of EM programs evolve.

Background

At present, EM programs are being developed in the groundfish and herring/mackerel fisheries.

Each of those programs will generate EM summary reports that have the same format and structure, but that have different data elements. Appendixes A, B, and C below details the summary report format for the groundfish fishery. Appendixes D, E, and F below details the summary report format for the herring/mackerel fisheries.

Technical Guidelines

File Format EM Summary Data files must;

1. Contain data in the specified standardized format that can be successfully loaded into GARFO databases using existing load routines.

2. Submitted in valid JSON (JavaScript Object Notation) format and in the manner and format described herein

RFQ EA133F-16-RQ-1822

ATTACHMENT 1

File Transfer Security EM Summary Data files must be transmitted to GARFO by file upload using a HTTPS POST.

To use this method, identify the appropriate fishery.

Groundfish: request the page at:

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1

Herring/Mackerel: request the page at:

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1

Include the JSON object in the Request Payload

● A submission will return a JSON object

● For a successful submission ‘status’ will be 'Report Accepted'

● A failure will return one or more of a variety of descriptive error messages

● For validation testing, add ‘/test’ to the end of the URL. For example, https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1/test

Quality Assurance/Quality Control EM summary data files submitted to GARFO will be subject to numerous quality control audits.

Submissions that fail audit checks will result in the rejection of the report and will require the EM service provider to correct and resubmit the report.

Corrections to Previously Submitted EM Summary Data Files Changes to previously submitted EM summary data files may be necessary to correct errors or amend records. When these situations are identified, the EM Service Provider will be required to amend the source data and resubmit the record to GARFO. All resubmissions will supersede the original submission in its entirety.

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1 https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1 https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1

Appendix A – Groundfish Export File Field Descriptions report_type: String: "EM_GROUNDFISH" vendor_id: Number: GARFO assigned vendor number vendor_api_key: String: GARFO assigned key vendor_software_version: String: (optional) EM vendor software version if applicable vessel_permit_number: Number: GARFO assigned 6 digit vessel permit number vessel_name: String: vessel name associated with vessel permit number trip_id: Number: 14 digit eVTR generated trip identifier nefop_indicator: Number: (0,1) was a NEFOP observer on the trip? 0=No, 1=Yes total_hauls: Number: total number of hauls on the trip reviewed_hauls: Number: total number of hauls reviewed hauls: Array: contains all elements described below haul_id: Number: sequence number, relative to total hauls in trip, of haul being reviewed haul_start_datetime: Number: date and time at the beginning of haul back* haul_end_datetime: Number: date and time at the end of haul back* catch_sorting_end_datetime: Number: date and time at the end of catch sorting operations* observed: Number: (0,1) was all discard data able to be collected from the haul? 0=No, 1=Yes vessel_specific_events: Array: see selections in Table 1. Values are strings.

crew_specific_events: Array: see selections in Table 2. Values are strings.

em_specific_events: Array: see selections in Table 3. Values are strings.

image_quality: String: (HIGH, MEDIUM, LOW) see definitions in Table 4 image_quality_reason: Array: see selections in Table 5 comments: String: notations from EM reviewer to describe vessel, crew, and EM events and image quality issues discards: Array: contains all elements described below species: String: species code of the observed discards.

see selections in Table 6.

count_discarded: Number: count of observed discards pounds_discarded: Number: calculated estimate of observed discards in pounds

* Datetime: values are represented as a ten-digit number in the format YYMMDDHH24MI. YY=two digit year, MM = two digit month, DD = two digit day of month, HH = two digit hour using 24 hour clock, MI = two digit minutes. All values are two digits padded with zeros. Full datetime must be a ten-digit numeric value. Dates and times should be determined based on local time on the fishing vessel at the time of fishing activity.

Table 1 – Fishing Operations Specific Events - Groundfish

PROTECTED SPECIES

INTERACTION

BROKEN CODEND GEAR CONFLICT

OTHER GEAR ISSUES OTHER MECHANICAL FAILURE

Table 2 - Crew Specific Events - Groundfish

DISCARDING OUT OF CAMERA

VIEW

CAMERA DIRTY CAMERA BLOCKING

FISH HANDLING IMPROPER

PROTOCOL

LUMP DISCARDING OTHER

Table 3 - EM Specific Events - Groundfish

CAMERA FAILURE SENSOR GAPS VIDEO GAPS

MEASURING SURFACE

VISIBILITY

OTHER SYSTEM ISSUES

Table 4 - Image Quality Definition - Groundfish

HIGH The imagery was very clear and the EM reviewer had a good view of fishing activities. Focus is good, light levels are high and all activity is easily seen.

MEDIUM The view was acceptable, but there may be some difficulty assessing discards.

Slight blurring or slightly darker conditions hamper, but do not impede analysis.

LOW The imagery is difficult to assess. Some camera views may not be available.

Imagery is somewhat blurred or lighting has largely been diminished. Some factors such as the fishing line going out of camera view or crew standing between the catch and the camera for extended periods of time may have also occurred.

Table 5 - Image Quality Reasons - Groundfish

NO ISSUES CAMERA FAILURE OUT OF FOCUS

CAMERA POOR ANGLE GLARE LIGHTING POOR

WATER SPOTS OTHER

Table 6 - Species Codes - Groundfish

Species Name Species Code Species Name Species Code

Atlantic cod COD Yellowtail flounder FLYT

Haddock HADD Witch flounder FLGS

Pollock POLL Winter flounder FLBB

White hake WHAK American plaice flounder FLDAB

Atlantic halibut HAL Windowpane flounder FLSD

Redfish RED Atlantic wolfish CAT

Ocean pout POUT No Groundfish Discards NGD

Appendix B – Groundfish Export File Format JSON object description. Object is HTTPS POST Request Payload.

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1

{ "report_type": STRING (EM_GROUNDFISH), "vendor_id": NUMBER, "vendor_api_key": STRING, "vendor_software_version": STRING (OPTIONAL), "vessel_permit_number": NUMBER, "vessel_name": STRING, "trip_id": NUMBER, "nefop_indicator": NUMBER, "total_hauls": NUMBER, "reviewed_hauls": NUMBER, "hauls": ARRAY

"haul_id": NUMBER, "haul_start_datetime": NUMBER, "haul_end_datetime": NUMBER, "catch_sorting_end_datetime":NUMBER, "observed": NUMBER, "vessel_specific_events": ARRAY{STRING}(OPTIONAL), "crew_specific_events": ARRAY{STRING}(OPTIONAL), "em_specific_events": ARRAY{STRING}(OPTIONAL), "image_quality": STRING, "image_quality_reason": ARRAY{STRING}, "comments": STRING, "discards": ARRAY "species": STRING, "count_discarded": NUMBER, "pounds_discarded": NUMBER https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1 https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1

Appendix C – Groundfish Export File example

Export file example

"report_type": "EM_GROUNDFISH", "vendor_id": 1, "vendor_api_key": "A1fr$44jj88986856", "vendor_software_version": "v1 .3", "vessel_permit_number": 123456, "vessel_name": "Nautilus", "trip_id": 12345616052207, "nefop_indicator": 0, "total_hauls": 3, "reviewed_hauls": 3, "hauls":

"haul_id": 1, "haul_start_datetime": 1512031345, "haul_end_datetime": 1512031405, "catch_sorting_end_datetime": 1512031435, "observed": 1, "vessel_specific_events": ["OTHER", "GEAR CONFLICT"], "crew_specific_events": [], "em_specific_events": [], "image_quality": "HIGH", "image_quality_reason": [], "comments": "All cameras working", "discards": [ "species": "HADD", "count_discarded": 65, "pounds_discarded": 22.9676 "species": "POLL", "count_discarded": 84, "pounds_discarded": 121.8596

"haul_id": 2, "haul_start_datetime": 1512031443, "haul_end_datetime": 1512031507, "catch_sorting_end_datetime": 1512031542, "observed": 1, "vessel_specific_events": [], "crew_specific_events": [], "em_specific_events": [], "image_quality": "HIGH", "image_quality_reason": null, "comments": "", "discards": [ "species": "HADD", "count_discarded": 85, "pounds_discarded": 27

"haul_id": 3, "haul_start_datetime": 1512031710, "haul_end_datetime": 1512031735, "catch_sorting_end_datetime": 1512031755, "observed": 1, "vessel_specific_events": [], "crew_specific_events": [], "em_specific_events": [], "image_quality": "HIGH", "image_quality_reason": null, "comments": "", "discards": [ "species": "HADD", "count_discarded": 50, "pounds_discarded": 10

Appendix D – Herring/Mackerel Export File Field Descriptions report_type: String: "EM_HERRING" vendor_id: Number: GARFO assigned vendor number vendor_api_key: String: GARFO assigned key vendor_software_version: String: (optional) EM vendor software version if applicable vessel_permit_number: Number: GARFO assigned 6 digit vessel permit number vessel_name: String: vessel name associated with vessel permit number trip_id: Number: 14 digit eVTR generated trip identifier pair_trawl_indicator: Number: (0,1) was this a pair trawling trip? 0=No, 1=Yes pair_trawl_permit_number: Number: GARFO assigned vessel permit number of sister vessel nefop_indicator: Number: (0,1) was a NEFOP observer on the trip? 0=No, 1=Yes total_hauls: Number: total number of hauls on the trip reviewed_hauls: Number: total number of hauls reviewed* discard_observed: Number: (0,1) were discard event(s) observed? 0 = no, 1 = yes hauls: Array: contains all elements described below haul_id: Number: sequence number, relative to total hauls in trip, of haul being reviewed haul_start_datetime: * Number: date and time at the beginning of haul back** haul_end_datetime: * Number: date and time at the end of haul back** observed: Number: (0,1) was all discard data able to be collected from the haul? 0=No, 1=Yes vessel_specific_events: Array: see selections in Table 7. Values are strings.

crew_specific_events: Array: see selections in Table 8. Values are strings.

em_specific_events: Array: see selections in Table 9. Values are strings.

image_quality: String: (HIGH, MEDIUM, LOW) see definitions in Table 10 image_quality_reason: Array: see selections in Table 11 comments: String: notations from EM reviewer to describe vessel, crew, and EM events and image quality issues discard_events: Array: contains all elements described below discard_type: String: (SLIPPAGE, OPERATIONAL, CULLED) discard_time: Number: date and time of discard event** discard_species: String: discarded species. see options in Table 12

* Hauls: The time in which a vessel first deploys gear until it is retrieved and aboard the vessel.

** Datetime: values are represented as a ten-digit number in the format YYMMDDHH24MI. YY=two digit year, MM = two digit month, DD = two digit day of month, HH = two digit hour using 24 hour clock, MI = two digit minutes. All values are two digits padded with zeros. Full datetime must be a ten-digit numeric value. Dates and times should be determined based on local time on the fishing vessel at the time of fishing activity.

Table 7 Fishing Operations Specific Events – Herring/Mackerel protected species interaction mechanical failure gear conflict other

Table 8 - Crew Specific Events – Herring/Mackerel

Discarding out of camera view Dirty camera Blocking camera other

Table 9 - EM Specific Events – Herring/Mackerel

Camera failure Sensor gaps Video gaps

Other system issues

Table 10 - Image quality definition

High The imagery was very clear and the EM reviewer had a good view of fishing activities. Focus is good, light levels are high and all activity is easily seen.

Medium The view was acceptable, but there may be some difficulty assessing discards.

Slight blurring or slightly darker conditions hamper, but do not impede analysis.

Low The imagery is difficult to assess. Some camera views may not be available.

Imagery is somewhat blurred or lighting has largely been diminished. Some factors such as the fishing line going out of camera view or crew standing between the catch and the camera for extended periods of time may have also occurred.

Table 11 - Image Quality reasons

No issues poor lighting Out of focus

Poor camera angle glare

Water spots other

Table 12 - Discarded Species list – Herring/Mackerel

MANAGED SPECIES PROTECTED SPECIES

HADDOCK MARINE MAMMAL

DOGFISH TURTLE

SMALL PELAGIC BIRD

UNKNOWN FISH UNKNOWN NON-FISH

Appendix E – Herring/Mackerel Export File Format JSON object description. Object is HTTP POST Request Payload https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1

{ "record_type": STRING (EM_HERRING), "vendor_id": NUMBER, "vendor_api_key": STRING, "vendor_software_version": STRING (OPTIONAL), "vessel_permit_number": NUMBER, "vessel_name": STRING, "trip_id": NUMBER, "pair_trawl_indicator": NUMBER, "pair_trawl_permit_number": NUMBER, "nefop_indicator": NUMBER, "total_hauls": NUMBER, "reviewed_hauls": NUMBER, "discard_observed": NUMBER, "hauls": [ "haul_id": NUMBER, "haul_start_datetime": NUMBER, "haul_end_datetime": NUMBER, "observed": NUMBER, "vessel_specific_events": ARRAY{STRING}(OPTIONAL), "crew_specific_events": ": ARRAY{STRING}(OPTIONAL), "em_specific_events": ": ARRAY{STRING}(OPTIONAL), "image_quality": STRING, "image_quality_reason": ARRAY{STRING}, "comments": STRING, "discard_events": [

"discard_type": STRING, "discard_time": NUMBER, "discard_species": STRING "discard_type": STRING, "discard_time": NUMBER, "discard_species": STRING

Appendix F – Herring/Mackerel Export File example

Export file example { "record_type":"EM_HERRING", "vendor_id": 1, "vendor_api_key":"A1fr$44jj88986856" "vendor_software_version": "v2.6", "vessel_permit_number": 123456, "vessel_name": "Nautilus", "trip_id": 12345616090112, "pair_trawl_indicator": 0, "pair_trawl_permit_number": null, "nefop_indicator": 0, "total_hauls": 4, "reviewed_hauls": 4, "discard_observed": 1, "hauls": [ "haul_id": 1, "haul_start_datetime": 1609020500, "haul_end_datetime": 1609020630, "observed": 1, "vessel_specific_events": [], "crew_specific_events": [], "em_specific_events": [], "image_quality": "high", "image_quality_reason": NO ISSUES, "comments": [], "haul_id": 2, "haul_start_datetime": 1609020730, "haul_end_datetime": 1609020845, "observed": 1, "vessel_specific_events": "OTHER", "crew_specific_events": "BLOCKING CAMERA", "em_specific_events": [], "image_quality": "MEDIUM", "image_quality_reason": "WATER SPOTS", "comments": "CREW NEEDS TO CLEAN LENSES", "haul_id": 3, "haul_start_datetime": 1609021015, "haul_end_datetime": 1609021145, "observed": 1, "vessel_specific_events": [], "crew_specific_events": [], RFQ EA133F-16-RQ-1822

"em_specific_events": [], "image_quality": "LOW", "image_quality_reason": "POOR CAMERA ANGLE","OUT OF FOCUS", "comments": "CREW NEEDS TO MAINTAIN EQUIPMENT", "haul_id": 4, "haul_start_datetime": 1609021300, "haul_end_datetime": 1609021400, "observed": 1, "vessel_specific_events": [], "crew_specific_events": [], "em_specific_events": [], "image_quality": "HIGH", "image_quality_reason": "NO ISSUES", "comments": "CREW CLEANED EQUIPMENT", "discard_events": [

"discard_type": "SLIPPAGE", "discard_time": "1605231355", "discard_species": "HADDOCK" "discard_type": "OPERATIONAL", "discard_time": "1605231630", "discard_species": "DOGFISH"

RFQ EA133F-16-RQ-1822