Attachment_1-Electronic_Monitoring_Data_File_Technical_Specifications.pdf

Attachment 1- Electronic Monitoring (EM) Summary Data Technical Specifications

FOR INFORMATIONAL PURPOSES ONLY

NOAA Fisheries Greater Atlantic Regional Fisheries Office Electronic Monitoring Summary Data File

Technical Requirements

Introduction

The NOAA Fisheries Greater Atlantic Regional Fisheries Office (GARFO) has begun implementation of Electronic Monitoring (EM) in select fisheries. EM is being implemented in the groundfish fishery to identify and enumerate groundfish discards by bottom trawl and gillnet vessels. EM is also being considered for implementation in the midwater trawl fisheries for herring and mackerel to verify the retention of catch. Both of these EM programs, as well as other variations of future programs, will involve deploying cameras onboard fishing vessels to collect video that would be reviewed and synthesized in summary reports. The content of the summary reports will vary between the different EM programs, but all summary reports will be comprised of data that would be used by GARFO for fisheries management purposes.

This document specifies such information as file format, security, submission protocol, and file structure for EM summary reports. The information contained in each summary report is subject to change as the needs of EM programs evolve.

Background

At present, EM programs are being developed in the groundfish and herring/mackerel fisheries.

Each of those programs will generate EM summary reports that have the same format and structure, but that have different data elements. Appendixes A, B, and C below details the summary report format for the groundfish fishery. Appendixes D, E, and F below details the summary report format for the herring/mackerel fisheries.

Technical Guidelines

File Format EM Summary Data files must;

1. contain data in the specified standardized format that can be successfully loaded into GARFO databases using existing load routines.

2. be submitted in valid JSON (JavaScript Object Notation) format and in the manner and format described herein.

File Transfer Security EM Summary Data files must be transmitted to GARFO by file upload using a HTTPS POST.

To use this method, identify the appropriate fishery.

Groundfish: request the page at:

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1

Herring/Mackerel: request the page at:

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1

Include the JSON object in the Request Payload

● A submission will return a JSON object

● For a successful submission ‘status’ will be 'Report Accepted'

● A failure will return one or more of a variety of descriptive error messages

● For validation testing, add ‘/test’ to the end of the URL. For example, https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1/test

Quality Assurance/Quality Control EM summary data files submitted to GARFO will be subject to numerous quality control audits.

Submissions that fail audit checks will result in the rejection of the report and will require the EM service provider to correct and resubmit the report.

Corrections to Previously Submitted EM Summary Data Files Changes to previously submitted EM summary data files may be necessary to correct errors or amend records. When these situations are identified, the EM Service Provider will be required to amend the source data and resubmit the record to GARFO. All resubmissions will supersede the original submission in its entirety.

RFQ 1305M218QNFFM0043

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1 https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1 https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1 https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_herring/v_1

Appendix A – Groundfish Export File Field Descriptions report_type: String: "EM_GROUNDFISH" vendor_id: Number: GARFO assigned vendor number vendor_api_key: String: GARFO assigned key vendor_software_version: String: (optional) EM vendor software version if applicable vessel_permit_number: Number: GARFO assigned 6 digit vessel permit number vessel_name: String: vessel name associated with vessel permit number trip_id: Number: 14 digit eVTR generated trip identifier nefop_indicator: Number: (0,1) was a NEFOP observer on the trip? 0=No, 1=Yes total_hauls: Number: total number of hauls on the trip reviewed_hauls: Number: total number of hauls reviewed hauls: Array: contains all elements described below haul_id: Number: sequence number, relative to total hauls in trip, of haul being reviewed haulback_start_datetime: Number: date and time at the beginning of haul back* haulback_end_datetime: Number: date and time at the end of haul back* haulback_start_lat: Number: latitude (in Decimal Degrees or Degrees:Minutes) where the start of haul back occurred ** haulback_start_long: Number: longitude (in Decimal Degrees or Degrees:Minutes) where the start of haul back occurred** catch_sorting_end_datetime: Number: date and time at the end of catch sorting operations* observed: Number: (0,1) was all discard data able to be collected from the haul? 0=No, 1=Yes reviewed: String: (FULL, PARTIAL, NONE) how much video of the haul was reviewed?

vessel_specific_events: Array: see selections in Table 1. Values are strings.

crew_specific_events: Array: see selections in Table 2. Values are strings.

em_specific_events: Array: see selections in Table 3. Values are strings.

image_quality: String: (HIGH, MEDIUM, LOW) see definitions in Table 4 image_quality_reason: Array: see selections in Table 5 comments: String: notations from EM reviewer to describe vessel, crew, and EM events and image quality issues discards: Array: contains all elements described below species: String: species code of the observed discards.

see selections in Table 6.

count_discarded: Number: count of observed discards pounds_discarded: Number: calculated estimate of observed discards in pounds weight_determined_by: String: see selections in Table 7. Value are strings discard_condition: String: see selections in Table 8. Value are strings

* Datetime: values are represented as a ten-digit number in the format YYMMDDHH24MI. YY=two-digit year, MM = two-digit month, DD = two-digit day of month, HH = two-digit hour using 24-hour clock, MI = two digit minutes. All values are two digits padded with zeros. Full datetime must be a ten-digit numeric value. Dates and times should be determined based on local time on the fishing vessel at the time of fishing activity.

** Location: values may be represented in either Decimal Degree or Degrees and Minutes, consistent with existing eVTR requirements. If using Decimal Degrees, values are represented as a six-digit number in the format DD.DDDD; if using Degree: Minutes, values are represented as a four-digit number in the format DD:MM. For either format, use a minus sign for latitude if below the equator otherwise omit; use a minus sign for longitude if west of the prime meridian otherwise omit. Please note that LORAN will not be accepted, despite current eVTR specifications.

Table 1 – Fishing Operations Specific Events - Groundfish

RFQ 1305M218QNFFM0043

PROTECTED SPECIES

INTERACTION

BROKEN CODEND GEAR CONFLICT

OTHER GEAR ISSUES OTHER MECHANICAL FAILURE

Table 2 - Crew Specific Events - Groundfish

DISCARDING OUT OF CAMERA

VIEW

CAMERA DIRTY CAMERA BLOCKING

FISH HANDLING IMPROPER

PROTOCOL

LUMP DISCARDING OTHER

Table 3 - EM Specific Events - Groundfish

CAMERA FAILURE SENSOR GAPS VIDEO GAPS

MEASURING SURFACE

VISIBILITY

OTHER SYSTEM ISSUES

Table 4 - Image Quality Definition - Groundfish

HIGH The imagery was very clear and the EM reviewer had a good view of fishing activities. Focus is good, light levels are high and all activity is easily seen.

MEDIUM The view was acceptable, but there may be some difficulty assessing discards.

Slight blurring or slightly darker conditions hamper, but do not impede analysis.

LOW The imagery is difficult to assess. Some camera views may not be available.

Imagery is somewhat blurred or lighting has largely been diminished. Some factors such as the fishing line going out of camera view or crew standing between the catch and the camera for extended periods of time may have also occurred.

UNUSABLE The imagery is unusable

Table 5 - Image Quality Reasons - Groundfish

NO ISSUES CAMERA FAILURE OUT OF FOCUS

CAMERA POOR ANGLE GLARE LIGHTING POOR

WATER SPOTS OTHER

Table 6 - Species Codes - Groundfish

Species Name Species Code Species Name Species Code

Atlantic cod COD Yellowtail flounder FLYT

Haddock HADD Witch flounder FLGS

Pollock POLL Winter flounder FLBB

White hake WHAK American plaice flounder FLDAB

Atlantic halibut HAL Windowpane flounder FLSD

Redfish RED Atlantic wolfish CAT

Ocean pout POUT No Groundfish Discards NGD

Groundfish Flounder Unknown

GFFLDR Groundfish Round Fish Unknown GFRD

Marine Mammal MAM Turtle TUR

Bird BRD Non-Fish NONFSH

Non-Groundfish NGF Fish Unknown FSHNK

Summer flounder FLUKE

Table 7 – Discarded Fish Weight Determined By***

Table 8 – Condition of Discarded Fish

***Discarded Fish Weight Determined By: when selecting UNDETERMINED, the reviewer will report a count, but no weight (value will be null); when selecting VISUAL or SCALE, the reviewer will report both a count and a weight.

Appendix B – Groundfish Export File Format

LENGTH VOLUMETRIC UNDETERMINED

VISUAL ESTIMATE SCALE SUB-SAMPLE

INTACT LUMF

JSON object description. Object is HTTPS POST Request Payload.

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1

{ "report_type": STRING (EM_GROUNDFISH), "vendor_id": NUMBER, "vendor_api_key": STRING, "vendor_software_version": STRING (OPTIONAL), "vessel_permit_number": NUMBER, "vessel_name": STRING, "trip_id": NUMBER, "nefop_indicator": NUMBER, "total_hauls": NUMBER, "reviewed_hauls": NUMBER, "hauls": ARRAY "haul_id": NUMBER, "haulback_start_datetime": NUMBER, "haulback_end_datetime": NUMBER, "haulback_start_lat": NUMBER, "haulback_start_lon": NUMBER, "catch_sorting_end_datetime":NUMBER, "observed": NUMBER, "reviewed": STRING, "vessel_specific_events": ARRAY{STRING}(OPTIONAL), "crew_specific_events": ARRAY{STRING}(OPTIONAL), "em_specific_events": ARRAY{STRING}(OPTIONAL), "image_quality": STRING, "image_quality_reason": ARRAY{STRING}, "comments": STRING, "discards": ARRAY "species": STRING, "count_discarded": NUMBER, "pounds_discarded": NUMBER, "weight_determined_by": STRING, "discard_condition": STRING(OPTIONAL)

RFQ 1305M218QNFFM0043

https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1 https://www.greateratlantic.fisheries.noaa.gov/apps/fol/base_api/em_groundfish/v_1

Appendix C – Groundfish Export File example

Export file example

"report_type": "EM_GROUNDFISH", "vendor_id": 1, "vendor_api_key": "A1fr$44jj88986856", "vendor_software_version": "v1 .3", "vessel_permit_number": 123456, "vessel_name": "Nautilus", "trip_id": 12345616052207, "nefop_indicator": 0, "total_hauls": 3, "reviewed_hauls": 3, "hauls": [{

"haul_id": 1, "haulback_start_datetime": 1512031345, "haulback_end_datetime": 1512031405, "haulback_start_lat": 41.4430, "haulback_start_lon": -66.8570, "catch_sorting_end_datetime": 1512031435, "observed": 1, "reviewed": "FULL", "vessel_specific_events": ["OTHER", "GEAR CONFLICT"], "crew_specific_events": [], "em_specific_events": [], "image_quality": "HIGH", "image_quality_reason": [], "comments": "All cameras working", "discards": [{

"species": "HADD", "count_discarded": 65, "pounds_discarded": 22.9676, "weight_determined_by": "LENGTH", "discard_condition": "INTACT"

"species": "HADD", "count_discarded": 65, "pounds_discarded": 0, "weight_determined_by": "UNDETERMINED", "discard_condition": ""

"species": "POLL", "count_discarded": 84, "pounds_discarded": 121.8596, "weight_determined by": "LENGTH", "discard_condition": "INTACT"

"haul_id": 2, "haulback_start_datetime": 1512031443, "haulback_end_datetime": 1512031507, "catch_sorting_end_datetime": 1512031542, "observed": 1, "reviewed": "FULL", "vessel_specific_events": [], "crew_specific_events": [], "em_specific_events": [], "image_quality": "HIGH", "image_quality_reason": null, "comments": "", "discards": [{

"species": "HADD", "count_discarded": 85, "pounds_discarded": 27, "weight_determined by": "LENGTH", "discard_condition": "INTACT"

"haul_id": 3, "haulback_start_datetime": 1512031710, "haulback_end_datetime": 1512031735, "catch_sorting_end_datetime": 1512031755, "observed": 1, "reviewed": "FULL", "vessel_specific_events": [], "crew_specific_events": [], "em_specific_events": [], "image_quality": "HIGH", "image_quality_reason": null, "comments": "", "discards": [{ "species": "HADD", "count_discarded": 50, "pounds_discarded": 10, "weight_determined by": "LENGTH", "discard_condition": "INTACT"

"species": "HADD", "count_discarded": 1, "pounds_discarded": 3, "weight_determined by": "LENGTH", "discard_condition": "LUMF"