Attachment 6 Topsail Overview.pdf

29-Jun-2020 PHC – Page 1

Topsail Overview

Introduction

Topsail is an automated reporting system built to assist monitoring of VMS-related activities in the Northeast. The systems consists of a Windows application server and Oracle database, maintained by the OCIO, in Seattle, Washington. Topsail combines information from the Nationwide VMS (vTrack) database with regional databases (collectively known as the NERO database). Regularly-scheduled reports are written to a shared folder on the NEDSHARE system in Gloucester, and some are sent by email to authorized stakeholders. Alerts to events of special interest are also sent by email or by text message (SMS).

Topsail was created and is maintained by the Topsail developer, a contract worker located with OLE in Gloucester, Massachusetts. Topsail programming is written in the Visual Basic and C# languages for .Net Frameworks. Data sources include NOAA Oracle databases accessed by ODBC connections, and Microsoft Access databases accessed by OleDb connections.

Topsail is maintained as an active and critical resource, with weekday monitoring and frequent updating by the Topsail developer, using a Microsoft Windows development workstation housed in Gloucester.

Topsail manages critical processes in support of both OLE and GARFO:

Sends VMS vessel trip start and trip end notification messages to vessel operators and to secondary email addresses.

Provides VMS compliance status and vessel power down LOE status to the GARFO Permits office.

Provides annotated NAFO catch and other messages to GARFO stakeholders.

Sends automated alerts to enforcement officers/agents and partners.

In calendar year 2019 Topsail sent 264,636 automated email messages, including 95,830 sent to @noaa.gov email addresses, 7,877 sent to @uscg.mil email addresses, 4,653 sent to other government email addresses (@dep.nj.gov), and 156,276 sent to non-government-designated email addresses.

Approximately 87,959 messages where sent to shipboard VMS email addresses.

When Topsail programming identifies a serious processing failure it sends an email notification to the Topsail developer. Minor processing issues are written to program log files, which the Topsail developer checks daily. The Topsail developer evaluates the cause and impact of each issue and failure, and effects corrective action. The causes of most issues and failures are temporary, such as brief or scheduled network or database service interruptions, in which case the impacted Topsail processes can be re-run. In more serious instances, Topsail programming and data sources must be investigated and corrected.

Due to the dynamic nature of fisheries management, Topsail programming often needs revision or enhancement to provide new reports to serve as fisheries monitoring tools.

Topsail Automated Processes

29-Jun-2020 PHC – Page 2

OneShot – Executes special programs that are queued to run only once, such as when a daily report needs to be regenerated to produce a special update. Runs once every 5 minutes.

VMSCleanUp.exe – Runs once each morning to remove outdated temporary files.

VMSDataProviders.exe – Produces a list of OleDb and ODBC data sources available on the Topsail computer. Run manually when investigating database connection issues.

VMSDeclarationsVTK.exe – Extracts new vessel declaration records from the vTrack database, annotating them to the Topsail database. Runs once each hour.

VMSFMCDataCacheDB.exe – Makes a local cache of selected vTrack information for use by programs that would otherwise be a burden upon the vTrack database. Runs every four hours.

VMSGIS.exe – Builds a local cache of vTrack geometries (similar to shapefiles). Used by programs that do lots of spatial queries. Also used to maintain a list of port and mooring locations. Runs daily.

VMSLogfileMgr.exe – Moves Topsail program log files to an archive folder in case they are later needed for investigating issues.

VMSNAFOComm.exe – Reads NAFO message information from vTrack and annotates it for sharing with GARFO, via the Topsail database. Runs once each half hour during the NAFO fishing season.

VMSNAFOReports.exe – Generates reports of NAFO messages, copying them to NEDSHARE and sending notifications to selected individuals. Runs once each half hour during the NAFO fishing season.

VMSNEAuxTrans.exe – Produces a daily report of Northeast VMS vessels that are equipped with auxiliary VMS transceivers.

VMSNECouncilMtgRpt.exe – Produces a daily summary of Northeast VMS vessel information, used for management reports.

VMSNEDealerLandings.exe – Produces a daily report of new dealer landings reported for selected vessels. Used for monitoring special vessels of interest.

VMSNEDeclareCheck.exe – Provides alerts to invalid trip declarations. Runs once each hour.

VMSNEDualRegion.exe – Produces a daily report of Northeast VMS vessels that report by VMS to other regions in addition to the Northeast.

VMSNEEmailAlias.exe – Produces a daily NEDSHARE report of known alias email addresses for Northeast VMS vessels.

VMSNEEmailHist.exe – Produces a daily count of automated emails from Topsail, organized by program and recipient. Helps in tracking where Topsail emails are being sent.

VMSNEExcludedVessels.exe – Produces a daily report of email addresses that are stopped from being sent automated emails. Emails are stopped for a variety of reasons, including requests from vessel owners.

29-Jun-2020 PHC – Page 3

VMSNEFormDuplicates.exe – Produces a daily report of Northeast VMS eForms that are near duplicates.

Used to alert to eForms that may need to be edited or removed from GARFO records if they are unintended duplicate submissions.

VMSNEFormParseError.exe – Produces a daily report of Northeast VMS eForm parsing errors within vTrack.

VMSNEFormRcvd.exe – Sends special notifications when selected Northeast VMS eForms are received.

Used to notify stakeholders who have asked for notifications to help them monitor selected vessels.

VMSNEFurloughed.exe – Produces a daily report of Northeast VMS vessels that are furloughed; vessels that have been deactivated in vTrack while on authorized powered-down exemption status.

VMSNEGmailMgr.exe – Runs every half hour to look for selected new email arrivals in the OLE Drop Box and reorganize them into specified folders.

VMSNEIndustryContactBackup.exe – Copies a daily back-up of the Industry Contact Log from NEDSHARE to the Topsail database, in case of accidental loss of the NEDSHARE version.

VMSNEIndustryContactRpt.exe – Generates a daily report of recent Industry Contact Log activities, used for producing team activity reports.

VMSNEInvalidOPR.exe – Produces a daily report of Northeast VMS eForms that have invalid Operator Permit numbers.

VMSNEInvalidVTR.exe – Produces a daily report of Northeast VMS eForms that have invalid VTR serial numbers.

VMSNEJeopardyLOE.exe – Produces a daily report of vessels whose authorization to be powered down will expire within the next 10 days, or which have already expired.

VMSNELoeLandings.exe – Produces a weekly report of Northeast VMS vessels that had dealer landings for dates when they were authorized to be powered down, and were therefore presumed to not be fishing.

VMSNELongTrips.exe – Produces a daily report of Northeast VMS vessels that have been on Northeast-declared VMS trips for 15 days or more, which is an unusual length of time.

VMSNEMgrRpt.exe – Produces a weekly summary of Northeast VMS vessel counts in various categories.

Used for producing management reports.

VMSNENonReporting.exe - Produces a daily report of Northeast VMS vessels that have not reported a position within the past 24 hours been reporting position as required. The report includes a list of vessels exempted from reporting and the reason for their exemption. Also sends automated notifications to vessels and vessel owners.

VMSNENoPayActivity.exe – Produces a daily report of Northeast VMS vessels that have been deactivated in the vTrack database for non-payment to a vendor.

VMSNEObsoleteSoftware.exe – Produces a daily report of Northeast VMS vessels that have submitted Northeast eForms using discontinued VMS software.

29-Jun-2020 PHC – Page 4

VMSNEOmnitracsRemaining.exe – Temporary program that produces a daily report of Northeast VMS vessels still identified as equipped with only discontinued Omnitracs equipment.

VMSNEOperatorWatch.exe – Temporary program that produces alerts to selected Northeast VMS eForm submissions. Used for monitoring compliance with operator sanctions.

VMSNEOverlookedBoats.exe – Weekly report of non-VMS Northeast vessels that have permits and activities histories indicating they should be using VMS.

VMSNEPortAreaRpt.exe – Report run at 7 AM and 7 PM US/Eastern Time to provide special vessel activity summaries around selected ports.

VMSNEPosFliers.exe – Daily report of Northeast VMS vessel position reports that appear to indicate unlikely excessive speed, which may indicate a VMS performance issue.

VMSNEPosOrphans.exe – Daily report of Northeast VMS vessel position reports that occurred inside the Northeast Fishery but which vTrack did not associate with a trip.

VMSNEPwdAtSea.exe – Daily report of vessels that sailed underway while on a powered-down declaration, which is not allowed.

VMSNEPwrDwnLoad.exe – Twice-daily program that downloads powered-down database information from NEDSHARE to the Topsail database for use by Topsail applications.

VMSNEPwrDwnTruants.exe – Daily report of vessels that sailed underway while on authorized powered-down status.

VMSNEReconfirmIVR.exe – Daily program that downloads Northeast VMS re-declarations from GARFO’s IVR declaration database. Alerts to new re-declarations.

VMSNEReportingRate.exe – Daily report of Northeast VMS vessels that have not reported positions at the required rate.

VMSNEReportPage.exe – Half-hourly program that uploads new Topsail reports to NEDSHARE so they can be viewed through the Topsail reports page.

VMSNERqmtNVR.exe – Daily report of Northeast vessels that may require VMS if they have a specific recent fishing history, but which currently don’t have VMS as a requirement. Requested by the USCG for vessels with multispecies permits that have not taken a multispecies trip this fishing year and have therefore not activated their VMS requirement.

VMSNESameDayTrips.exe – Daily report of vessels which have taken more than one LAGC scallop trip in the same calendar day.

VMSNESansPermit.exe – Weekly report of Northeast VMS vessels which currently hold no Northeast permits.

VMSNEScallopIFQLow.exe – Weekly report of vessels having a low remaining scallop IFQ quota.

VMSNEScalPrelandAdj.exe – Alerts to scallop pre-landing report adjustments, indicating that a vessel operator has submitted a change to a prior pre-landing report.

29-Jun-2020 PHC – Page 5

VMSNESftwFmt.exe – Daily report showing the software version number for the most-recent declaration or eForm submitted by each vessel. Used to identify vessel owners who need a VMS software update reminder.

VMSNESilentVoyage.exe – Weekly report of VTRs that identify trips which occurred on dates when no VMS position reports were received from the named Northeast VMS vessel.

VMSNETripAlert.exe – Provides special alerts for IVR-declared trips, as requested by certain stakeholders.

VMSNETripDeclareDB.exe – Program to aggregate Northeast VMS trip declarations from various sources, such as vTrack and IVR.

VMSNETripEmailRpt.exe – Provides a daily report of automated Topsail trip notification emails sent to vessels.

VMSNETripExemption.exe – Provides notification of Northeast VMS eForms with specific exemption information selected. Runs every half hour.

VMSNETripRpt.exe – Provides a summary report of recent Northeast VMS eForms received.

VMSNEUndeclaredTrip.exe – Provides daily and monthly reports of Northeast VMS trips that were not declared beforehand.

VMSNEValidOperator.exe – Nightly program that downloads Operator Permit information from GARFO for use by Topsail processes.

VMSNEVendorMigration.exe – Temporary daily report of vessels that change from obsolete Omnitracs VMS units to approved VMS units.

VMSNEVesDisparity.exe – Weekly report of selected vessel information that differs between the vTrack and GARFO Permits database.

VMSNEVesPopulation.exe – Produces a daily summary of Northeast vessel information, and provides a weekly vessel population report.

VMSNEVesselInfo.exe – Nightly program that combines vTrack and GARFO vessel information for use in Topsail reports.

VMSNEVesselTrips – Program running every half hour to generate trip start and end notification emails.

VMSNEVesselUpdate.exe – Program running every half hour to update current vessel information to Topsail.

VMSOpenTripsRpt.exe – Daily report to alert to open trips in vTrack, instances where a new trip is begun before a prior trip ended.

VMSORACheck.exe – Alert to upcoming expiration of passwords to NOAA Oracle databases used by Topsail. Runs daily.

VMSProcChk.exe –Half-hourly process that terminates stalled Topsail-related processes.

29-Jun-2020 PHC – Page 6

VMSPwrDwnList.exe – Daily report of vessels on authorized powered-down status.

VMSReadGmail.exe – Runs every half hour to check for selected new email arrivals in the OLE Drop Box.

VMSSendEmail – Program to send automated email notifications from Topsail, using the nmfs.ole.ne@noaa.gov email account. Runs every five minutes. This program manages the rate at which emails are sent, to stay within the email server quotas.

VMSSENonReporting.exe – Produces a report every four hours identifying Southeast VMS vessels that have met a non-reporting criteria. This is a SED-requested report.

VMSSizeChk.exe – Daily program to alert the Topsail programmer of disk space and file size issues.

VMSSupportLog.exe – Hourly report of selected new vTrack support log entries.

VMSTopsailTest64.exe – Half-hourly test of critical Topsail resources. Alerts to resource failures and halts Topsail processing if specific conditions occur.

VMSVES.exe – Nightly program to update a cache of vessel information for Topsail processes.

Topsail Program Libraries

Some programming written for Topsail is reused among application programs. The work is combined into custom Dynamic Link Libraries, DLLs, used by the Topsail programs. The custom Topsail DLLs are:

CLSFolder.dll – An object class for navigating files and folders.

VMSCommon.dll – A large collection of resources and functions shared among Topsail programs.

Written and maintained in the Visual Basic language, and frequently added to as Topsail services change. This library currently has more than 1700 small functions and subroutines, totaling 33,890 SLOC. All Topsail programs depend upon this library for configuration and service routines.

VMSGIS.dll – Custom routines for spatial location. Used for programs that must make many spatial comparisons. Also includes port locations derived from VMS and USGS data sources.

VMSPWD.dll – Custom programming to determine Northeast vessel VMS reporting requirements. Rules for VMS reporting are encapsulated in this library so that they won’t need updating across Topsail applications.

VMSVES.dll – Custom programming to supply Northeast VMS vessel information. Used by heavy-processing programs to avoid having to make repeated vessel information queries to the vTrack database.

Shared Resources

29-Jun-2020 PHC – Page 7

In addition to producing report and alert documents, Topsail processes also makes selected information available to GARFO through shared views in the Topsail database.

NEVMS.DCL_ALL_CODES_V – List of all currently-valid VMS declaration codes.

NEVMS.DCL_TRANSITION_PAIRS_V – List of all VMS declaration codes currently valid for re-declaration while at sea.

NEVMS.FMC_POS_NED_V – View into the vTrack database, giving the most-recent position for each Northeast VMS vessel. Implemented to correct issues with the similar view that is implemented in the vTrack database.

NEVMS.NEVMS_IVR_RCMF_CODES_V – Provides the GARFO IVR system with the current list of VMS declaration codes that are valid for re-confirmation through IVR. (Note this is NOT the same as the transition pairs re-declaration list.)

NEVMS.VMS_CANADIAN_TRANSSHIPMENT_V – Notification of required messages received from Canadian transshipment vessels in U.S. waters.

NEVMS.NAFO_SENT_V – Messages transmitted from OLE to NAFO through a dedicated vTrack interface, combined with vessel and trip information from a Microsoft Access database manually maintained on

NEDSHARE.

NEVMS.VMS_NAFO_TRIPS_V - Information about NAFO vessel trips.

NEVMS.VMS_NAFO_VESSELS_V – NAFO vessel information.

NEVMS.VMS_PWRDWN_INFO_V – Provides GARFO with Northeast VMS vessel power down LOE information.

NEVMS.VMS_VESSEL_STATUS_V – Northeast VMS vessel reporting status, used for permit renewals.

Depends upon the custom NEVMS.PWRDWN PL/SQL routines, which are also shared with GARFO.

NEVMS.VMS_TEST_UNITS_V – Identification of all known VMS test units, so that GARFO can keep test data from contaminating production data.

NEVMS.VMS_VESSEL_INFO_SUMMARY_V – Information about current Northeast VMS vessels. Used by GARFO when preparing bulletin mailing lists.

Ad Hoc Tasks

Topsail data and programming tools are also used by the Topsail developer to perform ad hoc tasks, provide data extractions, and produce special reports requested by OLE and other authorized stakeholders. In calendar year 2019, the Topsail developer carried out 99 such tasks, each of which was assigned to the Topsail developer because they were thought to be too large, complex or time-consuming to be carried out using vTrack, GARFO’s FishTank, or other tools used by non-programmer members of the VMS team. While not directly a part of maintaining the Topsail system, most of those

29-Jun-2020 PHC – Page 8 tasks made use of resources that were developed and maintained as part of Topsail. A few example ad hoc tasks from 2019 were:

VMS2079 – Extract of NAFO data from the prior year, requested by a GARFO analyst for review and reconciliation.

VMS2082 – Reconciliation of Northeast VMS vessel sector memberships in vTrack against those found in GARFO databases.

VMS2089 – Investigation of a specific vessel’s position report status and latency in comparison with vessels operating similar equipment in specified areas.

VMS2091 – Extraction of vTrack data and production of GIS shapefiles for a report by Carl Lemire on changes in vessel activities following shipwreck avoidance advisories.

VMS2092 – USCG request for many years of multispecies catch report data, going back prior to when the data was directly available to OLE.

VMS2106 – Request from SA Eusebio for historical trip data regarding a specific vessel. Data appeared to be missing from old GARFO sources, but the Topsail developer was asked to “deep dive” into the data sources to confirm whether NMFS had actually received the data. Some of the missing data was found.

VMS2110 – Assisted with extracting and preparing VMS data requested by BOEM for analysis of a wind farm proposal. Was later asked to investigate apparent anomalies found in the proposal, and discovered that the analysis by the proposing entity required revision.

VMS2112 – Worked to identify missing Northeast VMS eForms information, and verify resubmission by the vendor. Also had to identify unintended duplication submissions that then required correction.

VMS2119 – Extracted and prepared data for Carl Lemire to address an IS-1 request from SA Nickerson about landings at specific ports.

VMS2132 – Investigated discrepancy between VMS trip data and AMS trip data for a specific vessel. This is an example of a frequent task, where data-viewing tools available to VMS staff don’t show sufficient behind-the-scenes database information to explain discrepancies.

VMS2134 – For SA Eusebio, performed a search across several years of data to find VMS forms and VTRs bearing specific operator permit numbers.

VMS2140 – In coordination with Carl Lemire, provided scallop pre-landing compliance data, a task that required complex trip-specific information across hundreds of trips in order to determine compliance.

VMS2148 – For Carl Lemire, extracted dealer landings data for a specified set of dealers.

Software Upgrades

NMFS technology upgrades occur occasionally, and often require that a significant amount of developer time be dedicated to keeping Topsail services running while making required system changes.

29-Jun-2020 PHC – Page 9

OLE Drop Box

The Topsail developer also maintains email management rules within the OLE Drop Box (nmfs.ole.ne@noaa.gov), and enables delegate accounts and two-part-authentication rules.

Northeast VMS Unit Upgrades

The Topsail developer works with the Northeast VMS team to encode fishing declaration rules to assist with generation of fishing declaration lists and transition pairs. Helps also with validation and testing of new vendor software, and coordinating changes with GARFO software developers.

End.