Attachment 7 Topsail Programmers Guide.pdf

Programmer’s Guide to Topsail

20-Jul-2020 PHC.

The Topsail computer provides website reports and email notifications for the Northeast IS/VMS team. This document gives information for supporting the operation and programming for Topsail.

Table of Contents

● Overview – Introduction to Topsail

● Useful Information.

● Topsail Application Programs.

● Topsail Database Tables and Views – List of tables and views within Topsail’s Oracle database.

● Software Installed to Support Topsail – Third-party software used in Topsail operations.

● VMSCommon.mdb Settings – Global Topsail settings.

● Contact Information – Who to call.

● Topsail Backup – Where backup files are kept.

● Visual Basic Programs.

● VMSCommon Files.

● Topsail ODBC DSN configuration.

● Topsail OleDb connections.

● Chilkat .NET Mail Components

● MDB files.

● Daily Topsail Monitoring.

● Warnings and Advisories

● Shutting down and Restarting Topsail Services

● Appendix A - passwords. (Separate document, for security.)

Overview

HQTopsailP1 is a Microsoft Windows server located in Seattle, WA. Application programs for Topsail services run on HQTopsailP1 at scheduled intervals.

Programs for Topsail are written in Visual Basic 2019 and C# 2019, using .NET Framework 4.7.2.

More than twenty programs are scheduled to run periodically on Topsail. Some Topsail programs run as frequently as every 5 minutes, some run daily, some run only once weekly or monthly, and some are used only periodically for specialized tasks.

Topsail produces reports to send as email notifications to Northeast VMS staff and to VMS vessels at sea, and shares selected data to affiliate organizations within NOAA. Topsail reports are also written to the OLE NEDSHARE server.

Data for Topsail programs come from NOAA enterprise databases. Some of that data is available directly by Topsail programs and some is available through Topsail’s own Oracle database via database links. Some data is cached within Topsail’s Oracle database, and some of it simply passes through the database, conditioned by stored Oracle Views and database links.

Many Topsail programs also maintain small individual databases of processed data and program state information in Microsoft Access MDB files.

Useful Information

Note that all IP addresses is subject to change by the OCIO network team. Verify IP addresses with them. The IP addresses given here should not be assumed to be correct.

All Topsail processes are run under a Microsoft Windows user service account:

Computer: HQTopsailP1 (IP: 161.55.129.146)

Account: TopsailService

Password: [See Appendix A]

The Topsail service email account is:

Account: nmfs.ole.nevms@noaa.gov

Password: [See Appendix A]

The Topsail Oracle database is maintained by the OCIO at a location of their choice:

IP 161.55.25.123:1521

Schema: NEVMS

Account: NEVMS

Password: [See Appendix A]

The NERO Oracle database account for Topsail is:

IP 155.206.47.19:1522

Account: topsail

Password: [See Appendix A]

All Topsail programming and related files on HQTopsailP1 are located under:

HQTopsailP1 at E:\Topsail\...

All current source code for Topsail programs is kept on NOAA workstation OLENED4171913 (Jibsail) at:

C:\Users\Philip.Corcoran\Documents\NOAA\Source\NERO\Programs64

Back-up copies of Topsail program source code are periodically made to NOAA workstation OLENED4171913 (Jibsail) at:

I:\Jibsail\Source\NERO\Programs64

Some Topsail programs use MDB data files, many of which are protected by password:

Password: [See Appendix A]

All Topsail programs get their default configuration settings from the VMS_SETTINGS table of the E:\Topsail\Databases\VMSCommon.mdb MDB file. This is a very important file and most system changes will require parameter changes in the VMSCommon.mdb file. Whenever a change is made by a NOAA IT organization, the Topsail programmer should review the VMS_SETTINGS table to see if related parameter changes are also needed there.

All VMSCommon.mdb settings can be over-written for individual programs by defining them in the program-specific INI configuration files. The Topsail programmer must look into each program source code to determine what settings that program uses.

Kill Topsail Processing

In emergencies, all Topsail email notifications can be halted by setting TOPSAIL_RUN to FALSE in the VMSCommon.mdb database. That can be accomplished by opening a DOS command window on the HQTopsailP1 computer and typing:

topsail stop

Outgoing trip notification emails can also be halted remotely, by setting the Northeast trip generator persistence counter to 13 in the vTrack GUI application.

Topsail Program Directory

Topsail programs are written in the Microsoft Visual Basic and C# languages. Documentation for each program appears in the individual program source code.

Most Topsail programs depend upon settings in the VMSCommon.mdb file (described elsewhere). Those settings may also be over-written by entries in INI files specific to each program. Many of the programs maintain local data and state information in program-specific MDB files. In all cases the associated files will have the same root names as the programs:

<prog>.exe, <prog>.ini, <prog>.mdb.

For example, the VMSNEAuxTrans.exe first reads default settings from the VMSCommon.MDB file, and then may overwrite those parameters with settings in the VMSNEAuxTrans.ini file. The VMSNEAuxTrans.exe.ini file also directs the program to use file VMSNEAuxTrans.mdb for its operations.

Each Topsail program’s MDB file also contains a SETTINGS table. Some Topsail programs also accept command-line parameters. The order of precedence of Topsail program parameters is, with the lower-number taking precedence, is:

1. Command line.

2. Program INI file.

3. Program MDB SETTINGS.

4. VMSCommon MDB VMS_SETTINGS.

Topsail Programs

As of 29-Jun-2020, names and descriptions of Topsail report-generating programs have been moved to the “Topsail Overview” document. Please see that document.

Topsail Oracle Database

The Topsail Oracle database schema is named NEVMS, not “Topsail”.

The Topsail Oracle database user account is also named NEVMS.

The Topsail Oracle database password requires changing at least every six months.

Each Topsail Oracle database table includes instructive comments that are visible in

Oracle SQL Developer software.

Document “Data Extract Sources.docx” includes descriptions of active Topsail Oracle database tables.

Software Installed to Support Topsail Operations and Development

Install files are kept on Topsail in C:\Inetput\wwwroot\Programs\Downloads

Chilkat – SMTP email sending DLL.

Microsoft Access run-time library.

VMSCommon.mdb Settings

The Jet MDB 2003 database file located on Topsail at E:\Topsail\NEVMS\VMSCommon.mdb contains values available to all Topsail programs. Changing values in that database may affect all Topsail programs. Note however that each Topsail program may have its own INI file.

Settings within INI files will override global settings.

The VMSCommon.mdb database requires a password to open. The password is [See Appendix A]. Since HQTopsailP1 has only a run-time version of Microsoft Access installed, it is necessary to open and edit the VMSCommon.mdb database by remote file server connection. A logon to the Topsail programmer account on Jibsail is used for that operation.

Inside VMSCommon.mdb is a table named VMS_SETTINGS that contains the fields KEYWORD and CONTENTS. Values in the KEYWORD field identify individual settings, as given in the

CONTENTS field. A DESCRIPTION field describes the purpose of each setting.

Contact Information

William Semrau – Northeast Investigative Support Program Manager.

william.semrau@noaa.gov, 978-281-9151 (NOAA).

Louise Connolly – Desktop Support. louise.connoly@noaa.gov, 978-281-9334 (NOAA).

OLE Helpdesk – Data Center Support. OLE.Helpdesk@noaa.gov, 8-888-219-9228 (NOAA).

C# and Visual Basic Programs

The C# and Visual Basic programs written for Topsail are maintained using Visual Studio 2019 software. Each program is created as a .NET Framework 4.7.2 Console application and then changed to a Windows Application in the applications Properties. The Build configuration should be “All Configurations”, “Any CPU” with “Prefer 32-bit,” “Allow unsafe code,” and “Optimize code” all unchecked.

Topsail ODCB DSN configuration

Topsail uses Data Source Name (DSN) files to connect to enterprise NOAA Oracle databases.

Follow Oracle documentation for setting up TNS names files, and Microsoft documentation for associating DSN configurations.

Topsail OleDb connections

Topsail applications use Microsoft OleDb to connect to application MDB databases. Per Department of Commerce requirements every Topsail MDB file is protected by a password. See Appendix A for the MDB passwords.

Microsoft Access MDB files can hold up to 2 GB of data, but experience has shown that some update operations fail if MDB data storage exceeds 1 GB. Topsail MDB databases should be kept to under 1 GB in size.

Microsoft Access MDB files used by Topsail should never be opened in read-only or exclusive mode since that can cause conflict with concurrent applications.

Chilkat .NET Mail Component

Topsail programs use the Chilkat .NET Mail Component to send and manage automated Topsail emails. Chilkat is used because it interfaces well with the Google Gmail account used by the OLE Drop Box. OLE purchased a perpetual single-developer license for this component. Chilkat often urges the purchase of maintenance licenses, but that should not be necessary.

Note that McAfee anti-virus software installed on NOAA computers will block outbound emails unless specially configured. All Topsail programs that send emails are named with a prefix of VMS, so that a vms* rule entered into McAfee will let Topsail email messages pass.

Daily Topsail Monitoring

The following steps are recommended each morning to assure continued proper operation of Topsail services. In usual operations the daily monitoring takes 10 minutes or less.

1. Logon to the Topsail service email account (nmfs.ole.nevms@noaa.gov) and review the overnight reports and investigate any error messages.

2. Review all Topsail application log files for errors, warnings or signs of excessive runtime.

3. Each Topsail application ends with a code of zero if successful, or a non-zero code if failure occurred. The final line of each log file should be checked for the success code of zero.

Warnings and Advisories

Topsail processes have experienced failures due to file access contention. Occasionally some Topsail programs will be found to have ended unexpectedly, leaving incomplete log files. All Topsail applications report any failures encountered, therefore any unexpected termination occurs at the system level. Investigate the Windows system logs for causes.

Advisory to Programmers: Several Topsail programs send notification emails to a user community. There have been instances when source data corruption caused greatly excessive numbers of emails to be sent, sometimes at considerable cost and complaint. Throughout current Topsail programming will be found checks to halt excessive email operations.

Programmers who may be tempted to remove those checks as unnecessarily are strongly advised against doing so. Topsail program also includes checks to prevent sending emails to public addresses directly from the development workstation. Again, it is unwise to circumvent such protections.

Shutting Down and Restarting Topsail Services

Automatically scheduled services on the HQTopsailP1 computer are run by the Windows Task Scheduler (WTS). Disabling tasks in the WTS will halt those operations. In many cases the tasks are run as batch files, launched by the runshh program. Commenting out appropriate batch file lines is therefore also an effective way of disabling selected HQTopsailP1 processes. In fact, it is the preferred way, since it allows disabling processes by file serve access in cases where remote logon to HQTopsailP1 itself is not possible.

All Topsail processes are written with self-recovery as a goal. Shutting down Topsail for a reboot requires no special actions other than to shut the Windows Operating System down in an orderly fashion. Topsail processes will act appropriately to Windows OS shutdown signals.

-end-