PWS_Attachment_1_Daily_Operations_Guide_6_27_2016.docx

NRM Infrastructure Technology Support Services08 Fall

PWS Attachment 1

NRM Operations Guide

Morning/AM Checks Procedure

Revision List:

Contents

Background

This document lists the morning checks that are conducted by the primary DBA each morning. The primary DBA should conduct morning checks on the production server first. Once the production morning checks are complete, send an email to the DBA pdl mailing list with the status (either production is up, or list the issue(s) and expected time of resolution). If there are any production issues not resolved before 7:30 AM Eastern Time, then you must call DBA Lead so that notification can be sent out to the NRM community.

Next, conduct the morning checks on the other environments (nrmdev, nrmtest, trainA, NG Test and NG Dev. You can send a single email to the DBA pdl with the status on the other environments. Be sure to state if the environment is up, or list any issue(s) and expected time of resolution.

NRM dashboard URLs

Location of the NRM Password File on Compass https://ems-team.usda.gov/sites/fs-nfs-nrm-nrms/ops/OPS%20Security%20Library/Forms/AllItems.aspx?RootFolder=/sites/fs-nfs-nrm-nrms/ops/OPS%20Security%20Library/System%20Access%20Management&FolderCTID=0x0120005D1D232523659E4C82C985A099278433&View=%7bEE76D5DF-B500-468E-A211-1F7FD1FF503B

1. NRM Dashboard Checks Displaying the NRM dashboard

The primary DBA logs into NRM application for each instance and verify that we are able to display and/or execute forms, reports, user views, etc. successfully. We use the instructor20 eAuth account for all instances. Be sure to complete all checks on the production environment prior first before conducting the morning checks on NRMDEV, NRMTEST, NG Dev and NG Test.

Click on one of the links in the Database URLs section

When main page is displayed, click on the Default NRM dashboard link on the top left side of the page

Click on the “I agree” button on the USDA eAuth page

Log in with the instructor20 account and password Note: You have 3 failed attempts to login before the instructor20 account is locked.

Click on the button with the text ‘Log in using MY eAuth username/password’

The NRM dashboard should display in 10-15 seconds

Click on one of the links in the Database URLs section

When main page is displayed, click on the Default CDW dashboard link on the top left side of the page

Click on the “I agree” button on the USDA eAuth page

Log in with the instructor20 account and password

Note: You have 3 failed attempts to login before the instructor20 account is locked.

The CDW dashboard should display in 10-15 seconds

Topics -> CDW -> Almanac Reports-> bridges -> FY03 Bridge Almanac Reports Make sure that the report displays

2. Checks sent via email

There are several checks that are conducted at the server level or database level. Below is a list of checks that are automated. You should receive email messages for the below checks. Emails will start coming at 6:35 AM ET from all Unix Servers dochecks check

Everyday DBA’s get a “dochecks” email notification from all the nodes for all the instances and the DBA reviews each notification for any issues. These emails originate from the Oracle or FS Oracle User account, and have Dochecks vx.x in the subject line (x.x is the latest version of the script). Depending on the checks performed, for any issues, the DBA logs into the system to further review and resolves the issues accordingly. Below are the checks that are performed by the “Dochecks” on each instance:

3. RSYNC Job Check The RSYNC Job copies application code from PROD Mid Tier Servers to COOP Mid Tier Servers. The job sends a confirmation status email at 7 AM ET. The subject of the email is “rsync job status - Copying files from prod mid-tier to Coop mid-tier”. Please make sure that the RSYNC Job is running without any issues.

4. Daily OS Tape Backup Check Emails

Check the daily tape backup reports from NITC. You should receive email from Aptare Inc at 8 AM ET. Check to ensure that the Exit Code is 0. If any of the exit codes are not 0, then then send an email to the NITC helpdesk (nitcservicedeskDL@ocio.usda.gov) with a CC to NRM security (nrm_security@fs.fed.us) and DBA PDL, asking NITC to open a ticket and fix or explain the errors.

5. Grid Control Oracle 12c

Grid control has various checks in place for warnings and alerts. Check your email frequently for email from Grid Control and take action as appropriate.

https://infradb.fs.usda.gov:7799/em

6. Xymon monitor

Xymon is a monitoring tool that NITC uses to check various real time states of our servers (production and non-production). This monitoring tool will send emails for various warnings and alerts. Check your email frequently for email from xymon and take action as appropriate.

You can also monitor systems live by using this link:

https://xymon.edc.ds1.usda.gov/xymon/FS_INFRA_TIMBER

If you need an account to this system, please contact NITC (nitcservicedeskDL@ocio.usda.gov)

7. Production database backup check

An automated script has been scheduled that shows the status of the last backups on production and sends an email notification of the report to the DBA’s. Email is sent twice a day (07:00 and 15:00 ET) from the Oracle user, with a subject of Database RMAN backup checks vx.x (where x.x is the latest version of the script). Please check the email to make sure RMAN backup is completed successfully.

8. FMMI Section Needed

Special attention should be given to FMMI Jobs during AM checks.

9. Data Guard – archive log processing on COOP Automatic Check Emails The System Automatically checks whether PROD is in sync with Data Guard Databases. A notification will come at 7:01 ET for IWEBCDBCOOP and 7:06 AM ET for CDWCDBCOOP. Please go through these emails and make sure all the logs are applied to COOP. The subject of this email is “iwebcdbcoop is in sync” and “cdwcdbcoop is in sync”.

Checks from within Grid Control There are checks in the Grid Control to ensure that the data guard is shipping the archive logs and applying them to COOP. Grid Control sends e-mails when it breaks and cleared. If you notice it is out of sync wait for 1 hr or so before you start investigating (because DataGuard will try to sync automatically). If you need assistance with investigation why it is broken, contact Ravi Ponneganti.

Data Guard – manually checking

You can manually check the status of archive log processing by logging into the infradb.fs.usda.gov server with your personal account, and running the following commands.

sudo –u oracle –i cd ./.profile.iwecdb sqlplus / as sysdba select sequence#, applied from v$archived_log a where sequence# > (select (max(sequence#) - 50) from v$archived_log) and name = 'iwebcdbcoop' order by a.sequence#;

cd ./.profile.cdwcdb sqlplus / as sysdba select sequence#, applied from v$archived_log a where sequence# > (select (max(sequence#) - 50) from v$archived_log) and name = 'cdwcdbcoop' order by a.sequence#;

The applied column should be “YES” if the archive logs are currently being processed in the COOP environment.

10. Windows Servers - Air, NVUM, and WCATT

Click on the links below with IE, it will log you in to the air website. If the server is not up, send a note to NITC referring to the IP’s of the servers.

http://nrmwin.fs.usda.gov/nfs/nrm/air/ http://nrmwintst.fs.usda.gov/nfs/nrm/air/ http://nrmwindev.fs.usda.gov/nfs/nrm/air/

1 | Page image1.png