Reading Library 5 - Legacy SOP_ DSMC Event Management_Redacted.pdf

DSMC Event Management & Network Monitoring Standard Operating Procedure

Version 6.1 Page | iii

Table of Contents

1 Purpose

2 Scope

3 Background

3.1 Monitoring

3.2 Reporting

3.2.1 Events

3.2.2 Informational

3.2.3 Warning

3.2.4 Exception

3.2.5 Alerts

4 Trigger

5 Procedure

5.1 Prerequisites

5.2 Event Notification and Detection

5.2.1 Notifications of Exceptions

5.2.2 Notifications of Warnings

5.3 Event Logging

5.3.1 Exceptions (Including Outages)

5.3.2 Warnings

5.3.3 ASIs

5.4 Event Prioritization

5.4.1 Rapid Response Notification

5.5 DISANet Outage Reporting

5.5 1 Outage Reporting Status Change from P1 to P2

6 DSMC Account Requirements

7 DSMC Monitoring Tools

8 DSMC Monitoring Tool(s) Change Request

9 Roles and Responsibilities

Hist ori ca l R efe ren ce D oc um en t

Version 6.1 Page | iv

9.1 DSMC

9.1.1 DSMC Checks

9.2 Tier II

Appendix A: Roles and Responsibilities

Appendix B: Acronyms

Appendix C: NIPR Monitoring Policies

Appendix D: Related Documents/Links

Appendix E: References

Authorizing Signatures

Hist ori ca l R efe ren ce D oc um

Version 6.1 Page | v

List of Tables Table 1: DSMC Account Requirements Table 2: DSMC Monitoring Tools Table 3: Acronyms Table 4: NIPR Monitoring Policies

List of Figures Figure 1: Priority Matrix from DISA Process 440 Figure 2: Monitoring Request Workflow ca l R efe ren ce D oc um

Version 6.1 Page | 1

1 Purpose

This Standard Operating Procedure (SOP) outlines minimum requirements for Defense Information Systems Agency (DISA) Network (DISANet) Event Management (EM) and Network Monitoring (NM). The intent of this document is to promote consistency in the execution of EM activities and procedures, enabling the DISANet Services Monitoring Center (DSMC) to perform its mission of providing 24x365 network monitoring and reporting services (i.e., to report significant occurrences that affect the overall availability of the DISANet).

2 Scope

This Work Instruction applies to DISANet EM, utilizing the DSMC monitoring tools (e.g., SolarWinds, Netcool, Infovista, Jabber, and Swift Chat). DISANet Enterprise events are reported via monitoring tools, phone calls from site admins, and direct communication with the Global Service Desk (GSD).

SolarWinds events are shown 24x7x365 in the DSMC on NIPR and SIPR wall monitors dedicated for this purpose.

3 Background

3.1 Monitoring

The DSMC monitors numerous devices throughout the DISANet Enterprise, including circuits, switches, and servers on a global scale. The DSMC tracks latency on the DISANet Network and connectivity status for Internet Control Message Protocol (ICMP), Simple Network Management Protocol, and Windows Management Instrumentation.

3.2 Reporting

The DSMC is responsible for reporting outages, potential outage symptoms, and keeping the various tiers and teams updated on the current status of these events. They also provide daily reports on incidents and problems to include status and statistics of the network to senior operations staff. All DSMC activities must be documented within the Master Station Log (MSL) by the DSMC technicians. All reporting and notifications must be verified by two onsite staff members prior to distribution; the reports will include the initials of the staff members that generated and reviewed the notification. Additionally, the DSMC will report Priority 1 incidents via SIPR DCC Enterprise Virtual Watch Dashboard (EVWD). Instructions are available in the Secret booklet that is kept on the resource desk of the DSMC. Once the incident has been entered into the DCC EVWD, the DSMC will contact the DCC Battle Captain at 5-3505 or (301) 225-3505.

Use the following URL to access the MSL:

https://disa.deps.mil/org/SE6/OPSCNT/C/SitePages/MSL.aspx ca l R efe ren ce D oc um

Version 6.1 Page | 2

3.2.1 Events

An event is a change of state that has significance for the management of an Information Technology (IT) service or other configuration item. Events typically require IT operations personnel to take action, and often lead to incidents being logged.

Events fall under three categories: Informational, Warning, and Exception. Each of these categories is detailed in the following subsections.

3.2.2 Informational

The event does not require any major action. They are recorded in the MSL files and maintained for one week. At that point, the log entry can be archived for historical purposes. This type of event is used to check the status of a device or service, to confirm the state of an activity, or to generate statistics (e.g., user login, batch job completed, device power up and number of users logged into an application).

3.2.3 Warning

The event is generated when a device or service is approaching a threshold or Key Performance Indicator. Warnings are intended to notify the group/process/tool in order to take the necessary actions to prevent an exception. A device failure warning should be treated as an exception, because the risk for an incident is present (e.g., server CPU at 90% or collision rate on network increased by 15%).

3.2.4 Exception

A service or device is currently operating below the normal predefined parameters/indicators.

This means that business is impacted and the device or service presents a failure, performance is at a hazardous condition (HAZCON), or loss of functionality (e.g., web server down or CS coverage lost for several sites).

3.2.5 Alerts

An alert is a notification that a threshold has been reached, something has changed, or a failure has occurred. Alerts are generated from monitoring tools for warnings and exceptions, and managed by the EM process.

4 Trigger

This SOP is triggered when an item is categorized as meeting the requirements of an event (see Section 5.2). Hist ori ca l R efe ren ce

D oc um

Version 6.1 Page | 3

5 Procedure

5.1 Prerequisites

Prerequisites include the following categories (detailed in Section 3.2):

1. Events

2. Informational

3. Warning

4. Exception

5. Alerts

5.2 Event Notification and Detection

Any and all alerts will be acted upon 24x7x365.

Potential network events are detected in one of the following ways:

1. DSMC network tools have generated an event notification that cannot be cleared by refreshing the monitoring tool.

2. DSMC is notified about an event by site admins, GSD, Network Operations, Engineering, JRSS, DCC, or Infrastructure.

5.2.1 Notifications of Exceptions

Notifications of exceptions are as follows:

• The alert shows a DISANet site is down.

• The alert shows a DISANet server is down.

• The alert shows a port is down on a server.

• The alert shows a DISANet service is down.

• The alert shows a DISANet circuit is down.

• The alert shows a JRSS stack is down.

5.2.2 N tifications of Warnings

Notifications of warnings are as follows:

• The alert shows a high temperature warning.

• The alert shows low hard drive space on drive C: /.

• The alert shows that a server has been rebooted.

Hist ori ca l R efe ren ce D oc um

Version 6.1 Page | 4

5.3 Event Logging

5.3.1 Exceptions (Including Outages)

If the event is an exception, a Major Incident record shall be logged in the MSL until it is archived for historical purposes. Refer to DISA IT Enterprise Tier II Incident Management SOP, Major Incidents.

5.3.2 Warnings

If the event is a warning, an incident record shall be logged in the MSL until it is archived for historical purposes.

5.3.3 ASIs

If the event is an Authorized Service Interruption (ASI), a record shall be logged in the Incident Management Console of Remedy IT Service Management (ITSM). Any and all ASIs should be communicated to the DSMC prior to the scheduled interruption and immediately following the interruption. They are also added to the DISANet Scheduled Changes at the following link:

https://disa.deps.mil/org/SE6/DEMI/SitePages/Home.aspx

Notifications are also supplied by any higher agency (JMN or DCC) via chat or via SIPR email.

5.4 Event Prioritization

The technician shall identify and generate the priority by selecting the appropriate Urgency and Impact fields. A Priority 1 (P1) includes all VIP incidents; it also includes all Major Incidents.

The Incident Manager shall identify all P1, or critical, incidents immediately and verify that those incidents meet the urgency and impact criteria described in DISA Process 440 and additional prioritization requirements described below.

NOTE: Warnings will be prioritized as Priority 2 (P2).

Hist ori ca l R efe ren ce D oc um

Version 6.1 Page | 5

Figure 1: Priority Matrix from DISA Process 440

5.4.1 Rapid Response Notification

As part of an outage/outbreak related to Network Security affecting any site, the DSMC will notify the DISA Ft. Meade (FMD) SE List SE6 Rapid Response Team distribution list and open the following bridge:

Dial-In Number: 301-909-7350 Dial-In DSN Number: 312-723-7350 Access Code: 34721224

This bridge will allow for uniform communication and coordination of course of action. The DSMC will also call the Rapid Response Point of Contact (POC),

5.5 DISANet Outage Reporting

The DSMC will distribute outage reports after the initial occurrence and then every 4 hours thereafter throughout the resolution of ALL Major Incidents, i.e., Priority 1 (P1). Then, for all Priority 2 (P2) outages, outage reports will be sent every 24 hours until the FINAL resolution (unless directed otherwise by the DSMC Lead or the Program Manager; the Government Watch Officer may also advise).

Hist ori ca l R efe ren ce D oc um

Version 6.1 Page | 6

Once outage is reported, the DSMC shall follow the following process:

1. Create an ITSM BMC remedy ticket. Access using the following URL:

https://webitsm.csd.disa.mil

2. Log the incident in the MSL, found at the following link:

https://disa.deps.mil/org/SE6/OPSCNT/C/SitePages/MSL.aspx

3. Send the outage notification using the “Outage” template from the DISA FMD SE Mailbox SE6B Systems Monitoring Center mailbox.

4. Create a DCC EVWD event on SIPRNet using the DCC EVWD procedure. Once the event is created, the DSMC will contact the DCC Battle Captain at 5-3505 or (301) 225- 3505.

5. Enter outage information into Reason For Outage (RFO) using the following link:

https://disa.deps.mil/org/SE6/OPSCNT/C/Lists/RFO%20Tracker/AllItems.aspx

NOTE: All after-hour P1 or widespread outages that interrupt normal operations must be reported to SE6 and leadership:

1. DSMC Lead

2. On Call Watch Officer

3. Server/Infrastructure Lead (Tier III Lead)

4. Program Manager

Once the outage has been reported to program leadership, a Government POC must be notified. A Watch Bill list is provided by the Government to show who is on call during specific weeks. During the call to the Watch Bill manager DSMC/Tier III support are required to provide all details of the outage(s) and any additional vendor support needed. The On Call list and contact numbers for the Watch Bill Manager are available at the DSMC SharePoint site:

https://disa.deps.mil/org/SE6/OPSCNT/C/default.aspx. Information is stored on both the calendar and the Watch Officer Contact List page (refer to the sidebar menu link).

Please reference the After Hours Outage Notification Process (attached below) for the entire reporting process for outages that occur after hours:

After Hours Outage Notification Process

NOTE: If DSMC is unable to contact the Site Admin or vendor support, a team member must notify:

• Chief, IT Services Division

• Deputy, IT Services Division

• SE6 Duty Operations Manager

Hist ori ca l R ef en ce D oc um

Version 6.1 Page | 7

Attempt contact twice within ten minutes. If the on-call POC from the Watch Bill list does not answer, move down the following list until contact is made:

1. Chief, Server Operations

2. Chief, Infrastructure Operations

3. Deputy, IT Services Division

4. Chief, IT Services Division

NOTE: Current contact numbers for the above POCs are posted in the DSMC and can be located at the Site Admin link https://disa.deps.mil/org/SE6/SE63- CC/Lists/Site%20Admins/AllItems.aspx. If a Government POC cannot be reached, notify the

Program Manager.

The DSMC will send out outage reports within 30 minutes of the initial occurrence and every 4 hours thereafter through resolution for ALL P1 Incidents. The outage- eporting template is as follows:

Incident/Ticket Ref#:

Ops Impact: Outage, Degradation, or HAZCON Server or Service Impacted:

Classification:

Users Impacted:

Sites Impacted:

Buildings/Floors/Rooms Impacted Assigned Group:

Status:

Out Time: [Date] [Zulu Time] Z [Eastern Standard Time] EST In Time:

Duration:

Tier 4/Vendor Support:

Actions Taken:

Before sending notification, approval must be provided from one of the following individuals:

DSMC Lead, Tier III Lead, or Program Manager.

NOTE: In cases of network outages preventing the DSMC from sending emails on NIPR, the outage report will be generated and sent to the same recipients and distro-lists on SIPRNet. If both networks are down, the DSMC will follow the after-hour process and notify leadership and SE6 leadership via telephone.

5.5.1 Outage Reporting Status Change from P1 to P2

Any reporting or findings of an issue/outage affecting the entire site, either small or large, will be categorized as P1. All outages that are downgraded to P2 and placed in monitoring status require updates every 24 hours until the ticket is “Resolved”.

Hist ori ca l R efe ren ce D oc um

Version 6.1 Page | 10

Figure 2: Monitoring Request Workflow ist ca l R efe ren ce

D oc um

Version 6.1 Page | 11

(Note that the figure on the previous page is under revision.)

9 Roles and Responsibilities

The roles and responsibilities of the DSMC and Tier II are detailed in the following subsections.

9.1 DSMC

The roles and responsibilities of the DSMC are listed below:

• Provide DISA IT enterprise monitoring and reporting services.

• Provide 24x7x365 network monitoring and reporting services to perform the mission of the DSMC.

• Monitor IT services, DISANet servers, and network infrastructure devices and reporting to the DSMC Lead when established performance measures, thresholds, and/or tolerances are exceeded, or if deviations from established patterns/measures are detected. The DSMC Manager is then responsible for contacting Government leadership if Government support is needed. Examples of IT services currently monitored by the DSMC include but are not limited to: Directory Services, Job Scheduling (e.g., backups and patch deployment), DNS, DHCP, and VPN, Technical Controls, firewall, routers, switches, servers, server services and systems.

• Record all outages, alerts, interruptions, and ASIs in the MSL.

• Operate and administer the monitoring functions/capabilities for the DISA IT Enterprise

Network Management Systems (e g , SolarWinds, and Swift for the GSD, etc.).

• Report documentation to facilitate technical response, and track and report, in accordance with the schedule directed by the DSMC Manager or Government Watch Officer, the status of efforts until final resolution.

• Provide updates to the MSL:

(https://disa.deps mil/org/SE6/OPSCNT/C/Lists/MSL/AllItems.aspx#ServerFilter=FilterF ield1=ID-FilterValue1=5128-FilterOp1=Geq-OverrideScope=RecursiveAll- FallbackLimit=5128-ProcessQStringToCAML=1)

• Conduct technical analysis, as directed by the Government, to develop threshold criteria and enhance the processes and tools supporting the DISA IT Enterprise Event Management SOP.

• Monitor, classify, notify/communicate, and document/record events.

• Verify whether a network element outage was/is an unplanned outage.

• Identify an outage as a “Failure” or an “ASI” and record it in ITSM Remedy and MSL if a record for the outage does not already exist.

• Create an incident record if an event notification indicates warning or informational notification (e.g., low hard drive space).

Hist ori ca l R efe ren ce D oc um

Version 6.1 Page | 12

• Verify and consult with the Tier I GSD manager to collaborate on events that may have already triggered incidents or problem records. This is so that Tier I can correlate those incidents to problems in ITSM Remedy (e.g., the GSD coordinates with the DSMC when they see a surge in incoming tickets for one isolated event that may be caused by and related to a Major Incident).

• Support DISANet Event Management 24x7x365.

• Provide statistics and network performance and daily operational status reports when required.

• Create problem or incident records, in the detail allowable in the unclassified instantiation of ITSM Remedy, and escalating to Tier II.

• Record all DISANet incidents using the MSL.

• Maintain the MSL.

• Review all MSL entries from previous shift.

• Update SharePoint for any new contact information on the POC list.

• Create/update a DCC EVWD entry and notify the Battle Captain at 5-3505 or (301) 225-

3505.

9.1.1 DSMC Checks

DSMC is responsible for conducting the following morning check at 0400 each day, and post any concerns or issues in the DSMC MSL and pass down report. The purpose of this report is to track any issues that may affect normal operat ons during core hours:

SE6 DISANet Top Issues NEW.docx

• Check all monitoring tools.

o DSMC technicians are required to confirm all monitoring tools are logged in and updated prio to core hours shift. Swift is to be logged in at all times to communicate with the different Tiers in the “DSMC_Monitoring_Conference_Room”.

• Check ITSM Remedy website.

o Any outages or site issues will be reported to the appropriate team members prior to core hours and the GSD must be notified.

• Check and make sure all Domain Controllers (DCs) for HQS are up and operational, and that DoD users are on.

o This is the DC that provides roles such as security authentication requests (e.g., logging in, checking permissions, DHCP, DNS, and Active Directory); they need to be checked to make sure that DoD Visitors Services is running and that there are no issues with DNS replication. Thresholds have been added into SolarWinds to alert the Tier II Server Team when DoD Visitors Services stop.

Hist ori ca l R efe ren ce D oc um

Version 6.1 Page | 13

• DSMC runs disk space reports daily at 0400, on both NIPR and SIPR, on all DCs, and all DISANet Enterprise servers; these reports will be posted on the DSMC MSL.

o Tier III Swing/Mid-shift starts the cleanup from the 1600 report. The disk space report is ran again at 0400; any remaining issues the DSMC will create a ticket and assign to Tier II-Server team.

• DSMC Afternoon Checks.

DSMC is responsible for conducting the following afternoon check at 1600 each day, for inclusion in the 1700 “Top Issues” report. The purpose of this report is to track any issues that may affect normal operations during core hours the following day:

• The HBSS will email or call the DSMC only if there are any changes to rules and exceptions that have been made that could cause potential outages or issues to the network, server, or user workstation.

o Any issues or concerns provided to the DSMC will post in the DSMC MSL and provide in the pass down.

• DSMC is to run disk space reports daily at 1600, on both NIPR and SIPR, on all Domain Controllers- DC and all DISANet enterprise servers (these reports will be posted on the DSMC MSL). For each server that has 10% or less disk space, the DSMC will post in the DSMC MSL for Tier III swing shift to clean up and generate individual ITSM tickets to Tier II-Server OPS if any cleans up are remaining or incomplete using the attached instructions:

SCCM Disk Space Report on DCs.docx

9.2 Tier II

After-hours, weekend, and Government holiday Tier II support services are provided to all DISA IT Enterprise Infrastructure Components to improve performance and enhance the capabilities of DISA’s IT Enterprise Service Desk. All processes are aligned with the Defense Enterprise Service Management Framework.

The roles and responsibilities of Tier II are listed below:

• Troubleshoot and document all records escalated from the DSMC that are event and network related, which may require onsite presence, to involve fault isolation when a known solution is not available. These include: network systems, routers, switches, firewalls, servers, IDS, DNS, DHCP, etc.

• Track any provided detail reports on the events discovered by DSMC, as well as incidents created by DSMC/Tier I that are impacting the network.

• Acknowledge any events escalated to Tier II from the DSMC.

ist ori ca l R efe ren ce D oc um

Version 6.1 Page | 14

• Provide onsite support for network and event-related issues.

• Resolve certain event and network incidents that have known solutions for systems and/or services for which they have permissions.

• Support DISANet 24x7x365.

• Collaborate with Problem Management on all PM records by direction of the PM Team.

• Responsible for providing the DSMC with updates on any work or actions being taken to address outages, alerts, ASIs, etc.

• Update tickets in accordance with target dates.

ist ca l R efe ren ce D oc um

Version 6.1 Page | 19

Appendix D: Related Documents/Links

1. Master Station Log:

https://disa.deps.mil/org/SE6/OPSCNT/C/SitePages/MSL.aspx

2. RFO:

https://disa.deps.mil/org/SE6/OPSCNT/C/SitePages/RFO%20Tracker.aspx

3. Watch Officer:

https://disa.deps.mil/org/SE6/OPSCNT/C/default.aspx

4. Top Issues:

https://disa.deps.mil/org/SE6/OPSCNT/C/SitePages/Documents.aspx?RootFolder=%2Fo rg%2FSE6%2FOPSCNT%2FC%2FDSMC%20DocsSOP%2FTop%20 ssues%20Report %20Guide&FolderCTID=0x0120009CF03DF2DBAB6D46BC045BE6A2D3C42F&Vie w=%7BBBDA7E67%2DE5DD%2D4C70%2D91A2%2D627C7F84E535%7D

5. Change Calendar:

https://disa.deps.mil/org/SE6/DEMI/SitePages/Home aspx

6. ITSM Remedy:

https://webitsm.csd.disa.mil

7. Admin Contact List:

https://disa.deps.mil/org/SE6/SE63-CC/Lists/Site%20Admins/AllItems.aspx.

8. DSMC SharePoint site:

https://disa.deps.mil/org/SE6/OPSCNT/C/default.aspx.

ca l R efe ren ce D oc um

Version 6.1 Page | 20

Appendix E: References

1. Department of Defense (DoD). DoD Instruction No. 1253, Committee on National Security Systems (CNSS), Security Categorization and Control Selection for National Security Systems. March 2014.

2. National Institutes of Standards and Technology (NIST). NIST Special Publication (SP) 800-53 Security and Privacy Controls for Federal Information Systems and Organizations, Revision 4. Joint Task Force Initiative. April 2013.

3. National Institutes of Standards and Technology (NIST). NIST Special Publicat on (SP) 800-37 Guide for Applying the Risk Management Framework to Federal Information Systems: A Security Life Cycle Approach, Revision 1. Joint Task Force Initiative.

February 2010.

ca l R efe ren ce D oc um