SOP100-55-01 GNSC TMS

DISA CONUS STANDARD OPERATING PROCEDURE (SOP) 100-55-01 (Revision 1)

DISA CONUS

Global Network-Operations Support Center

Standard Operations Procedures

For

Trouble Management System

“ Remedy ” 31 January 2008 Table of Contents:

Section Title Page Number

Purpose

Applicability References Background General Responsibilities 6.1 Systems Control Officer 6.2 Senior Network Operations Center Representative/Technician 6.3 Network Operations Center Technician 6.4 Tactical Mission Director Procedures 7.1 Main Tm Action Request 7.1.1 Action Request Number 7.1.2 Status 7.1.2.1 Initiated 7.1.2.2 In Progress 7.1.2.4 Referred 7.1.2.5 Deferred 7.1.2.6 Pending

7.1.2.6.1 Less than 72 Hours

7.1.2.6.2 Greater than 72 Hours

7.1.2.7 Monitor 7.1.2.8 Fixed 7.1.2.9 Fixed HD 7.1.2.10 Canceled 7.1.2.11 Closed 7.1.3 Higher Level Outage 7.1.4 Submitter 7.1.5 Record Classification 7.1.6 Spell Check 7.2 Contact Info Tab 7.2.2 Role

7.2.3 Remarks 7.3 Action Info Tab 7.3.1 Action Description 7.3.1.1 Special Identifier 7.3.1.2 Network Identifier 7.3.1.3 Bandwidth 7.3.1.4 Location 7.3.1.5 Event Description 7.3.1.6 Additional Information 7.3.1.7 Examples of Action Description Field 7.3.2 Impacted Communities 7.3.3 Operational Impact Description 7.3.3.1 Significance Identifier

Title Page Number

7.3.3.2 Organization

7.3.3.3 Location 7.3.3.4 Describe Impact 7.3.3.5 Customer Complaint 7.3.3.6 Network Operations Command and Control Centers Entries 7.3.3.7 Examples of Operational Impact Description Field 7.4.4 Status or Resolution Summary 7.4.5 Zulu Monitor Status Start 7.4.6 Zulu Monitor Status Stop 7.4.7 Validation and Element Fields 7.4.7.1 Validation Source 7.4.7.2 Element ID Type 7.4.7.3 Element ID 7.4.8 Action Type 7.4.8.1 Authorized Service Interruption 7.4.8.2 Hazardous Condition 7.4.8.3 Information

7.4.8.4 Unscheduled Outage 7.4.8.5 Degradation 7.4.9 Assigned Priority 7.4.9.1 One (1) 7.4.9.2 Two (2) 7.4.9.3 Three (3) 7.4.9.4 Four (4) 7.4.9.5 Five (5) 7.4.10 Service Suite 7.4.11 Submitter Location 7.4.11.1 Submitter Site 7.3.11.2 Supporting Site 1 7.3.11.3 Supporting Site 2 7.3.11.4 Assigned Group 7.3.11.5 Assigned Technician 7.5 Fault Summary Tab 7.5.1 Zulu Fault Date Time Out 7.5.2 Zulu Fault Date Time In 7.5.3 Service 4 7.5.4 Fault Duration Hours 7.5.5 Location of Problem 7.5.6 Fault Category 7.5.7

RFO 1

7.5.8

RFO 2

7.5.9 Vendor 7.5.10 Vendor Ticket Number 7.5.11 RFO Remarks 7.5.12 Multiple RFO 7.6 Reporting Tab 7.6.1 GNC Reportable

Title Page Number

7.6.2 Local Brief 7.6.3 Local Priority 7.6.4

COMSPOT

7.6.5 Report Remarks 7.6.5.1 Condition of Event 7.6.5.2 Time/Date of Status 7.6.5.3 Course of Action or Resolution 7.6.5.4 Additional Information 7.6.5.5 Examples of Report Remarks Field 7.6.6 Action Log 7.6.7 Attachments

Disposition

Web Brief Examples

DISA CONUS Standard Operating Procedures (SOP) 31 January 2008 100-55-01 (Revision 1)

10 Sep 08 - This SOP was reviewed for accuracy by RN3 and DCCC. References were updated. No other changes apply.

OPERATIONS

Global Network-Operations Support Center (GNSC) Trouble Management System (TMS)

1. Purpose. This SOP provides guidance to Tier 0 – the DISN Customer Call Center (DCCC), Tier I - the DISA CONUS GNSC Systems Control Officers (SCO), Tactical Mission Director (TMD), the various Network Operation Centers (NOC), GNSC NetDefense, DISN Video Services (DVS), Defense Message System (DMS), and Tier II - DISA CONUS Engineering for documenting information in TMS tickets. Information exchange is critical in accomplishing the GNSC’s daily operational mission. This document will better facilitate the consistency and accuracy of vital network information; which will in turn assist in providing CONUS Global Information Grid (GIG) situational awareness, Network Operations (NetOps) continuity, and better communication flow within the GNSC and with other organizations. TMS is a center of gravity for the GNSC, providing essential documentation of present and past network outages, along with workload statistics. The consistency and accuracy of the information placed in TMS is what this document will address. Information within TMS tickets should answer the fundamental questions of Who, What, When, Where, Why and How. The guidelines and formats outlined in this SOP will be strictly followed, with the exception of the unclassified DMS which will follow reference 3.8.

This revised SOP cancels and supersedes previous SOP 100-55-01 dated 23 Jan 2007.

2. Applicability. This SOP applies to all personnel under the Operational Control (OPCON) of the GNSC and those that support the DISA CONUS GNSC using TMS.

3. References.

3.1. TMS User’s Guide, Version 6.0.1, 10 May 2006

3.2. DISA CONUS SOP100-55-04 CONUS GNSC Operations and DCCC Interactions dated 07 Nov 08 (This SOP replaced SOP06-08)

3.3. DISA CONUS SOP 100-55-02 (Rev 1), GNSC Network Escalation Process and Responsibilities Between Tier I and Tier II, 10 Sep 2008

3.4. NetOps Security Classification Guide, JTF-GNO

3.5. JTTP Joint, Tactics, Techniques, and Procedures between GNSC and GIG NetOps-Centers

3.6. Defense Message System Standard Operating Procedure SOP 1-3

4. Background.

4.1 The GNSC operates and maintains the CONUS GIG backbone and inter-theater links/trunks for the Department of Defense (DoD). The GNSC is OPCON to the Joint Task Force – Global Network-Operations Center (JTF-GNO), while supporting seven CONUS Combatant Commands (COCOM’s), the Service Branches and numerous DoD Agencies. The GNSC provides the day-to-day operations of engineering, monitoring, managing, and maintaining the DISA CONUS GIG backbone. The GNSC, an information broker, coordinates and correlates system outages, assesses network and operational impact, coordinates, and executes restoration based on established priorities for GNSC customers.

4.2 The following NOCs, network missions, and support elements are OPCON to the DISA CONUS GNSC: DCCC, TMD, the Optical Transport Network (OTN) NOC consisting of the missions Optical Transport System (OTS) and Optical Digital Cross Connect (ODXC); the Multi-Service Provisioning Platform (MSPP)/Enhanced Synchronous Optical Network (ESONET)/Commercial Vendor (CV) NOC, Promina NOC, Defense Messaging System (DMS) NOC, Defense Red Switch Network (DRSN) NOC, Defense Switch Network (DSN) NOC, DISN Video Services (DVS) Global Video Operations Center (GVOC), Net Defense NOC, Internet Protocol (IP) NOC. Missions under the IP NOC include the Asynchronous Transfer Mode (ATM) NOC, Secure Internet Protocol Router Network (SIPRNET), Non-secure Internet Protocol Router Network (NIPRNET), NIPRNET COIN, Medical Healthcare Services (MHS), Push Key, DISN-LES (Leading Edge Services), DISN Core Unclassified IP, DISN Core Classified IP, and Strategic Tactical Entry Point Information Assurance (STEP IA). When the word “NOC” is used in this document it encompasses the above NOCs and network missions (to include Tier 0, Tier I, and Tier II) and support elements, unless otherwise stated. The TNCC/CNOSC will provide operational assistance IAW reference 3.7..

4.3 The DCCC reports administratively to the Defense Enterprise Computing Center (DECC) Columbus Site Director. The DCCC is the primary customer contact center for fielding customer reported network events. The DCCC interfaces with customers, gathers all pertinent information related to customer reported events, and performs quality assurance to ensure customer satisfaction. This frees up time for NOC technicians to troubleshoot and resolve network events.

5.0 General.

5.1 The guidance in this document relates to documenting information in TMS tickets and the actions utilized to resolve the events. The procedures and responsibilities provided in this document are not all-inclusive but are fundamental guidelines. If DISA CONUS Management (including the SCO) request more or less information be placed in a ticket, to support a requirement, then the NOC Shift Supervisor and/or NOC technician will comply with the request. If a situation arises where the SCO is directing actions against this document, the NOC Shift Supervisor and/or NOC technician will perform the actions as requested and advise their management chain to clarify the incident.

5.2 This document serves as an instruction guide for TMS users. The instructions provided within show the logical relationship among the tabs and fields of TMS and the reasoning for the guidance in this document. For detailed instructions refer to TMS User’s Guide Version 6.0.1, Reference 3.1, or use the help menu in TMS (application software).

5.3 This SOP will not cover every tab and field in TMS. Numerous tabs and fields are not commonly used, are self-explanatory, or are menu-driven to facilitate inputs. The fields addressed in this SOP have the greatest importance to the GNSC operational mission by facilitating information-exchange and situational awareness.

5.4 Times and dates placed in TMS fields will be placed in the following format: time [in Zulu/ Greenwich Mean Time (GMT)], day, month, and year Example: 1435Z 13 JUN 06. To ensure time/date inputs are consistent, accurate, and universal users of TMS will set their computers to GMT.

5.5 When this SOP references an “event,” it is referring to any situation to be tracked in TMS (e.g., a systems outage, degradation of service, a request for service, incident, or the monitoring of a GIG condition).

5.6 Providing near real-time CONUS GIG NetOps situational awareness, during the reporting of an isolation and restoration of a network event, is critical to ensuring the GNSC staff have accurate information to make timely and logical decisions, and keeping GNSC customers informed. The primary tool for providing situational awareness to customers is the GNSC NetOps Web Brief (www.gnsc.disa.smil.mil). This web briefing is automatically updated using TMS entries made from specific fields in TMS tickets. Accurate and timely entries are vital to DISA CONUS Management, JTF-GNO and GNSC customers. The following subsection identifies the selected fields in TMS tickets that are used to generate the GNSC Web Brief. Sections found later in this document will address the posting of the fields and the detailed guidelines for documenting information in those fields.

5.6.1 The following TMS fields are posted on the GNSC NetOps Web Brief to provide situational awareness of events that are deemed “Briefable” under the GNSC Reporting Criteria, Reference 3.2:

· Action Description

· Operational Impact Description

· Report Remarks

· Zulu Date Time Out

· Zulu Date Time In

· Outage Duration Hours

· Location of Problem

· Element ID

· Status The aggregation of the above fields must answer the fundamental questions of Who, What, When, Where, Why, and How. The information in these fields should contain detailed facts about the event and provide a clear picture to DISA CONUS Management, JTF-GNO and GNSC customers. Avoid the use of acronyms, technical terms and jargon in these fields whenever possible. See the Appendices placed at the end of this document for examples of the GNSC NetOps Web Briefing.

5.7 The following fields in TMS are related and will present a consistent representation of the degree of reporting, and visibility to DISA CONUS Management and JTF–GNO:

· Assigned Priority

· Operational Impact Description

· Local Priority

· Local Brief

· GNC Reportable

These fields are associated in a hierarchy and together represent a level of impact and reporting criteria to show the ticket’s visibility. These fields will be discussed further in subsequent sections of this SOP.

6. Responsibilities. As a general policy the SCOs will go through NOC Shift Supervisor to request information. When the NOC Shift Supervisor is not available, the SCO will go to the second person in charge. The GNSC mission comes first and at times this policy maybe broken due to the SCO needing a quick answer on sensitive/high-interest events. The NOC Shift Supervisor needs to be informed as to what is happening within their area-of-responsibility and understand what network event information is important to the SCO desk. When this chain of command is not followed, the NOC Shift Supervisor will be back briefed upon their return by the NOC technician that provided the information.

6.1 Systems Control Officer (SCO).

6.1.1 Primary Responsibility. The SCO has the overall responsibility to ensure TMS tickets are accurately documented in accordance with (IAW) this SOP. As events unfold, the SCO must manage and review the information placed in TMS tickets to ensure compliance with this SOP. If a ticket must be corrected, the SCO will ensure the appropriate NOC or supporting section makes the proper corrections; except in cases where a ticket is SCO-generated, with the Assigned Group field populated with “CONUS SCO.” Tickets of this type are known as “SCO Tickets.” The SCO will be directly responsible for updating the ticket.

6.1.2 SCO Tickets. When events happen relating to facilities (e.g., power outage, air conditioning, etc.), satellite link, or commercial events, that impact multiple DISN services (that do not ride the same DISN transport), the SCO will open a ticket and document the events. SCOs will also open tickets on any unique event that doesn’t fall under the responsibility of a particular NOC. This includes events that happen in other Theaters that cause impacts within the CONUS Theater of operations. The SCO ticket will be designated as the Higher Level Outage (HLO) and all related NOC tickets will be correlated to the SCO ticket. SCO tickets will appear on the GNSC Web Brief under the Special Interest link of the web site.

6.1.3 Higher Level Outages Tickets. The SCO has the overall responsibility to ensure Lower Level Outages (LLO) are correlated with the HLO. It is up to the SCO, in the course of managing unfolding events, to discern and annotate the relationships amongst transport, service, and application failures. The overall objective of correlating a HLO is to present a concise and accurate picture of the event’s impact, in order to answer the fundamental questions of Who, What, When, Where, Why, and How. When HLOs are recognized, the higher level NOC will notify all appropriate NOC Shift Supervisor to ensure they have situational awareness. The SCO will have the ultimate responsibility to ensure these actions have transpired.

6.1.4 Service Alt-routing. If possible, alt-routing of DISN Services will be done to get the GNSC customers back in service as soon as possible. The SCO will interface with the NOC Shift Supervisor to identify priorities of circuits, based on network and operational impacts. The SCO will make the final decision concerning which circuits to alt-route first, but the NOC Shift Supervisor will provide options, such as availabilities and capabilities.

6.1.5 Authorized Service Interruption (ASI). When an ASI is not associated to a NOC; the SCO will open the ASI Initiated (AI) ticket in TMS and track the situation under an Action Request (AR) ticket.

When an ASI runs over the scheduled window, the NOC Shift Supervisor will notify the SCO and request an extension. The SCO can approve two separate one hour extensions. When an extension is approved, the SCO will verbally notify the JTF-GNO and document the reporting in the Action Log of the ASI ticket. If an ASI runs over two hours, an extension will have to be approved by the JTF-GNO. If an ASI affects any connectivity Outside CONUS (OCONUS) then the SCO will get extension approval from the JTF-GNO and advise the appropriate TNC.

6.1.6 Customer Complaints. Customer complaints will be taken very seriously and the reported operational impacts will be placed in the Operational Impact Description field of tickets. The SCO will ensure any and all customer complaint tickets are marked web-briefable. Customer complaints include complaints from COCOMs, Services, DoD Agencies, and end-users of DISN services. The NOC Shift Supervisor will ensure all customer complaints are reported to the SCO. The name of the SCO will be documented in the Action Log field of the ticket.

Customer Complaints are defined as a person requesting an event be escalated; has concerns with the time it is taking to resolve the event; is frustrated/irritated with the progress/actions taken to resolve the event; advises a senior officer/manager is complaining about the lack of support/service; advises the event is causing a work stoppage and is affecting the performance of their mission or network capabilities.

6.1.7 Escalation from Tier I to Tier II. The SCO has the authority per the direction of the Branch Chief to direct the NOC senior representative to escalate a ticket from Tier I to Tier II. For more information on escalation procedures refer to Escalation SOP, Reference 3.4.

6.2

NOC Shift Supervisor/Technician (Tier I and Tier II).

6.2.1 Primary Responsibility. The OPS engineer/NOC Shift Supervisor Tier I and/or Tier II representative are responsible for reviewing and performing quality assurance on all tickets under the management of their assigned NOC. During duty hours, Tier I is responsible for Tier I tickets and Tier II is responsible for Tier II tickets. Before Tier II leaves for the day, their tickets will be updated. If the SCO desk requires more details or clarity of a Tier II ticket, after normal duty hours, the appropriate Tier II civil service Branch Chief will be called at home to get the information. If a Tier II tech is working from home, they can call the appropriate Tier I NOC and request their ticket be updated.

6.2.2 SCO Notification. The NOC Shift Supervisor is responsible for ensuring events are reported to the SCO IAW GNSC Reporting Criteria, Reference 3.2. The NOC Shift Supervisor is responsible for ensuring a comspottable ticket is created as soon as possible. The ticket will be updated with more details as information/time becomes available.

6.2.3 Network Tool/System Failure. When a GNSC network monitoring, network management, and/or configuration management tool/system fail(s) (i.e., server, database) affecting GNSC network operations, the NOC Shift Supervisor will notify the SCO. In accordance with Table 1, NOC – System/Tool Responsibilities, the primary user of the tool/system will notify the appropriate System Admin (RN6) technician, open a TMS ticket, and will ensure all actions concerning the failure and restoration are properly documented in the ticket (either by the RN6 person or by the primary user). The SCO will call in a System Admin (RN6) person after normal working hours. When the SCO or NOC Shift Supervisor is notified that the problem has been resolved, they will verify the resolution by testing/using the affected system/tool. The ticket will be marked “Y” (yes) in the Local Brief field and marked “Briefable” in the Local Priority field IAW GNSC Reporting Criteria, Reference 3.2. When the primary user is a NOC, the NOC Shift Supervisor will provide input to the SCO regarding the NOCs ability to fully monitor and maintain the affected network(s) despite the system/tool failure.

INMS - This does not include the Integrated Network Management System (INMS) telemetry feeds that come in from the separate NOC Network Monitoring Systems (NMS). When a telemetry feed goes down, the SCO will notify the appropriate System Admin (RN6) technician responsible for the NMS to start troubleshooting/investigating the cause of the event. The System Admin (RN6) technician will resolve the problem or contact the appropriate network or INMS person to resolve the event.

Table 1. NOC – System/Tool Responsibilities

6.2.4 Higher Level Outages. It is vital that the NOCs maintain an open flow of communication in order to assess events as they are unfolding and to assist the SCO in identifying HLOs. The NOC Shift Supervisors are responsible for ensuring events within their own NOC are reported to the SCO. When an event occurs that impacts multiple technologies, but does not meet the criteria for the opening of a “SCO Ticket,” the SCO and impacted NOC Shift Supervisor will coordinate to identify the network technology apparently at the root of the event. The NOC that appears at the root of the problem will open the HLO ticket. The identified NOC will have the responsibility for actively coordinating, tracking, and documenting events, until the event is resolved. The NOC that maintains this HLO ticket will notify the DCCC of the HLO, when applicable, and keep the SCO advised of all significant updates.

When a Transport NOC (i.e. OTS, ODXC, MSPP, ESONET) opens a ticket due to an alarm on their Element Management Systems (EMS) that relates to a GNSC Service NOC (i.e. SIPRNET, NIPRNET, DSN) ticket, and the Service NOC is the root of the network problem, the SCO will not mark the ticket “Briefable” to the GNSC web brief. The Transport NOC will open the ticket for situational awareness.

6.2.5 Customer Complaints. The NOC Shift Supervisor will report all customer complaints to the SCO. The name of the SCO will be documented in the Action Log field of the ticket. Customer complaints will be taken very seriously and the reported operational impacts will be placed in the Operational Impact Description field of tickets.

6.2.6 Authorized Service Interruption. The NOC Shift Supervisor/Tier II analyst/engineer will ensure the SCO is informed of all ASI issues to include: start times, end times, extensions needed, early starts on ASIs, and ASIs continuing past their scheduled times.

When Tier II personnel are conducting an ASI from a site away from the GNSC, the Tier I NOC will open the ASI ticket and document actions. The Tier II personnel conducting the ASI will call the related Tier I NOC and advise them of all ASI related issues. The Tier I NOC will then notify the SCO. All Tier II ASIs conducted from home will follow this process. If the operational tempo is such that the Tier I NOC cannot support this type of action, the NOC Shift Supervisor will notify the SCO to come up with a solution.

6.2.7 Escalation from Tier I to Tier II. The Tier I NOC Shift Supervisor will advise the SCO when a ticket is being escalated to Tier II. For more information on escalation procedures refer to Escalation SOP, Reference 3.4.

6.3 NOC Technician.

6.3.1 Opening / Closing Tickets.

6.3.1.1 Network Alarms. Tier I NOC technician (excluding DVS NOC) will open tickets relating to alarms identified and verified on the NMS, INMS, and EMS.

6.3.1.2 Customer Calls. Customer call outages are to be handled by the DCCC. When a NOC (excluding DVS/NET DEFENSE/STEP IA/, DVS-II customers will go through the DCCC to report all DVS related events to the GVOC) is contacted directly, concerning a customer network event, NOC technicians must not simply provide customers with the DCCC phone number and terminate the phone call. NOC technicians will conference call all customers with the DCCC, and initiate a “soft handoff” with the DCCC. Although the DCCC will open the ticket, the NOC assumes equal responsibility with the DCCC to ensure customer satisfaction. If operational tempo is such that it is not possible to provide a “soft handoff” to the DCCC, the customer will be provided the DCCC telephone number and then forwarded to the SCO.

When a DCCC conference call cannot be established, due to heavy call volume at the DCCC, a voice-mail will not be left. The NOC technician will open a ticket for the customer and notify the DCCC of the situation later. When the event is resolved the ticket will be transferred to the DCCC...

6.3.2 Element Isolations. When a network element (e.g., node, switch, multiplexer) is isolated by an event, a “master ticket” will be opened to document the event and, when applicable, related to the HLO. When the element restores, the ticket will be placed in Fixed, Fixed HD, or Monitor status. At this point NOC personnel will confirm the restoration of each circuit/trunk (within the NOCs technology) connecting to the restored element and verify there are no residual outages. If any circuits/trunks are not restored at the time the element restores, the ticket will be disassociated from the HLO ticket and another RFO will be used to track the current reason for outage. The “master ticket” will not be used to track residual circuit/trunk that did not restore.

6.3.3 Impaired Circuits/Trunks. When a NOC has an impaired circuit/trunk that is repeatedly taking errors, a TMS ticket will be opened to track the problem. If an outage of the impaired circuit/trunk occurs and the outage is not clearly related to the already-detected errors, a new TMS ticket will be opened to document the outage.

6.3.4 Single Threaded Network Elements. When a trunk goes down that isolates a network element, the ticket will be opened on the trunk and the CCSD placed in the Element ID field of the ticket. The network impact as it relates to the element isolation will be documented in the Operational Impact Description field of the ticket. If the network element is the root of the problem, then a ticket will be opened on the network element, not the trunk.

6.3.5 Investigating Reason for Events. When an event is resolved the responsible NOC has 3 business days to obtain RFO information. After 3 business days if no RFO is obtained the outage ticket will be closed. NOCs will document attempts made to obtain a RFO in the Action Log to include the name of a person or site to which the attempt was made. The original outage ticket will not be used to document additional outages but a new outage ticket will be opened. If a RFO is known but is awaiting TAC case resolution or additional vendor analysis, the original trouble ticket will be closed and the initiator of the vender request will open an informational ticket to track resolution.

6.3.6 Authorized Service Interruption. The SCO will be advised of all significant issues relating to ASIs by the responsible NOC. ASI tickets will not be placed in “Closed” status by the NOC. The ASI Branch is the only group that closes ASI generated tickets. The NOC will track the ASI as it progresses and place the ticket in “Fixed” status once completed. When an ASI runs over the scheduled time an extension can be requested through the SCO. The name of the SCO or the JTF-GNO CDO that approves the extension(s) will be placed in the Action Log field of the ticket. When an extension is not approved and/or the site cannot restore the network(s) to normal operations after the schedule ASI window has elapsed, a separate outage ticket will be opened and the situation verbally briefed to the SCO.

6.3.7 DISN Customer Call Center (DCCC). IAW DISA CONUS GNSC and DCCC SOP, Reference 3.3, the DCCC will open and close TMS tickets for customer-reported issues. The DCCC will follow the same guidelines in documenting events in TMS tickets as outlined in this document.

6.4 Tactical Mission Director (TMD).

6.4.1.

Master Mission Tickets. The TMD will open a ticket for the life cycle of the operational mission. This ticket will be referred to as the “master” ticket. The master ticket will contain all related documentation for the mission and will be an information reserve for all correspondences relating to the TMD support of the tactical mission.

6.4.2 Tactical Mission Outages. The master ticket will not be the sole source used to document events relating to network outages impacting tactical missions. The master ticket will summarize all events occurring during the life of the mission. When an event not the users fault occurs that causes an impact to ongoing tactical missions, the TMD will open a TMS ticket to track the event. When applicable, the event will be related to a HLO. The TMD will mark the ticket “Y” (i.e. “yes”) in the Local Brief field and verbally notify the SCO and document the name of the SCO in the Action Log field of the ticket. The SCO will make the ticket “Briefable” in the Local Priority field IAW GNSC Reporting Criteria, Reference 3.2.

The TMD position is not a 24x7 position, but is on-call 24x7. When an event happens outside normal working hours, the SCO will open the TMD ticket, document the event, and back brief the TMD.

7.

Procedures.

The key point regarding documenting events in TMS is ensuring complete and up-to-date information is recorded in all tickets. If actions aren’t documented in the TMS tickets, the actions didn’t happen and there will not be any historical data to perform trend analysis or determine workload metrics. Detailed documentation takes a lot of time and effort, but failure to fully document can cost far more time and effort. The GNSC operates 24/7/365. The sheer volume of tickets that are worked daily can be overwhelming and the only thing that keeps operations stable is the accurate and timely documentation of events. When tickets are not documented appropriately, the GNSC management has to justify the actions taken by NOC technicians. Numerous hours are lost trying to research what happened and the reasoning behind the event. Failure to fully document actions lead to wasted time for other NOC technicians, as they must duplicate troubleshooting efforts.

7.1

Main Tm Action Request.

7.1.1 Action Request Number. The AR Number (a.k.a. “ticket number”) is auto-generated when a ticket is opened. The TMS ticket number is a unique identifier for every event tracked in TMS. When an outside party interfaces with the GNSC regarding an event tracked in TMS, the ticket number will be provided. Providing this number and emphasizing its importance to the outside party will save time down the road. Experience has shown that many hours are lost within the GNSC when customers call without ticket numbers, leading to searches for applicable tickets. Giving ticket numbers to the customers will, in some cases, allow some customers to review tickets in TMS themselves, without having to engage NOC representatives.

7.1.2 Status. This field is filled by selecting an option from a drop-down menu. The purpose of the field is to show the current state of troubleshooting for the event. The field must be populated when opening a ticket. The options in the drop-down menu are discussed in the following sub-sections. Note that when changing the Status field, the Status or Resolution Summary, and when applicable, the Report Remarks field(s) will also be updated explaining the change of status.

7.1.2.1 Initiated. This is the default option in the Status field when opening a ticket and is generally the correct option at the time of ticket creation. The field should remain in this status only as long as no action has been taken to resolve the event. Tickets being transferred from the DCCC to a GNSC NOC will be in an Initiated Status.

7.1.2.2 In Progress. The Status “In Progress” will be used when the ticket has been assigned to a technician and actions are underway to resolve the event. Most tickets will show this option from the time investigation actions are initiated until a different status becomes applicable to the ticket. TMD will use In Progress field in the mission's master ticket to show the tactical mission's process of establishing DISN services.

In Progress is also the appropriate value when transferring ticket responsibility between Tier I and Tier II.

7.1.2.3 Related.

When the problem relates to a GNSC TMS HLO ticket, TMS will automatically mark the ticket as Related. When the HLO ticket is updated the child ticket(s) will be automatically updated in the status resolution summary and action log with any new updates.

7.1.2.4 Referred. This option is used when action for the event has been turned over to an entity such as, a commercial vendor or service agency. The Status or Resolution Summary and, when applicable, the Report Remarks field(s) will include the following: whom the ticket has been referred to, when a response or resolution is expected, and the level of escalation with the commercial vendor (when applicable). When a ticket is in this status, the commercial vendor will be called for an update. The SCO will direct the NOC Shift Supervisor as to how often the NOC will call the commercial vendor for an update. Updates from the vendor will be annotated in the ticket’s Status or Resolution Summary field.

7.1.2.5 Deferred. This option will be used when the event is not being actively worked, typically because a decision on how to proceed is forthcoming from DISA CONUS Management or the customer. The name, rank/title/position, and contact information of the highest ranking person on site that authorized the deferment will be documented in the Status or Resolution Summary field of the ticket. Tickets with this status will be updated every 24 hours, unless a specific follow-up date has been identified.

7.1.2.6 Pending. This status is selected when the event is awaiting some other action, such as the on-site arrival of parts, a technician, or a customer representative. This status will also be used when awaiting approval of an ASI request or the start of a tactical mission. The following guidelines will be followed to avoid tickets being kept in Pending status for an excessive amount of time.

7.1.2.6.1 Less than 72 Hours. When a ticket is predicted to be in Pending status for less than 72 hours, it will be updated every 8 hours or once per shift. A typical reason for this type of delay is that we’re awaiting action to be taken after a holiday or weekend. The exception to this rule is when a specific time/date for a resumption of restoration actions has been identified and approved by the customer. The name and telephone number of the customer representative approving the plan to resume restoration actions at the identified time/date will be documented in the Action Log and in the Contact Info Tab of the ticket.

7.1.2.6.2 Greater than 72 hours. When a ticket is predicted to be in Pending status for more than 72 hours, it will be updated every 3 days, unless a specific time/date has been identified and approved by the customer. After 7 days have passed since putting a ticket marked Major in Pending status, the SCO will be notified. The name of the notified SCO will be documented in the Action Log. The SCO will consider whether escalation of the issue is required or if the Briefable ticket should be marked Residual.

7.1.2.7

Monitor. The concept of placing a ticket in “Monitor” status is to maintain visibility within the NOC, after resolution and ensure it is closely watched for recurring problems. TMD will use "Monitor" status in the mission's master ticket when the tactical user's requested DISN services are up and until the mission has ENDEXED. A ticket will be placed in Monitor status when:

· The event came to resolution for unknown reasons

· It appears or is assumed the event may reoccur

· The customer requests a period of monitoring

· The event was of high interest to GNSC management

· The NOC or Tier II deem it necessary

In most instances, the monitoring period will be no more than 24 hours. However, there will be instances when the monitoring period needs to be extended. The following are instances when a ticket will be left in “Monitor” status longer than 24 hours:

· The customer requests an extended monitoring time

· Identifying and tracking impairment issues on a circuit or trunk

· Identifying and tracking intermittent issues on a card, hub, node, switch, etc.

The length of monitoring period depends on the severity of the event and the visibility/interest level; however, an ending date and time will be identified and stated in both the Action Log and Report Remarks fields of the ticket. For significant events (i.e., router or switch isolation) the monitoring period will be 2 hours. If the event happens after 2 hours a new TMS ticket will be opened. This procedure will assist in providing Situational Awareness (SA).

During the monitoring period, a ticket will be reviewed and status verified every 8 hours or once a shift. Verifying the status is not a simple cut and paste action from element management systems. A technician will review performance matrices, logs, and/or contact the customer to confirm the status.

After the monitoring period ends, the ticket can be placed in “Fixed” or “Fixed HD” status if no problems or minimal events reoccurred. If events continue, but do not cause an outage, an impairment ticket will be opened and the original ticket closed. When an outage reoccurs, the ticket will be placed back to “In Progress” and new troubleshooting actions will be taken to resolve the event. If the event is related to the first problem a new RFO will be created and the ticket will remain open to continue tracking the event. If it is determined the event is unrelated to the first problem, the first ticket will be closed and a new ticket opened. In situations such as a satellite link being affected by weather, the ticket will remain in monitor status until the weather has cleared the area of the satellite facility.

7.1.2.8 Fixed. This option will be used when:

· The ticket is waiting to be Closed

· The event is resolved, but requires a quality assurance check by the Operations Engineer and/or NOC Shift Supervisor

· The ticket is waiting to be briefed to DISA CONUS Management

· If the event is resolved and the NOC is awaiting a RFO.

Tickets marked “Briefable” in the Local Priority field; it will remain in the “Fixed” status until it is briefed at the GNSC’s Morning Ops Brief.

7.1.2.9 Fixed-HD. This option is used when transferring a ticket back to the DCCC after the appropriate NOC has restored the event. If during the act of restoration the NOC has the customer on the line and has verified with the customer that they are satisfied with the action taken, the NOC will conference in the DCCC with the customer on the line. This will ensure the earliest possible customer satisfaction verification and closure of ticket. Only the Operations Engineer and/or NOC Shift Supervisor may place the ticket in “Fixed-HD.” The DCCC is then notified that it should take action to close the ticket. Before a ticket is placed in this status, the Operations Engineer and/or NOC Shift Supervisor will notify the SCO if the ticket has a COMSPOT.

7.1.2.10 Canceled. This status is used for tickets canceled due to the existence of a duplicate ticket or when it is determined that a ticket was opened unnecessarily IAW Reference 3.3. If it is determined the DCCC ticket needs to be canceled, the ticket will be reassigned to the DISN Help Desk.

7.1.2.11 Closed. This option will be selected only when the event has been resolved and all actions have been documented in the ticket. The Operations Engineer and/or NOC Shift Supervisor will perform a final quality assurance check on the ticket before selecting this option. Only the Operations Engineer and/or NOC Shift Supervisor (including the DCCC) has the authority to change the status of a ticket to “Closed.” If a COMSPOT has been sent on the ticket, the Operations Engineer and/or NOC Shift Supervisor will contact the SCO to have a “Final” COMSPOT sent before closing the ticket. Once a ticket has been placed in a “Closed” status, it cannot be updated by anyone other than the SCO.

A SIPRNET ticket can be closed after a port has been administratively shut-down for 5 days due to non-response from the customer.

7.1.3 Higher Level Outage. When a ticket is opened on an event that has caused another event, the AR Number (a.k.a. ticket number) of the HLO will be placed in the Higher Level Outage field for all LLO tickets that relates to the HLO event. This field is critical in correlating tickets for the GNSC Web Brief. When the ticket number is placed in the HLO field of a ticket and the Local Priority field is marked “Briefable” it shows up as a LLO or “child ticket” under its HLO or “parent” ticket on the GNSC Web Brief. TMS does not require this field to be populated, so the NOC technician working the LLO tickets must be aware of this field and understand its importance when correlating tickets to higher-level events. The NOC technician working LLO tickets will be responsible for placing the ticket number of the correlated ticket in the HLO field. The SCO is responsible for marking the ticket “Briefable” in the Local Priority field.

7.1.4 Submitter. This field is automatically populated with the TMS username of the person that opens the ticket.

7.1.5 Record Classification. Identifies the classification of the event. A value for this field must be determined when opening a ticket. When the classification of the event is unknown, over-classification is recommended. In determining the classification of a ticket, the consolidation of all the information in the fields and tabs must be considered. Refer to NetOps Security Classification Guide, Reference 3.5, published by the JTF-GNO for TMS ticket classification guidance.

7.1.6 Spell Check. Everyone using TMS will use spell check on entries made in tickets. Having misspelled words in entries detracts from the meaning/importance of a ticket and distracts the reader from understanding and retaining the ticket’s content.

To use Spell Check in TMS; a user must first change the option in the Spelling & Grammar section of Microsoft Word to “Ignore words in UPPERCASE.” The procedure for this is: open a Word document; go to Tools on the menu bar; click on Options; click on Spelling & Grammar; unclick “Ignore words in UPPERCASE”; and press OK.

7.2 Contact Info Tab.

7.2.1 Whenever a person is involved in the resolution of an event, their contact information will be placed in the Contact Info tab before being placed in any other fields. Past tickets may be used by the GNSC to find contact information to assist in resolving current outages. When a NOC technician opens a ticket that relates to a tactical mission or a STEP asset the technician will add the TMD (Last Name=TMD) to the list of contacts. This ensures the TMD receives and E-mail notification on the new ticket and subsequent updates. This has to be accomplished when a ticket is created. If it is added after it is saved and no other fields are updated no email will be generated.

7.2.2 Role. This is a menu-driven field. Select the relationship of the contact person to the event. In some cases, clarification is needed. Specifying that a person is “the caller” is worthwhile, but adding that the caller is at “Tinker AFB OK” is even more helpful. Any additional supporting information will be added to the Remarks section.

7.2.3 Remarks. There are separate fields for commercial and DSN phone numbers; however, TMS doesn’t currently provide a field for a mobile/cell phone number. It can be very helpful to have this information; therefore, technicians are strongly encouraged to ask people for mobile/cell phone numbers. Record those numbers in the Remarks field.

7.3 Action Info Tab.

7.3.1 Action Description. A 500-character free text field used to record a brief description of the event. Text is required in this field when opening a ticket. When the field is modified and saved it is automatically appended to the Action Log field. This field is critical in providing an account of the event on the GNSC Web Brief. The information placed in this field will be a concise and accurate description of the event/network outage. A reader should be able to easily understand the fundamental issue merely by reading the information in this field. Ticket and telephone numbers are not to be placed in this field. The information placed in this field will include items described in the following subsections, in the order presented below, with the exception of items that do not apply need not be mentioned. Clarifying information may be placed within and between the items below, but any such information must be concise and placed in parentheses.

7.3.1.1 Special Identifier. Special identifiers are used to denote certain types of tickets and/or ensure tickets appear at the appropriate locations on the GNSC Web Brief. When a ticket relates to PUSHKEY, HJ, ODM, TMD, EXERCISE, COOP, CHRONIC, RESIDUAL the first word in the field will be “**PUSHKEY**”, “**HJ**”, “**ODM**”, “**TMD**”, “**EXERCISE**”, “**COOP**”, “**CHRONIC**”, “**RESIDUAL**”, “**DISN LES** respectively. When any ticket is opened by the DCCC, it will have the value “DCCC” in this field. To distinguish between different OTN technologies, the appropriate identifiers “OTS” or “ODXC” will be used. When technology identifiers are used, all other required special identifiers will immediately follow.

7.3.1.2 Network Identifier. This refers to the Command Communications Service Designator (CCSD) or network element identifier (e.g., “NPRW143”) that the event impacts. When entering a CCSD in a ticket, the CCSD will stand alone (e.g., “74ER”). There is no need to label it as a CCSD (e.g., “CCSD 74ER”). Up to three CCSDs will be referenced individually, separated by slashes (e.g., “74ER (256KB) / 241X (4MB) / 24RX (10MB)”). When four or more CCSDs are affected by an event, reference the CCSDs as an aggregate (e.g., 6 CIRCUITS). When a node has a plain number (e.g., 45) the node will be represented by an “N” or spelled out (e.g., “N45” or “Node 45”). For network identifiers that have classified and unclassified counterparts, state the classification behind the identifier (e.g., “DMS-NIPRNET,” “JIDMACDILL.3-CLASS”). When a CCSD is unknown but the source IP is, the DCCC will list the source IP address. The IP NOC will research and update with the correct CCSD. If available, the node and interface may be listed

7.3.1.3 Bandwidth. When an event is related to three or less CCSDs, the circuit’s bandwidth or data rate will always be placed in parentheses behind the CCSD [e.g., “241X (T1)” or “241X (1.544MB)”]. When four or more CCSDs are affected by an event, no bandwidth or data rate is required.

7.3.1.4 Location. This portion of the entry identifies the location(s) of the network element. When the event relates to a CCSD, both endpoints will be identified and separated with the word “TO” to separate the locations (e.g., “MACDILL AFB TO SCOTT AFB”). If the CCSD remains in one location, clarify the technologies involved [FAIRCHILD (ATM) TO FAIRCHILD (ESONET)]. Inserting the state after the location is not required if the location is well-known. When a location is outside of the United States, specify the country.

7.3.1.5 Event Description. This field will contain a brief, concise description of the reported event. State how the network element is or will be affected.

7.3.1.6 Additional Information. Additional information should be kept to a minimum (e.g. event alarms). Information of value generally falls into the categories described above. Specific items that should not be considered “additional information” include: vendor ticket numbers, government contract, dispatched personnel, and contact information.

7.3.1.7 Examples of Action Description Field.

OTS - 6ABC - LINK 3 (1 OF 14 - OC-192 - 10GB) SHEPPARD AFB TO KELLY AFB DOWN BECAUSE OF A FAILED AMPLIFIER.

DCCC 76M3 (12MB) LACKLAND AFB TX (N14 S1/0/3) TO FT BRAGG NC DOWN DUE TO BAD KG-194 AT FT BRAGG.

DCCC CCSD NOT PROVIDED BY CALLER. SOURCE IP: XXX.XXX.XXX.XXX (LOCATION) DESTINATION IP: XXX.XXX.XXX.XXX (LOCATION) TRACE STOPS: XXX.XXX.XXX.XXX.

DCCC **AFTER THE FACT** 23RH (T1) **NOVA** SAN DIEGO TO HICKAM AFB DOWN DUE TO LOSS OF SIGNAL.

71RH (T1) FT LEWIS TO FAIRCHILD AFB WA DOWN DUE TO A COMMERCIAL CABLE CUT.

**HJ** 26XY (128KB) MACDILL AFB (N45 S1/8/2) TO SOUTH FLORIDA HIDTA MIRAMAR CA DOWN DUE TO CUSTOMER HAVING NO KEYMAT.

DCCC 6GQ3 (DS-3) CHEYENNE MOUNTAIN TO SCOTT AFB DEGRADED DUE TO TRUNK TAKING ERRORS.

UCXWBRA056 (CELL EXCHANGER) FT BRAGG DOWN DUE TO POWER OUTAGE.

8 CIRCUITS, CORPUS CHRISTI DOWN DUE TO A POWER SUPPLY FAILING ON A CSU NEST.

24G4 (T1) / 26UT (256KB) / 2654 (2MB), NORTH ISLAND TO SAN DIEGO DOWN DUE TO A TECHNICIAN REMOVING CROSS-CONNECTS.

DCCC NGA (120.45.13.65) IS NOT RESPONDING TO PING COMMANDS.

DMS-NIPRNET, BACKBONE UNABLE TO CONNECT TO PRIMARY AND

BACKUP GROUPWARE SERVERS.

DMS-SIPRNET, NCTAMS LANT, NORFOLK UNABLE TO DELIVER TEST

MESSAGES.

NIDHUN1-UNCLASS, HUNTSVILLE IS NOT ACCESSIBLE VIA SSH.

ARLINGTONJTAMDO-HQDA-SC, MULTIPOINT (13 ROOM) SECURE CONFERENCE (1700Z –1900Z). SUBJECT: NIMBLE TITAN.

KIRTLANDAFRL-MDA-SD TO BAGHDADEMDMCU-EUR-DD, POINT-TO-POINT SECURE CONFERENCE (1600Z-2000Z). SUBJECT: PLANS BRIEF.

7.3.2 Impacted Communities. A free text or pull-down menu driven field. The menu shows the COCOM, DoD Services, Agencies, and customers. More than one option can be selected in this field; however, selecting from the COCOM list is mandatory. The COCOM Network Operations Security Centers (CNOSCs) utilize this field to filter the GNSC TMS tickets and gather situational awareness within their area of responsibility. Selecting the correct COCOM from the menu is very important in assisting the CNOSC to identify their tickets. See the “COCOM AOI” (Areas of Interest) link from the GNSC Web Brief webpage for more information. We will also indicate in special/high interest customer or community of interest in this field. If they are not already listed they will be typed in the field.

7.3.3 Operational Impact Description. A 500-character free text field used to record a brief description of the network impact (NI) and operational impact (OI) of the event. This is a key field and is required when opening a ticket. The NI and OI will be addressed in this field. The NI can be identified (as it relates to DISN connectivity) by stating if the organization is isolated, has alt-connectivity, or has other DISN services. This information can be obtained through system admin resources.