Sop100-55-04 GNSC Ops and DCCC.doc

DISA CONUS Standard Operating Procedure (SOP) 100-55-04

DISA CONUS Standard Operating Procedures (SOP) 07 November 2008 100-55-04

OPERATIONS

CONUS Global NetOps Support Center (GNSC) Operations and

DISN Customer Call Center (DCCC)

1. Purpose. This SOP provides guidance for the interaction and areas-of-responsibilities between the Global Network Operations Support Center (GNSC) Branches, Systems Control Officers (SCO), Network Operation Centers (NOCs), DISA CONUS IP Services Branch, Tactical Mission Director (TMD), and the DISN Customer Call Center (DCCC). This SOP cancels and supersedes DISA CONUS SOP06-08 dated 27 Oct 2006.

2. Applicability. This SOP applies to the GNSC branches, SCOs, NOCs, DISA CONUS IP Services Branch, TMD, and the DCCC.

3. References.

3.1. DISA CONUS SOP 100-55-01 (Rev 1), GNSC Trouble Management System (TMS), 31 Jan 2008

3.2. TMS Users Manual, Version 7.7.7, 30 April 08

3.3 TMD SOP 05-03, 26 Aug 2005

3.4 DISA CONUS SOP 100-55-05 Generating and Delivering NIPRNet/SIPRNet Customer Circuit Performance Reports, 19 Nov 08

4. General.

4.1 The DCCC is operationally controlled (OPCON) by DISA CONUS GNSC and reports organizationally to DISA, DISN Customer Service Division, NS7.

4.2 The DCCC is the primary customer contact center for all DISA CONUS NOCs and networks under GNSC OPCON. The DCCC will interface with the following: Optical Transport Network (OTN: includes Optical Transport System [OTS] and Optical Digital Cross Connect [ODXC]), Multi-Service Provisioning Platform (MSPP), Military Healthcare Systems (MHS), Asynchronous Transfer Mode (ATM), Commercial Vendor (CV) NOC, Tactical Mission Director (TMD), DISN Core, DISN Core IP, Defense Message System (DMS), Defense Red Switch Network (DRSN), Defense Switched Network (DSN), Enhanced Synchronous Optical Network (ESONET), Internet Protocol (IP) Core, Non-classified Internet Protocol Router Network (NIPRNet), Push Key, Promina, Secret IP Router Network (SIPRNet), DISN Video Services (DVS), Voice Over IP (VoIP), and Secure Mobile Environment - Portable Electronic Device (SME-PED - in progress}.

4.3 The DCCC shall contact the GNSC SCO in determining the appropriate actions to take if the contact issue is not covered in this documentation.

4.4 The DCCC shall interface with the DISA CONUS Engineering Division, IP Services

Branch, CONUS IP Tier II (NIPRNet and SIPRNet) to coordinate the generation and delivery of circuit performance reports.

4.5 The DCCC shall support customers that request assistance in dealing with Tactical Missions through DISA CONUS TMD (see Ref 3.3).

4.6 For the purpose of this document, Action Request (AR) will be referred to as “trouble ticket(s).”

4.7 When a reference is made in this document concerning “contact” it relates to the method in which customers notify the DCCC, via telephone, fax, and/or e-mail.

5. Responsibilities.

5.1 The SCO Desk Position shall:

5.1.1 Have the overall responsibility to ensure the procedures outlined in this document are followed. If a situation arises that is not covered in this document or if additional clarification is required, the DCCC and the NOCs will contact the on-duty SCO. The SCO will determine the correct course of action to resolve the situation. If any conflict arises between the interacting parties identified in this SOP, and the issues cannot be resolved, the issue will be elevated to the SCO for final resolution.

5.2 DCCC shall:

5.2.1 Upon receipt of a customer contact (via phone, email, etc): open customer trouble tickets; determine initial fault identification; denote if the customer’s incident is potentially related to an existing root cause outage; assign accurate trouble tickets to the appropriate NOC; monitor ticket status and provide updates/feedback to the customer; perform quality assurance (QA) with customers; and close all DCCC/customer generated trouble tickets.

5.2.2 Take appropriate action to resolve any incidents/issues within their scope, such as: providing status updates to customers upon request; assisting customers with general questions on Authorized Service Interruption (ASI) submission; and requests for operational and non-operational circuit information that can be obtained from configuration management and provisioning management databases. The DCCC will forward issues beyond their scope to the appropriate area of responsibility. The DCCC will escalate unresolved or operational incidents that need immediate attention to the GNSC SCO.

5.3 The NOCs shall:

5.3.1 Troubleshoot customer trouble tickets assigned to them by the DCCC. The primary fields in TMS that the NOCs will maintain are: Action Log; Status or Resolution Summary; RFO Remarks; and Report Remarks. These fields shall remain current throughout the troubleshooting process. After a trouble ticket has been resolved, the appropriate NOC will mark ticket status “Fixed HD” and reassign the resolved customer trouble ticket to the DCCC for QA and closure. The NOC will open and close trouble tickets for network impacting Integrated Network Monitoring System (INMS) or Element Management System (EMS) alarms.

5.4 Tactical Mission Director (TMD) shall:

5.4.1 Open master TMS tickets and monitor ongoing tactical missions for the lifecycle of the mission. The TMD is a rotating position. The person sitting at this position provides situational awareness briefs to the DISA CONUS Commander regarding all ongoing tactical missions. The TMD is responsible for opening, tracking, and closing TMS trouble tickets on DISN outages that affect tactical missions and, when applicable, correlate the tickets with the Root Cause Outage (RCO). The TMD acts as a liaison between the GNSC and the tactical end-user. Included in the liaison duties, the TMD collects and assesses network and operational impacts for the GNSC regarding DISN outages (see Ref. 3.3).

5.5 DISA CONUS IP Service Branch shall:

5.5.1 Ensure timely and accurate NIPRNet/SIPRNet Customer Circuit Performance Reports.

6. Procedures.

6.1 Customer Contact Routing:

6.1.1 Upon receipt of a customer contact, the DCCC shall identify if the caller is a local DISN service customer, a technician, technical control facility (TCF), help desk, or Service Network Operations and Security Center (NOSC). If the caller is a technician, TCF, help desk, or Service NOSC, the DCCC will open a customer trouble ticket. If the caller is a local DISN service customer, the DCCC must determine if the customer has contacted their respective Service NOSC (Air Force, Army, Marine Corps, and Navy) or local area network (LAN) representative for troubleshooting and reporting assistance. If the customer has not contacted their Service NOSC or LAN representative, the DCCC will guide the customer in making that contact. Should the customer’s LAN representative not be available, the DCCC will conference the customer to their respective Service NOSC for assistance. The Service NOSC will make the appropriate contact if a DISN related incident exists.

6.1.2 If a customer calls directly to a NOC, the NOC technician will conference the customer with the DCCC. The DCCC will create a ticket with the NOC and the customer on the line. If the ticket will be assigned to the NOC receiving the original call, the DCCC will assign the ticket and drop off the line allowing the NOC to continue working with the customer. If the ticket will be assigned to another NOC, the NOC who received the original call will drop off the line so the DCCC can explain the proper reporting procedures to the customer and assign the ticket to the appropriate NOC. This will ensure a proper hand off of information and educate the customer regarding proper reporting procedures. The DCCC can be contacted at DSN 850-4790, Commercial 1-800-554-DISN (3476).

6.2 Opening, Documenting, and Routing Trouble Tickets

6.2.1 To avoid duplicate trouble tickets, both the DCCC and the NOC technicians will query TMS for an existing trouble ticket on the affected circuit/trunk before opening a trouble ticket. If a NOC ticket and a customer DCCC ticket are both opened for the same CCSD, the oldest ticket, based on the Zulu date/time the ticket was initiated, shall be the lead ticket. The initiated Zulu date/time is the first entry made in the Action Log field. The Submitter of the duplicate ticket will be responsible for replicating applicable information into the lead ticket. If the duplicate ticket is one submitted by the DCCC it will be assigned to the DCCC for cancellation. The NOC technician will keep the Status field as “Initiated”, change the assigned group to “CONUS DCCC OPS” or CONUS DCCC DVS” and will enter a statement in the Status or Resolution Summary field indicating the customer trouble ticket is a duplicate of an existing NOC ticket. (i.e. Duplicate of Trouble Ticket 12345).

6.2.2 If there is an existing network alarm trouble ticket created by a NOC, the DCCC shall:

6.2.2.1 Add the caller to the network alarm trouble ticket. Add “DCCC” as the first entry to the action description filed. Change the Submitter Site field to GNSC DCCC OPS COLUMBUS OH. or GNSC DCCC DVS COLUMBUS OH as applicable. Review the impact description. If the customer’s impact is more severe than what is stated, the impact description will be updated. If the impact is less severe, the customer’s impact will be added to the Action Log field of the trouble ticket. Advise the appropriate NOC that a customer has reported the outage and the NOC trouble ticket has been modified to a DCCC ticket.

6.2.2.2 The DCCC will obtain the necessary information from the customer contact to determine if the incident reported is related to the DISN. The DCCC will use the Integrated Network Management System (INMS), Trouble Management System (TMS), and applicable configuration management and/or provisioning management databases to correlate the incident with current DISN anomalies. Should the DCCC require assistance in determining the incident being reported is related to the DISN they will conference in the appropriate NOC Shift Lead for assistance. The NOC Shift Lead will determine if the customer’s reported incident is related to the DISN and will direct the DCCC to open a trouble ticket if appropriate.

6.2.3 When opening a trouble ticket, the DCCC will obtain the appropriate information from the contact to properly complete the customer trouble ticket. The information will be thorough enough for NOC technicians to clearly understand the reported incident without consulting the customer for clarification. Should the DCCC require assistance in gathering the required information from the contact, they will conference in the appropriate NOC Shift Lead for assistance. The DCCC and the NOC Shift Lead will work together in gathering the required information.

6.2.4 If the DCCC opens a customer trouble ticket on a circuit where the HIT/SI field is auto-populated, they will mark the Local Brief field “Y” in the “Reporting Tab” and will notify the appropriate NOC.

6.2.5 Once the customer trouble ticket has been opened, the DCCC will assign the ticket to the appropriate NOC. At that point the NOC shall take the ticket for action. To ensure the ticket gets noticed by the appropriate NOC, the DCCC shall call the NOC Shift Lead to advise them a new customer trouble ticket has been placed in their queue.

6.2.6 When a new customer trouble ticket appears in the NOC queue, the NOC Shift Lead shall:

validate the trouble ticket; change the status from “Initiated” to “In Progress” and assign the ticket to a NOC technician. If the NOC Shift Lead determines the ticket is not assigned to the correct NOC or information is missing, the Shift Lead will call the DCCC to resolve the issue.

6.2.7 All NOCs will follow the established guidelines for working incidents to resolution. The assigned technician will ensure all attempts to troubleshoot the incident are adequately documented in the ticket until the incident is fixed to the customer’s satisfaction. The Status or Resolution Summary field in the ticket should always reflect the most current status. If the ticket is marked “Y” in the Local Brief field, the Report Remarks field should also reflect the most current status. The Status or Resolution Summary field and, when applicable, the Report Remarks field shall contain a course of action, estimated time of restoration, and reason for outage (RFO). The RFO Remarks field shall be populated when a resolution is established. The Assigned Technician field in ticket is not a mandatory field; however, if the DCCC needs to call the NOC to obtain more information about a ticket, knowing the name of the technician would facilitate locating the assigned technician.

6.3 Trouble Tickets Relating to Root Cause Outages (RCO).

6.3.1 When there is a root cause outage (RCO) (i.e. node isolation, root cause transport) anomaly that affects numerous customers, the NOC will notify the DCCC of the outage. The DCCC will use the appropriate configuration management database to identify potential users affected by the RCO.

6.3.2 When the DCCC receives a customer contact/ticket that potentially relates to a RCO, the ticket will be assigned to the appropriate NOC. The NOC will review the customer’s trouble ticket to ensure the customer’s incident is indeed related to a RCO. If the NOC verifies that the customer’s incident is directly related to a RCO, the NOC will annotate the customer ticket with the root cause ticket number in the Root Cause Outage (RCO) field of the trouble ticket. This will establish the TMS parent/child relationship. TMS automatically changes the Status Field to “Related”. The NOC will annotate the Status or Resolution Summary field in the trouble ticket stating the cause of the incident is due to a RCO. The NOC will then reassign the ticket to the DCCC for monitoring.

6.3.2.1 Should a ticket that resides in the DCCC queue have the “local priority” set to briefable by the SCO, the SCO will return the ticket to the applicable NOC until it is briefed and the local priority field is cleared. Should the DCCC discover a ticket assigned to them that has the “local priority” set to briefable, they will contact the SCO to remove the “local priority” briefable flag or, if needed, the SCO will reassign the ticket to the applicable NOC until the briefable flag is removed (Note: This is applicable to all tickets).

6.3.3 If the NOC determines the customer’s incident is not related to the RCO, they will treat the ticket just like any other customer generated trouble ticket. NOTE: It is very important that the parent/child relationship be established so that tickets can be processed and closed properly.

6.3.4 In the event multiple customer trouble tickets are opened within a short period relating to the same geographic location, the DCCC shall notify the SCO desk of the outages. This notification will facilitate the recognition of significant outages.

6.3.5 The DCCC will monitor the customer ticket to resolution, periodically providing the customer with updates from the RCO ticket. When the RCO has been resolved, the NOC will notify the DCCC of the resolution. The DCCC will replicate the information from the Fault Summary tab of the RCO ticket to the customer ticket. The DCCC will notify the appropriate customers the incident has been resolved. If the customer is still experiencing a problem when the RCO has been resolved, the DCCC will reassign the customer ticket back to the appropriate NOC. That NOC will treat that ticket like any other customer trouble ticket and work the problem to resolution.

6.4 Closing Trouble Tickets.

6.4.1 Briefable tickets will be kept within the NOC in a “Fixed” status until they are briefed. Once briefed, the NOC will change the status to “Fixed HD” and send the ticket back to the DCCC within 3 business days for the QA and closure process. Before a resolved customer trouble ticket can be reassigned back to the DCCC for QA and Closure, the NOC technician must ensure it has been properly prepared for closure. See Ref 3.1 for guidance regarding preparation of tickets for closure.

6.4.2 When the NOC verifies the trouble ticket is ready for closure, the Status field in the ticket shall be changed to “Fixed HD.” The Assigned Group field is required for trending, analysis metrics and the crystal reports generated through TMS. All appropriate fields must be filled out correctly before setting the ticket to “Fixed HD”. Note: If a ticket is marked “Briefable” in the Local Priority field, it shall stay in a “Fixed” status until the ticket is briefed at the GNSC Morning Brief. After each briefing the GNSC SCO will remove “Briefable” from all tickets that were briefed. The tickets will then be set to “Fixed HD”.

6.4.3 The DCCC will contact the customer to perform a QA, verifying the customer’s satisfaction with the resolution of their trouble ticket. The DCCC will annotate the name, organization, and telephone number of the individual contacted and who authorized closure of the trouble ticket.

6.4.4 If the customer declines closure of a ticket because they feel the incident is not resolved, the DCCC Shift Lead will contact the NOC Shift Lead. If the NOC believes the incident is a premise issue, the DCCC Shift Lead and the NOC Shift Lead will try to resolve the customer’s issue. If the DCCC Shift Lead and the NOC Shift Lead cannot resolve the issue with the customer, they will escalate the issue to the GNSC SCO desk position by adding the SCO to the conference to allow the SCO to make a final decision.

6.4.5 If the DCCC is unable to reach the customer to obtain authorization to close the trouble ticket, the DCCC will annotate each attempt (including customer name and phone number) in the Action Log field. The DCCC will make two attempts to contact the customer by telephone or email. If there is no contact or response within 48 business hours, the DCCC shall close the trouble ticket at the discretion of the DCCC Shift Lead.

6.5 Net Health Reports. The DCCC will open customer trouble tickets requesting NIPRNet and SIPRNet performance reports. The trouble ticket will be assigned to DISA CONUS Engineering Division, IP Services Branch and/or DISA CONUS IP Tier II. To ensure the trouble ticket gets noticed, the DCCC will call the appropriate Tier II section and advise them of the new ticket. The Tier II section shall create the reports. If problems arise in the creation of the reports, due to Information System malfunctions or the report has special requirements, the IP Services Branch shall coordinate with the DISA CONUS System Admin Branch to resolve the issue. The IP Services Branch shall forward the report to the customer via email, update the ticket with the customer’s email address and date sent, and reassign the trouble ticket to the DCCC for QA and closure. If the customer has questions concerning the report, the DCCC shall conference the customer with the appropriate Tier II section.

6.6 Network Advertisements. The DCCC will open a customer trouble ticket for Network

Advertisement Changes. The trouble ticket will be assigned to the appropriate IP Tier II group. Upon completion of the network advertisement change the IP Tier II technician will update the Status or Resolution Summary field with date of completed action and reassign the trouble ticket in to the DCCC for QA and closure.

6.7 Tactical Mission.

6.7.1 A “master ticket” is created for each mission. The “master ticket” will remain open for the duration of the mission. When a customer contact is received for an Integrated Tactical Strategic Data Network (ITSDN) mission, the DCCC will obtain the mission number and search TMS for the master ITSDN ticket. If the customer is not reporting an outage, but requires information, the DCCC should conference the customer with the TMD. The TMD will work the issue. If the customer contact is made after duty hours, the DCCC shall contact the GNSC SCO. The SCO will determine if the TMD needs to be called in to handle the issue or if the issue can wait until normal business hours. If the customer contact is reporting an outage during the mission, the DCCC shall open a ticket and assign it to the TMD. The DCCC shall call the TMD to notify them that a customer ticket has been generated.

7. Disposition.

7.1. This SOP will remain in effect until cancelled, replaced, or rescinded.

FOR THE COMMANDER:

OPR: RN32

1 Enclosure: Definitions Page

Copy to:

DISA CONUS All

NS7/GS27

DCCC

Enclosure 1

ITEM DEFINITION

Alarm Trouble Ticket

Trouble ticket open due to an INMS or EMS Alarm

Customer Trouble Ticket

Trouble ticket open due to a customer’s reporting of an incident.

Incident

Any event that is not part of the standard operation of a service and that causes, or may cause, an interruption to, or a reduction in, the quality of that service

Initial Fault Identification Making an initial determination of which network or application the incident is caused by

Root Cause Outage –Transport Outage due to a transport anomaly that affects numerous customers Root Cause Outage–Nodes

Outage due to a complete Node Isolation

Normal Hours

Mon-Fri 0730-1600 Central Time

Problem

Unknown underlying cause of one or more incidents

DEFENSE INFORMATION SYSTEMS AGENCY

DISA CONUS

P.O. Box 25860

Scott AFB, IL 62225-5860

� EMBED MSPhotoEd.3 ���

PAGE

_1218458243.vsd DCCC creates trouble ticket for Net Health Report

DCCC assigns trouble ticket to Tier 2

Standard report available?

Tier 2 produces report

DCCC contacts customer for QA

Tier 2 sends report to customer

Tier 2 Updates trouble ticket and assigns to DCCC

DCCC closes AR

Yes

Tier 2 coordinates with RN6 to develop report

No

Net Health Reports

_1282629418.vsd DCCC creates customer trouble ticket

DCCC assigns trouble ticket to NOC

Related to RCO?

NOC troubleshoots customer trouble ticket

NOC assigns customer trouble ticket to DCCC

RCO open?

DCCC updates customer trouble ticket

DCCC contacts customer for QA

Root Cause Outage (RCO) Process

Closes trouble ticket

_1202620807.bin