SOP100-55-02

DISA CONUS Standard Operation Procedure (SOP) 100-55-02 (Rev 1)

DISA CONUS STANDARD OPERATING PROCEDURE (SOP) 10 September 2008

100-55-02 (Revision 1)

OPERATIONS

Global NetOps Support Center (GNSC)

Network Escalation Process and Responsibility

Shared between Tier I and Tier II

1.

PURPOSE. The purpose of this SOP is to describe the policies and procedures that define the interactions between Tier I and Tier II in executing the escalation process in support of Incident Management. This SOP details the processes and procedures required to communicate effectively between Tier I and Tier II support organizations in order to manage and control the Defense Information Systems Network (DISN) and the services managed by the Network Operations Centers (NOC) physically located in the GNSC. This revised SOP provides updates to the POCs listed in the enclosures. This revised SOP cancels and supersedes previous version dated 15 Feb 2007.

2.

APPLICABILITY. This SOP, along with the appendices and enclosures, applies to the following GNSC Tier I NOCs and their Tier II counterparts: Optical Transport Network (OTN) NOC [includes Optical Transport System (OTS) and Optical Digital Cross Connect (ODXC)], Multi-Service Provisioning Platform (MSPP)/Enhanced Synchronous Optical Network (ESONET) NOC, Asynchronous Transfer Mode (ATM)/Black-Promina NOC, Defense Switch Network (DSN) NOC, and Internet Protocol (IP) NOC.

3.

REFERENCES. This document must be used in conjunction with the following references for its effective utilization.

3.1 DISA CONUS SOP 100-55-04 CONUS GNSC Operations and DISN CONUS Call Center (DCCC) Interactions dated 07 Nov 08 (replaced DISA CONUS SOP 06-08) 3.2 DISA CONUS SOP 100-55-01 (Rev 1) GNSC Trouble Management System (TMS), dated 31 Jan 2008 4.

BACKGROUND. The GNSC uses a multi-tiered network management philosophy. There are three functional management areas called “Tiers” within the GNSC. These tiers consist of Tier 0, Tier I, and Tier II. Tier 0 performs customer liaison functions between customers and the upper Tiers. The Tier I structure provides triage Incident Management; the restoration of incidents that are common, frequent, and within their scope and capabilities. The Tier II structure provides problem management, continuous improvement, and process control. The processes cover all configuration, accounting, performance, engineering, security / system administration, and resource capacity management to include planning and regulating the operational network elements. When Tier I has an incident that is outside of their scope and capabilities, the incident is escalated to Tier II for resolution. Defined escalation procedures and criteria’s between Tier I and Tier II are required to maintain stable Incident Management within the GNSC.

5. RESPONSIBILITIES

5.1 Systems Control Officer (SCO). The on-duty SCO is responsible for maintaining and providing situational awareness to stakeholders in-accordance-with (IAW) DISA CONUS Reporting Criteria, Reference 3.3, and has the overall responsibility in ensuring incidents get resolved in the most proficient and expedient manner possible. This is accomplished by making sure the appropriate resources are allotted to the incidents for their speedy resolution. The on-duty SCO will perform this task by ensuring TMS tickets are escalated IAW this document. The on-duty SCO works with the GNSC Tiers to correlate network incidents and ensures correct documentation of TMS tickets IAW GNSC SOP for TMS (reference 3.2.).

5.2 Tier 0 – DCCC. Tier 0 is included in this SOP for informational purposes only. It is the 24x7 help desk or primary customer contact call center where all customers initiate the incident and problem resolution management process. Tier 0 is responsible for fielding the initial call, opening a TMS ticket, gathering all customer information and symptoms, and then assigning the ticket to the appropriate NOC. Upon restoration of service, the assigned NOC will transfer the TMS ticket back to the DCCC help desk, where they will follow up with the customer to ensure all concerns were satisfied and resolved before officially closing out the ticket. This process in handling TMS tickets between DCCC and the various NOCs are outlined IAW CONUS GNSC Operations and DCCC Interactions (reference 3.1.).

5.3 Tier I. Tier I is a collection of technologically oriented NOCs that focus on Incident Management of the operational DISN. Tier I provides 24 x 7 incident management, problem control, and error identification for any network events identified through the element management tools and DCCC escalated TMS tickets.

5.3.1 NOC Hierarchy. The following is the fundamental Tier I NOC hierarchy.

5.3.1.1 Network Controller. Network Controllers are the first “line of defense” in resolving network incidents. They troubleshoot and resolve any incident and document the actions via TMS tickets opened by the DCCC and self generated tickets opened from alarms on local Element Management Systems (EMS) and Network Management Systems (NMS).

5.3.1.2 Shift Lead. The Shift Leads are responsible for Operational Awareness of their network(s) during their shift. They provide guidance and supervision to Network Controllers on their respective shifts. They lead shift operations by prioritizing network incidents and monitoring Network Controllers performance in resolving network incidents. They are the second “line of defense” in the resolution of network incidents.

5.3.1.3 Operations Engineer. The Operations Engineers are the Subject Matter Expert (SME) for their respective technologies within Tier I. They are the highest technical hierarchy for Tier I. Operations Engineers typically work during normal hours of operations (7am – 4pm, Mon – Fri), but are on-call 24x7 so they may be contacted at any time by their respective Shift Leads.

5.3.1.4 Tier I Technology Lead. The Tier I Technology Leads are responsible for the day-to-day operations and management of the various DISN networks and technologies. They provide mentoring, guidance, and management to all personnel within their area of responsibility. They receive the technical evaluations from the Operation Engineers and confirm/validate all possible actions are exhausted and determine if escalation of the ticket is required. Tier I Technology Lead typically works during normal hours of operations (7am – 4pm, Mon – Fri).

5.4 Tier II. Tier II is a collection of technologically oriented teams that focus on problem management, error control, process control, and continuous improvement of the current operational DISN network and limited scoped customer support future planning. Tier II provides 8 x 5 duty day and 24 x 7 on-call coverage of problem management, process control, and escalated incident resolution for any network events identified through the element management tools, escalated TMS tickets, or customer complaints. Tier II personnel identify, troubleshoot, and resolve network problems, escalated incidents, and initiate the process management procedures.

In order to facilitate effective communications, Tier II will post and maintain a comprehensive list of its contacts at T:\Division Info\RN3\RN32\Tier II Contacts to ensure Tier I has a standard location for Tier II POCs to refer problems. This will include Tier II on-call schedules. Once an escalation is received, it is the Tier II POC responsibility to ensure resources are available to work the issue. It is the responsibility of the Tier II Technology POC to ensure the timely resolution and ticket documentation during the life of the Tier II escalation. Tickets will be updated on a near real time basis, every two hours when working active problem management activities. At a minimum all escalated tickets, including those pending a long-term resolution, will contain weekly updates, unless a specific date is identified in the ticket for scheduled action such as an Authorized Service Interruption (ASI) or parts delivery. Tickets will be updated more frequently as needs dictate.

6.

PROCEDURES

6.1 Internal Tier I Escalation. The Tier I internal escalation process of an incident is depicted in the Steps below. At each Step of the progression, the incident’s assigned personnel will exhaust all known capabilities. Before the escalation of a trouble ticket, a review of troubleshooting methods will be completed. However, it is critical that the timeline, in the Escalation Criteria located within Section 7 of this SOP, be followed. The internal Tier I escalation process must be completed before the timeline is expended.

Tier I Escalation Steps:

Step 1: Network Controller to Shift Lead Step 2: Shift Lead to Operations Engineer Step 3: Operations Engineer and Tier I Technology Lead agree to escalate to Tier II based on the Escalation Criteria.

Step 4: On-duty SCO is advised of the escalation.

After normal duty hours, the Shift Lead will contact the Operations Engineer by telephone and in agreement with the on-duty SCO, they will determine if the ticket needs to be escalated to Tier II. If the Operations Engineer cannot be contacted, the Shift Lead and the on-duty SCO will consult and come to an agreement as to if an escalation is required. The Tier I Technology Lead will be advised of the escalation by the Operations Engineer and/or on-duty Shift Lead during normal working hours.

6.2 Process from Tier I to Tier II.

6.2.1 Step 1 – Tier I Process. This is the starting point of communications between Tier I and Tier II regarding a potential escalation. After it is determined the TMS ticket will be escalated based on the Escalation Criteria in Section 7, the Tier II Technician will be contacted as outlined in Section 6.2.2.

6.2.2 Step 2 – Issue escalated to Tier II. This step involves the escalation of the operational incident to the appropriate Tier II Technology POC.

6.2.2.1 Escalations During Normal Duty Hours (7am – 4pm, Mon – Fri). It is the responsibility of the Operations Engineer to ensure the responsible Tier II Technology POC has been made aware of any escalation via voice (either in-person or via telephone) and e-mail (for historical record). See Attachment 2 for a list of appropriate individuals. This is the Tier II IP, Transport, or Switched Services Technology Lead. In all cases of escalation, the respective Tier II Branch Chief and Tier II contract manager will be emailed. Tier I will fully document the escalation event in the TMS ticket and notify the on-duty SCO.

6.2.2.2 Escalations After Normal Duty Hours. The escalations shall be made to the on-call Tier II technician for the applicable technology with a follow-up email to the individuals listed in Attachment 2. It may be necessary for Tier I to call the Tier II Technology POC or the Tier II Contractor Manager for a more expeditious response due to the nature of the incident.

6.2.2.3 Tier II technicians may request more information from Tier I to validate and/or verify the incident before proceeding.

6.2.3 Step 3 – Assigned to Tier II Resource. After determining all necessary information has been provided, the Tier II Technology POC will assign the TMS ticket to a Tier II technician for resolution. In the event that the escalation occurred during other than normal duty hours, the on-call technician is the assigned Tier II resource by default. If the on-call technician has not been able to resolve the issue, the Tier II Technology POC may reassign the TMS ticket as necessary. For issues that require Tiger Team assistance, the Tier II Technology POC will notify the RN54 Branch Chief via voice with a follow-up email containing the details of the issue. The email will be sent to the RN54 Branch Chief, Tiger Team Lead, and the Tier II Contractor Manager. If the RN54 Branch Chief accepts the escalation, (s)he will assign the ticket to a Tier II Tiger Team resource.

6.2.4 Step 4 – Appropriate procedures invoked in Tier II. Once the TMS ticket has been assigned to a Tier II technician, that person will take ownership and update the TMS ticket appropriately, IAW GNSC SOP for TMS, Reference 3.2. The Tier II owner of the TMS ticket will update it with significant findings/events as the status of troubleshooting/resolution progresses on a real-time basis whenever possible. In the event the Tier II owner is troubleshooting from a remote location (i.e., from home over a secure VPN), (s)he will provide status to the appropriate Tier I NOC to have the TMS ticket updated. Upon arriving at DISA CONUS for the start of his/her next workday the Tier II technician will review the ticket and provide updates as necessary.

NOTE: As network-centric warfare continues to become the norm throughout the Department of Defense, the need for status updates and situational awareness continues to grow. Because of this, escalation of an issue no longer precludes the requirement for timely documentations. Leadership, customers, stakeholders, etc., require information in order to effectively plan missions that rely upon the underlying DISN networks for their accomplishment.

6.2.4.1 Step 4a - If the day-to-day Tier II mission is impacted because of an escalation, the Tier II Branch Chief and Tier II Technology Lead shall be notified via supervisory chain.

6.2.5 Step 5 – Ticket Closure. Once the Tier II technician believes (s)he has resolved the incident (s)he will set the ticket to “Fixed” status, fully document all actions taken to resolve the issue, and reassign the TMS ticket back to Tier I. At this point, (s)he will send an email to the Tier I Technology Lead and appropriate Operations Engineer for closure. Upon notification, the Tier I Technology Lead or his/her designated representative will verify that the incident has been resolved and take appropriate action according to GNSC policies and procedures, outlined in GNSC SOP for TMS (reference 3.2.).

6.2.6 Step 6 – Validation by Tier I. Tier I will verify the resolution of the incident returned from Tier II. If the issue has not been resolved, Tier I will document the TMS ticket with the applicable details and transfer the ticket back to Tier II following the procedures identified in Step 2 above.

6.3 Process from Tier II to Tier I – Notifications.

6.3.1 Problem Identification. During routine performance monitoring of the various networks, Tier II occasionally identifies problems (outages, impairments, mis-configurations, etc.). When these problems are identified, Tier II will create a TMS ticket and update the Action Log with the information that supports their findings. Once created, the Tier II technician will determine if the problem is the type typically worked by Tier I or Tier II. If the problem is normally worked by Tier I, the TMS ticket will remain in “Initiated” Status and then assigned to the appropriate Tier I group. The Tier II technician will email the appropriate Tier I NOC Operations Engineer with a courtesy copy to the Tier I and II Technology Leads, providing necessary details surrounding the problem (CCSD, TMS ticket number, etc). The Tier II technician will follow up with a phone call to the Tier I Operations Engineer (NOC shift lead if Operations Engineer is not available) to ensure the e-mail has been received and the issue is understood. Tier I will then begin working the problem as they would any other trouble ticket. If the problem is retained in Tier II, the Tier II technician will open a ticket and ensure the ticket is assigned to Tier II per the procedures listed in section 6.2.3 above.

6.3.2 HAZCON/Alarm Notification: Anytime that Tier II is performing work on DISN nodes such as IOS upgrades, adds, moves, changes, or any other action that has the potential to generate EMS alarms, or adversely affect a DISA customer, Tier II will notify the appropriate Tier I NOC prior to and after completion of those actions.

7.

ESCALATION CRITERIA. Escalations are based on Significant Identifiers assigned to TMS tickets and the Operations Engineer and/or the Tier I Technology Lead not having a clear path to resolving the incident. What this means is Tier I has not identified a root cause and/or a course of action to resolve and fix the incident (i.e., replacement of equipment). Continuing to loop test, working with site personnel to narrow down the problem, etc., does not constitute “clear path to resolution”.

The appropriate Operations Engineer and/or Tier I Technical Lead will evaluate each incident based on the timeline and the incident having a “clear path to resolution”. However, if at any time during the course of troubleshooting Tier I determines the incident is beyond their scope and capabilities, they have the authority to escalate to Tier II.

The escalation timeline and question (“clear path to resolution”) are not hardened rules for escalation. There is an applicable exception, which will delay or prevent an escalation. If the exception applies, the incident will be closely monitored and will be reassessed at every Reevaluation Interval until the incident is resolved or a reevaluation determines the TMS ticket needs to be escalated. The on-duty SCO will be advised of each evaluation and the logic behind the decision. This information will be documented in the Action Log of the TMS ticket.

NOTE: The on-duty SCO has the authority to direct or delay an escalation based on Network and/or Operational Impact of the incident. However, if there is an objection from the Tier I Technology Lead or Operations Engineer, the NetOps Branch Chief or the Assistant Branch Chief will determine if the incident will be escalated.

7.1 Escalation Timeline. The Significant Identifiers that are assigned to TMS tickets are based on definitions outlined in the DISA CONUS GNSC Reporting Criteria (reference 3.3). There are four Significant Identifiers used within the GNSC to categorize the impact severity of an incident: Significant, Major, Minor, and None. These assessments are based on Network and Operational Impact of an incident. Each Significant Identifier relates to an escalation timeline sited in Table 1 below. An impact may change during the duration of an incident so Tier I in conjunction with the on-duty SCO, must continuously assess an incident’s impact until its resolution.

Significant

Identifiers Incident Elapsed

Time (hours) Reevaluation

Intervals

Table 1

NOTE: These timelines are based on the availability of resources (technicians, equipment, and parts) to fully and effectively troubleshoot the incident. The Incident Elapsed Time is based on an accumulation of active troubleshooting (total troubleshooting hours) performed by Tier I. The Incident Elapsed Time stops when: a TMS ticket is placed in a Pending, Deferred, or Referred Status; a commercial vendor (AT&T, MCI, Qwest, etc.) is repairing/troubleshooting an incident; and/or time spent by CACI FSE or FNS FE traveling to installations to assist or repair outages. Any of the above actions will cause a delay in the active troubleshooting of the incident; thus time will not count towards the above timeline. The TMS ticket Action Log field will have to be reviewed to determine the elapsed time. It is the responsibility of the Operations Engineer, Tier I Technology Lead, and the on-duty SCO to determine the Incident Elapsed Time.

EXCEPTION: When the Incident Elapsed Time expires and Tier I thinks more time is required to resolve the incident (i.e., due to the fact they have not exhausted all known common tactics and techniques) the on-duty SCO will be notified of this situation and must agree to keep the TMS ticket within Tier I.

7.2 Significant Identifier Examples:

7.2.1 Minor: A single trunk on a node (be it ATM, Promina, ESONET, or IP) that is out of service or unusable when there are multiple trunks of equal or greater bandwidth that are still up and passing traffic on the same node and there is no customer impact.

7.2.2 Minor: Any type of circuit outage where alternate connectivity is available and is causing no disruption of user service.

7.2.3 Minor: Circuit down to a premise router when the customer advises the issue is not service affecting.

7.2.4 Major: Service related isolations caused by either Transport degradation or outages, i.e., NIPR Node 150 (Service) is isolated because an ESONET (Transport) OC-3 is down.

7.2.5 Major: Backbone circuit issue that is isolating service customers.

7.2.6 Significant: Node isolations that are isolating multiple services at multiple customer locations.

7.2.7 Significant: Any transport outages that isolate multiple technologies without a clear path to resolution. Example: ATM has an OC-12 outage that isolates Ft. Bragg from IDNX, NIPR, and SIPR.

8.

CHRONIC ISSUES. Chronic issues are defined as a series of incidents involving the same Element ID in which the problem has not been resolved. When a chronic issue is identified, a TMS ticket will be opened to track the issue. If the problem is discovered by Tier II, it will be handled in accordance with Section 6.3.1 above. All other problems will be worked by Tier I as they would any other TMS ticket. Chronic incidents will be escalated within the Escalation Criteria of Section 7.

9.

DISPOSITION.

9.1 This SOP will remain in effect until cancelled, replaced, or rescinded.

9.2 This SOP will be reviewed on an annual basis or as needed. As new networks and technologies fall under the Operational Control (OPCON) of the GNSC, this SOP may require updates in order to incorporate different processes. Deviations from the SOP must be fully coordinated between the DISA CONUS Operations and Engineering Division Chiefs or their designated representatives.

FOR THE COMMANDER:

3 Enclosures:

1. Definitions

2. Tier I and Tier II POCs

3. Tier II Hierarchy Enclosure 1

DEFINITIONS

Clear Path to Resolution – A condition that exists when a technician has a high degree of certainty that a problem/incident can be resolved by following a set of prescribed steps. (This is largely a judgment call)

Escalation – The process of transferring a trouble (incident or problem) ticket from Tier I to Tier II to be worked.

Incident – An unplanned interruption to an IT service or reduction in the quality of an IT service. Any event which could affect an IT service is the future is also an incident.

Incident Management – The process responsible for managing the lifecycle of all incidents. The primary objective of Incident Management is to return the IT service to customers as quickly as possible.

Long-term Outage – Any outage that exceeds one continuous week in duration.

Node – A central routing/distribution point of DISN data and/or services. Generally comprised of one managed device, but frequently consisting of several devices connected via transmission facilities.

Problem – The root cause of one or more incidents.

Problem Management - The process responsible for managing the lifecycle of all problems. The primary objectives of Problem Management are to prevent incidents from happening, and to minimize the impact of incidents that cannot be prevented.

Enclosure 2

Tier I and Tier II POCs LTC David McCarthy, RN3 NetOps Division Chief

DSN 770-8653

Mark Heffron, RN5 Engineering Division Chief

DSN 770-8656

Wayne Harrington, RN32 NetOps Branch Chief

DSN 770-9098

CAPT Levi Porter, USMC, RN32 NetOps Officer

DSN 770-9816

Craig Ord, RN51 Transport Branch Chief

DSN 770-9995

Ken Kohlberg, RN52 IP Branch Chief

DSN 770-9054

Rodney Santoyo, RN53 Switched Services Branch Chief

DSN 770-8657

Rickey Wright, RN54 Quality Assurance Branch Chief

DSN 770-9959

System Control Officer (SCO)

DSN 770-8626

Ryan Jackson, DGS Tier I Contractor Manager

DSN 770-9996

Derek Green, DGS Tier II Contractor Manager

DSN 770-9959

Mike Sutton, Tier I Transport Technology Lead

DSN 770-9831

Jason Schneider, Tier II Transport Technology Lead

DSN 770-9952

Mark Tubb, Tier I IP Technology Lead

DSN 770-9977

Bill Million, Tier II IP Technology Lead

DSN 770-9930

Paul Fedro Tier I Switched Services Technology Lead

DSN 770-8714

Dean Fite Tier II Switched Services Technology Lead

DSN 770-9929

Mike Palmer Tier II Tiger Team Lead

DSN 770-8742

Enclosure 3

Tier II Hierarchy Escalation e-mails will be sent to the following individuals.

Transport related:

Craig Ord

Jason Schneider Mike Sutton

Derek Green Ryan Jackson

SCO Desk All other engaged parties IP related:

Ken Kohlberg

Bill Million

Mark Tubb

Derek Green Ryan Jackson

SCO Desk All other engaged parties

Switched Services related Rod Santoyo

Jose Rodriguez

Dean Fite

Kevin Collier

Paul Fedro

SCO Desk All other engaged parties

Enclosure 3

DEFENSE INFORMATION SYSTEMS AGENCY

DISA CONUS

P.O Box 25860

Scott AFB, IL 62225-5860

� EMBED MSPhotoEd.3 ���

PAGE

_1202620807.bin