Atch 16_CITS Archit_NetOps SB.doc

Combat Information Transport System (CITS)

Information Transport System (ITS) Architecture FY09 Version

1 January 2009

1.0 INTRODUCTION

1.1 Purpose

The existing basewide information transport system for many Air Force bases cannot support the future information needs of the warfighter. The proliferation of new information systems and an increasing reliance by the Air Force on information technologies require that the base information transport systems be modernized to meet the needs of the 21st Century. At the same time, budget reductions throughout DoD create a demand for greater efficiency in the way we implement our information and information transport systems.

The Combat Information Transport System (CITS) is a multi-billion dollar Air Force acquisition program to provide fixed-base information infrastructure and network management/network defense (NM/ND) capabilities to meet the wartime, crisis, and day-to-day multimedia information transport needs of MAJCOMs, functional areas, and Air Force Bases. CITS is the Air Force component of the National Information Infrastructure (NII) and the Defense Information Infrastructure (DII). CITS was founded in 1997, when the AF saw the need for a centrally managed and funded Program Management Office (PMO) that would provide the reliable, secure, manageable, affordable, and robust network infrastructure necessary to support all AF network communications over the Wide Area Network (WAN). The CITS charter is to provide centralized Command and Control (C2) and Information Assurance (IA) tools, upgrade existing base backbones with high speed data transport capabilities, upgrade and manage base telephone switches and management systems, provide network management of information systems, and to provide the training, sustainment capabilities, and help desk activities necessary to support those core functions. CITS is responsible for the information transport of all network traffic in the AF, both on the Non-Secure Internet Protocol Router Network (NIPRNET) and the Secret Internet Protocol Router Network (SIPRNET). This infrastructure provides the warfighter and wing command center with full access to real-time C2 information during day-to-day operations and contingencies.

The CITS program encompasses optical cable systems, digital voice/data/video/ sensor/imagery systems, allied support, network management and operations systems, information assurance systems for fixed and initial deployed networks and life-cycle management resources. CITS comprises an integrated set of Commercial-off-the-Shelf (COTS) networking components, as well as commercially available NM/ND software applications that are hosted on COTS computer hardware. The use of commercially proven COTS hardware and software allows rapid acquisition of networking functionality with relatively low risk.

1.2 Direction

CITS is being implemented in accordance with the CITS Operational Requirements Document (ORD) and per direction of the CITS Program Management Directive (PMD) 0026(10).

1.3 Scope

CITS comprises the following major component subsystems:

· Information Transport System (ITS)

· Network Management and Network Defense (NM/ND) and

· Voice Switching System (VSS) and Telecommunications Management System (TMS)

Collectively, these subsystems provide the networking infrastructure, management mechanisms, security features, and associated services (e.g., directory services, billing) for voice, data, video, sensor, control, and other forms of wired and wireless electronic communications at fixed AF facilities (bases, GSUs, etc.) throughout the world. They also include standard interfaces for interworking with DoD and commercial long-haul communication facilities and for interoperation with the communications and networking components of other DoD services.

CITS is comprised of the Information Transport System (ITS), the Network Management and Network Defense (NM/ND), and the Telecommunications Management System and Voice Switching System (TMS/VSS). Although interfaces between the ITS and the NM/ND and TMS/VSS are discussed briefly, only the ITS component of CITS is documented in detail in this document.

The ITS comprises a common, base wide backbone transport network (cable plant, other transmission facilities, and switching components), links from the backbone network to specific end buildings, interfaces to external and internal networks, components required to implement the various transport services (data, voice, video, etc.), and interfaces with the NM/ND. Each of these is discussed in more detail in the following sections.

In general, the ITS does not include distribution facilities within buildings (except for cabling among multiple communications closets) or end-device user attachments. However, the CITS program may procure equipment required to interface existing end-building systems with the ITS backbone network. CITS will provide the basic unclassified backbone and backbone-to-end building transport capabilities for both unclassified and classified services.

1.4 CITS Architecture Framework

Combat Information Transport System (CITS) provides war fighters at fixed Air Force facilities (bases, GSUs, etc.) throughout the world the means to exchange critical mission command and control information with both fixed and deployed DoD elements. CITS also enables communications among core mission-support organizations and, in its ultimate implementation, will support all fixed-facility Air Force communications throughout the world. CITS comprises the base-level, Air Force component of the world-wide Global Information Grid, with integrated transport for classified and unclassified voice, data, video, imagery, control, sensor data, and other information sources. Integrity, availability, security, and other key characteristics of a global network are provided by the Network Management/Network Defense (NM/ND) components of CITS.

Figure 1 (CITS OV-1 Operational Architecture) outlines the high-level operational concept of the CITS Program. The CITS domain comprises all of the fixed-facility components within the grey area of the diagram, i.e., fixed bases (including contingency or “temporary” bases), geographically-separated units (GSUs), etc. [Note: Although the NM/ND portion of CITS may support temporary and GSU locations, the ITS does not.] Although the AFNOSC and MAJCOM NOSCs are located on Air Force bases, they are considered separate functional entities, because of their functional roles and the manner in which they connect to and interwork with other AF and non-AF entities. Note that the NOSC also supports many of the application, database, and ancillary servers for user applications and communications.

As illustrated in the Operational Architecture diagram, CITS enables classified and unclassified mission and support users to communicate with similar users at AF facilities throughout the world. The CITS domain also facilitates the flow of management (network monitoring, control, configuration, security, etc.) data between Network Management/Network Defense (NM/ND) components within a CITS facility as well as between that facility and the MAJCOM NOSCs and AFNOSC. CITS also enables information exchange between fixed AF facilities and other AF units (deployed, mobile, etc.), as well as information exchange with other services and DoD organizations. Finally, CITS provides the mechanisms by which information exchanges with the global civilian information and telephony networks are enabled and controlled.

Note from the Operational Architecture diagram that long-haul, wide area network data transport is actually (and predominantly) provided by the Defense Information Systems Network (DISN), although some use of leased commercial transport also exists. However, CITS provides the mechanisms (gateways, security boundaries, logical circuits, etc.) by which information is exchanged and controlled across the DISN. CITS does not provide wide area data transport, but is a user of long-haul, wide-area data transport services provided by the Defense Information Systems Agency (DISA).

Figure 2 (CITS SV-1 System Interface Diagram) offers a high level description of how the three major components of CITS are related to each other and the systems that interface with them. As noted in the diagram, the ITS portion of CITS does not include the first 400 feet where CITS users and computer systems are typically located. Nor does ITS include the NM/ND components which provides services to ITS. At some locations, the NM/ND components may actually reside within a designated ITS facility.

While the ITS does not include either network or telephone end appliances, the ITS backbone system is being designed so that it can meet all of the high reliability, availability and rapid recovery requirements for both of these systems.

2.0 GENERAL CHARACTERISTICS THAT DEFINE THE BASELINE ARCHITECTURE

Before describing the proposed baseline architecture for the ITS, we outline a number of assumptions and discuss the major technological and topological characteristics that typically define a campus-wide Local Area Network architecture.

First, the CITS program provides full network interconnection to core buildings and those selected non-core buildings with a defined requirement for VoIP service. Furthermore, even if a base does not have a current requirement for VoIP, the ITS backbone should be designed to support the requirement at a future date without requiring a complete system replacement. The ITS portion of CITS is assumed to comprise a predominantly “fixed plant,” ground based system with one or more discrete nodes at which communications are switched, routed, or otherwise transferred via one or more connecting transmission links to other nodes and to end-user devices connected to those nodes. This is a typical configuration for most fixed plant networks, whether for telephone, data, or other communications.

Such networks are typically defined by the characteristics of the switching nodes and their interconnecting transmission links, including:

· Number and distribution of nodes,

· Switching technologies used (to include, for example, circuit switching, frame switching and routing, cell switching),

· High reliability, rapid reconfiguration and overall robustness are important characteristics when a data network is being considered as a host platform for C2 applications and integrated voice data systems,

· Transmission technologies (optical fibers, copper wires, point-to-point radio systems, etc.), and

· Connectivity among nodes and end-user systems (e.g., star, meshed, or bus networks, as well as hybrid combinations and hierarchical configurations).

The geographical distribution, number, and interface requirements for end-user devices influence the selection of a specific switching technology (or technologies) and the overall network topology. Such requirements include data rates and traffic characteristics, as well as reliability, maintainability, availability, and security. Selections are also influenced by local constraints, such as the need to interwork with existing or proposed on- and off-base networks and systems and, of course, by fiscal limitations.

2.1 CITS Definitions

Several definitions are useful before proceeding. First, note that ITS installations are not fully funded and, therefore, cannot include every possible building on a base. Only those buildings that are identified as having mission impacts will initially be connected to the ITS. Such buildings are defined to be core buildings and, as discussed in Appendix A, are further divided into three categories; these are intended to help tailor solutions that are in alignment with the mission content of the facility.

Core buildings are logically grouped into a number of discrete geographical areas or zones. Each zone is served by a single, major switching node that is called an Information Transfer Node (ITN). Physically, an ITN comprises one or more switching components plus any transmission and ancillary equipment required for connections to other ITNs and to End Building Nodes (EBN). EBNs comprise the switching subsystems that are found in each building and that connect end-user equipment (workstations, terminal, servers, etc.) via a local subscriber link to the ITN serving the zone in which the building resides. EBNs also provide local switching among end-user systems. Finally a transit ITN is an ITN that is located in a building or other facility where there are concentrations of transmission systems and distribution equipment, but with no local users or attached EBNs.

The ITS backbone network (or, simply, the backbone) comprises all ITNs and the transmission subsystems (Data equipment, fiber optic cables, copper cables, radio systems) that interconnect them. EBNs and the transmission links that connect EBNs to ITNs are not considered part of the backbone. By definition, each ITN will usually connect to two or more other ITNs (with an average typically greater than two), although, in a few special cases (where extreme local geography or cost considerations override) an ITN may be connected to the backbone via a single transmission link. Thus the backbone is typically (and by objective) a partially meshed network, although fully meshed networks (every ITN connected to every other ITN) may be implemented on bases with only a few ITNs.

Similarly, EBNs will typically have a single physical connection to the primary ITN within the geographical area. EBN to ITN physical links will typically be configured to support a single uplink data connection. However, depending upon the unique requirements identified for each EBN location (e.g. current or planned VoIP service), additional uplink connections to the backbone may be included in the design. The need for additional circuits will typically apply only to selected EBNs, and the added logical uplinks may utilize dual (or even multiple) homed connections to one or more ITNs.

To assist with the design process, four types of non-standard EBNs are defined:

(1) Key, mission-critical EBNs; are EBNs that cannot tolerate the extended failures associated with a cable cut, or even the shorter outages that occur from equipment failures. This applies only to those EBNs with critical, time-sensitive missions. Mission-critical EBNs are candidates for including dual, physically diverse cable connections in the outside cable plant design. Any discussion of the need for a physically diverse path to any EBN should be based on a documented mission need statement for the building as well as a complete assessment of supporting factors such as availability of alternate facilities, backup power, and perimeter security. In most cases, this complete analysis will demonstrate that the physically diverse path will not add sufficient capability to warrant the additional expense.

(2) Other, Important-EBNs; include those EBNs to which the loss of connectivity may otherwise endanger life or property, and could significantly impact the base’s ability to support mission requirements. EBNs with C2VGLAN requirements will likely fit into this category. Dual homed connections to one or more ITNs may be needed to meet the higher availability and reliability requirements, but a physically diverse cable path is not typically necessary.

(3) Very large EBNs; include those for which a single equipment failure might isolate hundreds of users and, perhaps, result in CITS system availability objectives not being met. Again, dual homed connections to one or more ITNs may be needed, but a physically diverse cable path is not typically necessary.

(4) Medical EBNs; include any facility that is dedicated to medical support functions and some mixed-use facilities with significant medial support operations.

Note also that dual homing has two different implementations. In its broadest interpretation, dual homing includes installation of separate fiber cables, each of which is routed to a different ITN. Less costly configurations are also possible, depending on what level of service the design is trying to achieve. For example, separate fiber pairs in the same EBN cable sheath can be terminated on different network modules within the same ITN, to achieve improvements in equipment availability. Another option is the use of separate fiber pairs in the same EBN cable sheath, each of which is connected to a different ITN by patching one of those pairs through an inter-ITN trunk to another ITN (again, to lessen the impact of equipment failures on overall system availability).

These concepts and definitions are illustrated and summarized in Figure 3. This diagram provides an example of a typical partially meshed topology for a backbone network, as well as subscriber access connections to the backbone.

Finally, the medical enclave, comprises dedicated and mixed-use facilities with at least 25 percent medical personnel and/or more than 40 computers or servers for medical support. These facilities are usually the hospital, clinic, and any other buildings (pharmacy, dental clinic, medical laboratories, medial warehouses, etc.) that are covered by the Law of Armed Conflict (LoAC) and subject to those laws and restrictions. The medical enclave meets multiple mission requirements for the medical community. Medical buildings will be physically and/or logically homed to the main medical enclave (MME) (usually the hospital) at which the majority of medical servers are located and/or from which communications to the external (off-base) medical community are established. Current law requires that communications among facilities in the medical enclave be segregated from other base communications. The critical nature of the medical enclave requires that its communications be robust and reliable. The Health Insurance Portability and Accountability Act of 1996 (HIPAA) adds a requirement to ensure medical security and privacy when transporting medical information. At the same time, LoAC prohibits medical facilities from supporting command-and-control and other mission traffic. Most or all of these objectives can be satisfied as illustrated in Figure 3. Connections among medical facilities can be provided via dedicated point-to-point cables, via separate fibers in existing cables, by transport over the ITS backbone via dedicated VLANs or in conjunction with a VPN, and (not illustrated) via dedicated time-division multiplexed channels (e.g., SONET tributaries on the ITS backbone). The MME can be dual-homed via physically separate cables to redundant networking components within a single ITN. LoAC has been interpreted as prohibiting connections from the MME to multiple ITNs, since the MME can, in principle, be used as an alternate path for non-medical traffic.

2.2 Alternative Backbone Transport Media

Because of the cost associated with installing new fiber optic cable, there are some circumstances where alternative transport media should be considered. These locations include, but are not limited to backup and/or secondary cable routes that are added for reliability purposes and excessively long or difficult cable routes. Figure 3, which defines the relationships between ITNs and various EBNs serviced by the ITS architecture, also addresses the possible application of alternative backbone transport media. The technical alternatives that may be considered on a case-by-case basis include:

· Point-to-Point and Point-to-Multi-Point radio systems and

· Line-of-Sight optical systems

Depending on the installation location, trunk encryption devices may be required. A cost benefit trade-off between the options may be required to help select the “Best Value” option.

2.3 Classified Transport

In the past, classified transport in the base environment employed point-to-point link encryption. Point-to-point link encryption is wasteful of fiber bandwidth and the point-to-point link is subject to complete failure if a fiber is broken.

Network layer encryption operates at the IP layer, which has been designated as the convergence layer for net-centric operations. The convergence of encrypted classified data with unclassified data makes more efficient use of fiber bandwidth and provides higher availability than a dedicated point-to-point link. With the High Assurance Internet Protocol Interoperability Specification (HAIPIS), there is a shift from link-layer encryption to network-layer encryption. Convergence of encrypted classified transport (at multiple levels) with unclassified transport on a common black core on a base network is analogous with the planned implementation of the GIG for wide area transport.

The ideal, of course, would be the use of multilevel secure user devices with end-to-end encryption and authenticated, per session encryption keys. Since such devices are not yet available at reasonable cost, the current baseline architecture prescribes encryption with Inline Network Encryption (INE) devices with transport over the base (ITS) network for connections between classified enclaves and the SIPRNET Service Delivery Point (SDP).

2.4 Interfaces to Other Local and External Networks

Finally, the architecture is influenced by local/external networks and communications systems with which ITS must interconnect. These include interfaces to:

· A variety of existing unclassified and classified data networks,

· The Network Management and Network Defense (NM/ND),

· NIPRNET (via NM/ND),

· SIPRNET,

· Voice switching system, and

· Various legacy video, sensor and imagery systems.

2.5 ITS Design Constraints for VoIP

In order to support VoIP systems over CITS, the ITS Layer 3 network transport architecture and protocols may need to be designed to support an even higher level of availability and reliability equal to that being achieved today by circuit-switched telephony networks. CITS ITS Architecture Design should follow the guidelines contained in the DISA document titled: Appendix 3, Generic Switching Center Requirements (GSCR). Based on an interpretation of basic telephony requirements from other sources, this document addresses the associated reliability and availability requirements for a Layer 3 transport network that is designed to handle VoIP and/or C2VGLAN systems.

As examples of the higher availability/reliability requirements that are mandated:

· Partial outages effecting more than 64 phones are limited to less than 3.0 minutes/year; thereby affecting the number of EBN uplinks required.

· Disruption of stable, active calls due to hardware failures affecting trunk circuits are limited to values which will affect the capacity of ITN Backup Power.

Based on these requirements, an ITS Backbone design that will support C2 Voice Grade LANs (C2VGLAN) systems will require a higher degree of network redundancy and UPS power may be included for any LAN equipment or LAN segment supporting more than 64 users. It further requires use of network CoS/QoS protocols and effectively limits bandwidth occupancy (number of IP phones) over ITS Backbone network segments.

The high availability and maintenance of calls (with no more than 2 seconds down-time due to equipment or link failures) is often achieved by use of highly redundant, dual collapsed backbone architecture for connecting ITNs to the SDP/DCO. In addition, any LAN access switches supporting more than 64 users will require dual logical connections to the ITS backbone distribution switches. This may impact the number of fibers needed in the backbone OSP cable design. All network equipment requires UPS power that can provide 2 or 8-hour backup power depending on whether ordinary C2 or Special C2 users are located on that segment.

An example of a dual collapsed backbone hardware configuration is shown in Figure 4. In this figure, each EBN with more than 64 IP phones may require a second logical connection to another ITN to meet the availability requirements. The backbone trunk fiber count may also need to be re-sized to support the dual connections for all buildings that will either now or in the future service more than 64 IP phone users.

(NOTE: This also includes Core 4 buildings that will need to be dual connected to the backbone network if their size warrants.)

The ITS backbone network should be designed to meet basic CITS reliability / availability requirements first. Buildings that require immediate IP phone service may necessitate that the design be augmented to meet the additional requirements in GSCR Appendix 3. Buildings for which IP phone service will not be required any sooner than two years need not meet the additional network requirements such as dual connectivity and UPS. However, the OSP cable system should be capable of supporting the C2VGLAN architecture in the future.

3.0 ITS BASELINE ARCHITECTURE (CIRCA 2006)

3.1 Overview

In past years, the baseline architecture for CITS installations has been based on Asynchronous Transfer Mode (ATM). ATM was considered to be the only technology that satisfied all of the requirements of the CITS ORD, while doing so at a reasonable system cost. Other technologies, SONET in particular, were judged capable of meeting CITS requirements, but were generally considered to be too expensive.

Recent changes in transport technologies have resulted in a new direction for the CITS architecture as described in this document. Such changes include:

1. Large decreases in the cost and size of SONET equipment,

2. Enhanced SONET capabilities that are generally summarized in the term “Next Generation SONET (NGS),”

3. Movement of networking equipment vendors away from ATM in the enterprise and campus markets, as well as diminishing vendor support for “technology refreshment” of existing ATM components and software,

4. Reduced training (and interest) relative to ATM networking, as well as a perception that ATM is more complex to configure and maintain,

5. Additional emphasis within the DoD community for convergence on an IP-based network architecture,

6. Increasing availability and much lower costs for Gigabit Ethernet, line-rate routers, etc., combined with growing support for “quality-of-service” and new applications (e.g., VoIP) in IP and Ethernet based networks.

As a result of these changes, ATM no longer remains a viable component of the CITS architecture. However, CITS continues to consider the use of alternative networking technologies and their ability to support mission critical applications. Specifically, CITS has added alternative (non-fiber optic) media as a backbone transport option and is addressing new and evolving requirements for increased voice and data integration. Wireless systems (both on the backbone and as an alternative for the first 400 feet) are fully embraced by the CITS ITS Architecture.

The architecture for the CITS ITS now takes a somewhat different approach (relative to prior architectures) to providing a common transport infrastructure for Air Force bases. Instead of defining a single technology as the basis for information transport on all bases, the new approach allows flexibility in selecting and applying a transport technology (or technologies) to be used on each base. The intention is to promote cost-effective solutions that are tailored to the unique requirements of each base, while ensuring that those solutions satisfy the principles and direction of the CITS ORD.

The ITS architecture also allows a Gigabit Ethernet only approach (with restrictions as discussed later) or a hybrid Gigabit Ethernet / SONET frame transport network if circuit requirements warrant the technology.

The following paragraphs describe each of these in some detail.

3.2 ITS Alternative Architectures

3.2.1 SONET-Based ITS Architecture

SONET, or Synchronous Optical Network, is the standard for long-haul optical transport of telecommunications traffic in North America. SONET was developed as a standards-based technique for multiplexing of synchronous and near-synchronous digital services and for transporting those services over optical fibers. Unlike earlier time division multiplexing systems, SONET offers a number of advantages, including:

· Enhanced operations, administration, maintenance, and provisioning (OAM&P) services,

· Redundant and survivable ring architectures with rapid service restoration,

· Timing distribution for synchronous networking,

· Compatibility with other transport services such as ATM, TDM, etc.

· Interoperability among equipment from various SONET vendors.

It is not the purpose of this document to describe SONET in detail. The reader is referred elsewhere for this information. Suffice it to say that SONET, as originally defined, was a time division multiplexing and line transport system for optical fibers. It included options for implementing various physical network topologies, ranging from point-to-point links to two and four wire ring networks with high-speed protection switching. It also included mechanisms for adding and dropping tributary (i.e., lower speed TDM or unstructured data streams) services. Until recently, it was costly to implement and provided limited flexibility and poor efficiency for handling non-circuit or non-connection oriented requirements (e.g., connectionless frame transport such as Ethernet and IP).

“Next Generation SONET” (NGS) builds on the SONET foundation to enhance its applicability to enterprise networks. NGS seeks to lower overall system costs by greater integration of SONET components (more functionality in smaller and lower cost chassis), by providing more efficient and flexible support for frame-based transport services, and by simplifying and enhancing the management and control of SONET based services. Unfortunately, many of the standards for NGS are still being worked, and a commonly agreed definition of what constitutes NGS remains to be developed. 2006 CITS Architectures will not typically include NGS-based ITS designs; however, integrated SONET / Ethernet designs have been implemented in instances where rapid restoration of failed Ethernet links was a design requirement.

Wavelength Division Multiplexing (WDM) greatly extends the capabilities of SONET and NGS systems by multiplying the number of such systems that can be supported on a single pair of optical fibers (or even a single fiber). Prices for WDM compatible SONET/NGS equipment are dropping rapidly (though still quite high), while the capabilities for WDM multiplexing and switching are increasing.

Within the last few years, SONET/NGS vendors have begun to offer equipment with features and prices that make SONET highly competitive. SONET, in conjunction with Ethernet, has always been capable of satisfying the requirements defined in the CITS ORD; however, the cost of SONET equipment was generally prohibitive. Since this has changed, SONET is now a viable component of the CITS architecture, however NGS-based architectures are still considered an exception unless driven by a local base requirement.

WDM, especially Dense WDM (DWDM), based architectures remain expensive for the campus/base area network (where fibers are relatively inexpensive to install). Therefore, WDM and DWDM based architectures remain beyond the scope of a typical CITS installation. Recognizing, however, that WDM prices may drop rapidly, the CITS architecture must also support possible future transitioning to WDM operation. Current requirements for single mode fiber installation are intended to address this provision.

Figure 5 [CITS SV-2(c)] provides a notional configuration for a SONET/NGS/WDM-based CITS architecture. Depending on the size of the base and number of ITNs, a SONET-based ITS will comprise one or more SONET rings, configured using a subset of the optical fiber links that make up the partial mesh backbone network. To eliminate single points-of-failure, each ring in a multiple ring backbone network will interconnect to at least two independent points on every other ring to which it connects. Those interconnection points will enable adding-and-dropping individual digital streams and/or tributary exchanges between the backbone rings.

The SONET multiplexers in ITNs (which may consist of one or more physical chassis) must be capable of handling structured and unstructured digital data streams ranging in rates from DS1 (possibly fractional DS1) through OC-48, and also be able to produce aggregate steams up to OC-192 (~10Gbps). These capabilities may be integral to a single unit, or implemented by insertion of different interfaces and/or line modules. The ITN SONET multiplexers should also support WDM (at least at the OC-48 and OC-192 rates) either by changing individual line modules or by wavelength selection within a module. The resulting multiple wavelength carriers can then be combined via either an integrated WDM multiplexer or an external device.

When an EBN requires significant circuit transport services (e.g., multiple T1 lines), a SONET multiplexer may be installed. However, the SONET multiplexer selected for such an EBN may be a much smaller device than that used in an ITN, since it does not need to support higher transport rates (e.g., OC-48, OC-192) or WDM features. Typically, an EBN multiplexer will aggregate multiple T1 or fractional T1 signals for transport over a lower rate SONET link (e.g., OC-3, OC-12). As illustrated in Figure 6, connections between such EBNs and an ITN may be implemented as individual point-to-point links, or as an EBN ring which combines several uplinks from multiple EBNs into a single ring (logical ring implemented as a physical star network with the ITN as the hub). The latter must interconnect with the main backbone ring (preferably by time slot exchange within the backbone SONET/NGS multiplexer).

Frame transport services (also illustrated in Figure 5) are provided by GE-capable switch-routers that are interconnected with the SONET backbone multiplexers. Their inter-switch trunks (GE or NxGE) are carried by the SONET backbone between ITNs. Links to individual EBNs are provided in a manner similar to downlinks in previous architectures (e.g., GE or FE downlinks to building switches), except that, in those EBNs with SONET multiplexers, the downlink may be carried over the aggregate SONET stream. Note that the frame switch-routers can be individual, stand-alone units or integrated into the SONET or NGS equipment. In addition, the inter-switch links can be point-to-point connections that ride on the aggregate backbone SONET stream, or logical links implemented through NGS features such as “resilient packet rings” (RPR).

In general, data network engineers try to implement relatively “flat” campus/base local area network topologies, such at the redundant collapsed backbone configuration of Figure 4. The flat topology offers advantages relative to data throughput, delay, routing, etc. Unfortunately, the SONET physical ring topology is inconsistent with this design objective; that is, implementation over a ring network of a logically meshed or collapsed backbone network requires use of multiple tributary streams on many links. Since many of today’s SONET/NGS multiplexers map a GE data link into an OC-48 tributary stream, the implementation of multiple OC-48 links requires an aggregate OC-192 stream (or use of multiple wavelengths). Unfortunately, OC-192 modules, as well as OC-48 modules designed for DWDM, still tend to be very expensive. This leads to the next architectural alternative: the SONET/Gigabit Ethernet overlay configuration.

3.2.2 Hybrid SONET/Gigabit Ethernet Overlay ITS Architecture

Previous ITS Architecture documents described an ATM/GE overlay architecture, in which additional fibers in the backbone cable plant are used to implement separate, but interconnected and interworking, ATM and GE backbone networks. The same approach can be applied usefully to a hybrid SONET and GE overlay configuration, as illustrated in Figure 7 [CITS SV-2(d)]. That is, separating the GE frame-transport network from the SONET circuit-transport network results in several significant cost-performance benefits. The SONET rings carry only backup frame-transport connections (which can be lower data rate tributaries) and circuit-transport services (which, for most bases, results in relatively small aggregate bandwidths). Consequently the requirement for OC-192 backbone links is reduced to OC-48, OC-12, or even OC-3, with a large decrease in the cost of implementing the SONET network and no appreciable increase in the cost of the GE overlay network.

In a similar manner, EBN links can be implemented in two components: A GE downlink to a local frame switch(s), plus a low aggregate bandwidth SONET downlink which carries only circuit-transport services. This allows lower cost SONET equipment to be used in the EBNs without significantly affecting overall capabilities.

With sufficient meshing of the GE switch-routers, the backup connections to the SONET rings can be largely eliminated. It can be shown that, in many networks, these are not necessary to meet RMA requirements, they increase network cost and complexity, and they increase management complexity. Thus only the minimum number of interconnections required to ensure communications for network management and similar systems need be implemented.

In addition, the hybrid SONET/GE overlay architecture allows for other cost reductions, such as the use of smaller SONET multiplexers, minimally configured multiplexers, or possibly none at all in those ITNs that have few or no circuit-transport requirements and that anticipate no future requirements for circuit-transport services.

3.2.3 Gigabit Ethernet ITS Architecture

As part of the CITS program, most bases are choosing to implement, as a minimum, a stand-alone Gigabit Ethernet as their primary backbone transport service, with no circuit transport switches or multiplexers. These bases typically identify few or no requirements for circuit-mode transport beyond standard telephone service, they see no need for implementing and maintaining a complex SONET infrastructure, and they may, in fact, have already initiated efforts to place some circuit services onto the GE network. A notional GE-only ITS is illustrated in Figure 8 [CITS SV-2(e)], which shows all frame-mode data traffic carried via a fully routed GE backbone over a partial mesh topology. As discussed previously, a dual collapsed backbone architecture is also an alternative GE backbone solution. Telephone services are provided via a centralized circuit switch (with some VoIP services riding the GE backbone). In addition, a very small number of point-to-point connections are implemented via the use of fiber or copper cable for DS1 connections (such as encrypted data streams or VTC services).

In the past, the CITS Program Office and AFCA have maintained that GE-based circuit transport services do not yet satisfy the requirements defined in the CITS ORD. In particular, it was not clear that networks using frame-based transport could provide either real-time service guarantees or meet the stringent delay characteristics required for many circuit-mode services. This is particularly a concern should the frame network be stressed by severe traffic overloads (e.g., by denial-of-service attacks). Hence, on most bases, a GE-only backbone is not considered to be a satisfactory approach for the ITS. However, it may be considered on a case-by-case basis, with a request by the Base/MAJCOM and appropriate approval from the program office and/or AFCA.

In addition, USAF is currently trying to standardize techniques for convergence of services, particularly voice services, over a frame-switched IP network. This action is in response to efforts at a number of AF bases to install Voice-over-IP (VoIP) systems. These systems have, to some extent, relied on vendor-proprietary solutions and may not be interoperable with similar systems from other vendors; consequently, Air Staff recently placed a hold on additional VoIP installations . The desired approach is one in which the converged services will be based on interoperable standards and equipment.

Convergence of services on the (IP) frame-switched network is currently thought to be the future of communications. While the technology is not yet completely mature, does not yet satisfy all transport requirements, and sometimes relies on vendor proprietary solutions, it is maturing rapidly. Consequently, any frame-based or hybrid SONET/frame-based network architecture must include a number of features required for future implementation of IP-converged services. Specifically, CITS networks must support some level of end-to-end “Quality-of-Service” (CoS) for IP traffic . This means that all frame and circuit switches used in a specific CITS network implementation (as well as frame switching components of NGS SONET equipment) must support IP and/or Ethernet Quality-of-Service mechanisms, and interwork properly. It also means that all transport components must work together in a seamless manner that allows end-to-end IP traffic to be prioritized according to service or administrative requirements and transported or rejected/dropped according to assigned priorities and traffic levels.

Note that the general QoS requirements in the CITS ORD encompass frame-mode services. Integration of mission-critical classified services over the backbone network requires support for precedence and priority handling of such services. Similarly, future implementations of VoIP, streaming-mode video over IP, and other non-classified, non-critical services will work well only if the infrastructure provides suitable QoS mechanisms. Therefore, all frame-mode switches, routers, switch-routers, etc., used on the CITS backbone and within the EBNs must support some level (TBD) of QoS and interwork appropriately with other QoS devices/mechanisms across the ITS.

The QoS requirements typically extend from user-to-user across an individual base’s ITS for those devices or applications that are capable of requesting QoS support from the network. However, most users or applications do not yet request QoS services from the network, but many still require (static) priority handling by the network. For those, the QoS requirement can be considered to extend across the ITS from network ingress to network egress. NMS switch administration, end user port configurations for precedence marking/remarking, frame acceptance/dropping, queuing strategies, etc. are some typical examples of managed QoS requirements. Managed QoS configurations can be based on port IDs, MAC addresses, IP addresses, IP or TCP ports, and a number of other networking parameters.

All network components (layer 2 and layer 3 switches, routers, etc.) must support standards-based mechanisms for signaling QoS information to the other components in a path and for consistently acting on the information thus passed. This can be a challenge in a multi-vendor environment where queuing approaches, priority control concepts, etc., are not generally standardized and may vary subtly.

Where a base has either a current or planned requirement for VoIP systems, it may be necessary to reassess the backbone cable plant design to insure that there are adequate fiber counts to meet the VoIP / C2VGLAN requirements for availability, redundancy and recovery; as well as the spares.

The Secretary of the Air Force has recently directed that as of October 2003, all networking equipment ordered for installation anywhere within the Air Force must be capable of supporting IPv6 network protocols. The technical details on how this directive will be implemented by CITS are provided in the ITS BPD, which is updated annually. In general, network vendors that will be supplying new hardware for purchase after 1 October 2003 must certify that they have a roadmap for meeting IPv6 requirements.

3.3 Topology of the ITS

3.3.1 ITS Fiber Optic Cable Plant

Single-mode optical fiber cables comprise the great majority of backbone and local subscriber links in the ITS. Single-mode fiber is the only medium considered to possess the high bandwidth-distance performance characteristics required to satisfy both immediate and future requirements for data transport within and across the base environment. However, multimode fibers may be allowed when cables are reused and considered as supplements to single-mode fibers on new ITN-EBN subscriber links (when technically feasible and cost-effective). For new cable installations, the baseline architecture for the ITS calls for:

· A maximum of 36 single-mode fibers per link between ITNs and

· 12 single-mode fibers per local subscriber link from an ITN to an EBN; or,

· Optionally, a maximum of eight single-mode and four multimode fibers per local subscriber link, and

· Optionally, up to 24 single-mode and 12 multimode fibers per ITN-ITN trunk, where distances are 550 meters or less between ITNs.

These fiber counts were selected as a compromise between the objective integration of services and the need for a transitional period during which various legacy systems must be supported, and as a compromise between cost and reliability/availability considerations for spare fibers. Sufficient fibers are provided on the backbone links to support any reasonable parallel configuration of SONET, and frame-switched networks.

As outlined above, four multimode fibers can be included in a hybrid EBN cable configuration, and up to 12 multimode fibers between ITNs, although it is expected that few, if any, bases will actually install additional multimode fiber.

Where determined to be cost-effective, hybrid cables can accommodate a combination of SONET, and frame transport mechanisms on EBN links, as well as other, legacy transport applications.

Note:

Those bases that have either a current or planned requirement for VoIP systems and need a C2VGLAN capable backbone may need to investigate backbone cable requirements more carefully. While this Architecture Document does not attempt to define a specific solution, it is recognized that the additional availability, reliability, and recovery requirements may necessitate additional design features.

3.3.2 Physical Network Topology

The physical topology chosen for the ITS is a partially meshed backbone with (mostly) single subscriber links from the backbone to the EBNs (see Figure 3). The rationale for the meshed backbone is the high availability objective specified in the CITS ORD, and the requirement for redundant transmission links between each ITN and the remainder of the backbone. That is, multiple physically independent links (a minimum of two) provide alternate paths that are capable of accommodating link failures or regions of traffic congestion within the backbone network.

While it might appear that two links would suffice everywhere (i.e., a topological ring network), a meshed network designed according to established traffic engineering principles more effectively utilizes the capacity of both transmission and switching subsystems, while affording an extra degree of redundancy for enhanced availability. On the other hand, a fully meshed network would be excessive and unnecessarily expensive on all but the smallest of networks. Here too, it is recognized that those bases that have either a current or planned requirement for VoIP systems and need a C2VGLAN capable backbone may need to investigate backbone cable requirements more carefully.

Where the installation of multiple optical fiber links (even two) is prohibitively expensive, alternatives to optical fiber cables, such as point-to-point radio or optical systems, must be considered. Such links may be economically and technically feasible. If necessary, these alternative links may have lower capacities than the primary link. In exceptional cases, connecting an ITN without physical diversity will be permitted only with explicit MAJCOM and AFCA coordination and approval.

Redundant subscriber links are desirable from the point of view of survivability and availability. However, it is considered economically infeasible to implement two or more links from every EBN located on a “typical” Air Force base, where the number of EBNs may exceed 100. Thus it is expected that most EBNs will be singly connected to one of the ITNs on the backbone network. Certain mission critical buildings, or buildings with large user populations (that do not already host an ITN), should be considered candidates for dual homing to multiple ITNs. On small bases with only a few ITNs, it may be necessary to dual home one or more of the larger EBNs to meet ORD requirements for availability. Again, these are local design issues.

3.3.3 Logical Network Topologies

Logical network topologies for the SONET, and frame-switched backbone networks are overlaid on this partially meshed, physical topology. The logical topology for a SONET configuration is a ring or multiple rings implemented on top of the underlying cable plant; this logical topology may or may not fully use the partial mesh configuration of that cable plant. Finally, the frame-switched network may also follow the physical configuration, as shown in Figure 9, with appropriate Layer 2 and Layer 3 routing protocols acting to prevent routing loops. Note that the frame-switched network need not completely parallel the SONET components of the backbone. However, all traffic on the network must have access to diversely routed paths between ITNs.

Other logical network topologies for the frame-switched network can also be overlaid on the partially meshed physical configuration of the backbone network. As illustrated in Figure 5, a popular implementation of Gigabit Ethernet networks uses a dual switch, redundant collapsed-backbone network with a second layer of distribution switches or switch-routers. The distribution switches are dual-homed with an uplink to each of the collapsed-backbone switches; the secondary link may be provided via a tributary link riding on a SONET backbone. EBNs are connected, in turn, to the distribution switches.