Attachment 0002 CEG Design Document.docx

Unclassified 26 Apr 2019 Version 1.6

Revision HistoryIEA/IE5 6910 Cooper Ave Ft Meade, Md

Commercial Ethernet Gateways (CEG) Design Document Version 1.6

26 Apr 2019

Version Number

Description of changes

1.0 1.1

1.2 1.3

1.4 1.5

1.6 21 Sept 2018 27 Sept 2018

1 Oct 2018 1 Oct 2018

2 Oct 2018 5 Feb 2019

26 Apr 2019 Original Draft Changed the Threshold MTU of NNI from 4472 to 4400. Added section on E-Line Redundancy. Clarified access to JRSS for E-line and Layer-3 VPN. Also pointed out that customers connecting directly to CSP for Layer-3 VPN may have option of using TDM interfaces. Made many minor editorial changes Made many editorial changes and clarifications Updated document show classification as Unclassified. Spelled out any undefined Acronyms and provided missing definitions. Crossed out some VLAN numbers from VLAN tables so document can released in RFI. Removed all encryptor vendor names.

Made many editorial changes Section 2.1: section added that not all CSPs can be expected to support both E-Lines and Layer-3 VPN on same NNI; therefore separate NNIs for E-Lines and Layer-3 VPN is acceptable for this design.

Section 2.1.1: clarified that Customers submit CAPs.

Section 2.1.2: clarified that J-PE does strict route filter ingress on J-PE and that J-PE drops any default routes received from CSP.

Section 2.4: Customers are responsible for deploying Unclass Converged CE routers (Not the CSP) Figure 2-6: Clarified who owns/manages encryptors Added section 2-10 which clarifies CSP owned aggregation devices at off-campus locations Section 2.3.1 change objective MTU from 9100 to 9000.

Introduction

The world is quickly transitioning away from Time Division Multiplexing (TDM) technologies and migrating to Ethernet. The DoD CIO office directed the entire DoD to optimize communications to Ethernet based services to reduce TELCOM costs. See DoD CIO memo (subject: circuit optimization, dtd 5 May 2016) for details. At the same time, point-to-point commercial leases are typically taking us 6-9 months to procure. This is unacceptable.

DISA urgently needs an enterprise solution to efficiently aggregate Ethernet circuits from a Commercial Service Provider (CSP) to get away from the existing point-to-point model. Virtualization using Virtual Local Area Network (VLAN) (802.1Q) between the DISN Joint Provider Edge (J-PE) router and the CSP PE routers over a high capacity link is the Commercial Ethernet Gateway (CEG) solution that this document will explain and provide the design details. Virtualization of CEG solution provides many benefits to include:

-Reduce equipment costs (estimated 20:1 reduction in Ethernet ports on J-PE routers)

-Reduce number of tail leases (estimated 20:1 reduction of tail leases on DISN side)

-Provides a DISN on-ramp for Off-net customers to access DISN services via commercial Ethernet services. This solution will significantly reduce the DISN provisioning timeline because the physical path is pre-built (only VLANs need to be added on J-PE). The DISN will not be required to install new circuit(s) or equipment.

-DISN trunks (J-PE to J-PEx) can also be provisioned across CEG solution (assuming we can get an acceptable performance guarantee.)

See Figure 1-1 for a high-level overview of how Off-net customers can access DISN services from the CEG.

Figure 1-1 CEG High-Level Overview Design Details

E-Lines and Layer-3 VPN peerings:

The high capacity trunk between the DISN PE and the CSP PE is referred to as the Network-to-Network Interface (NNI). There are (2) types of Multiprotocol Label Switching (MPLS) services that will be extended from CSP to the DISN J-PE router over the NNI. One is E-line and the other is MPLS Layer-3 VPN. An E-Line (also referred to as Ethernet Private Line) is a service type for connecting exactly (2) Ethernet end-points with one another. E-Lines and Layer-3 VPN CAN both be present on the same NNI. For example, an NNI trunk with (20) active VLANs can have 18 E-Lines coming from CSP and (2) Layer-3 VPN Peerings. See Figure 2-1 for a high-level diagram of NNIs to CSPs. Not all CSPs can be expected to support both E-Lines and Layer-3 VPN on same NNI; therefore separate NNIs for E-Lines and Layer-3 VPN is acceptable for this design.

Figure 2-1 Connecting to CSPs over NNIs E-Lines over NNI from CSP (These are just tail circuits separated by VLANs) This is expected to be, by far, the most common type of VLANs from the CSP. In most cases, the off-net customer just wants to get to a DISN service via an economical E-Line service from a CSP. As long as the CSP has one or more Ethernet NNIs with DISN J-PE routers, this is a viable option. Customers connecting to J-PE via E-Line in this way can access any of the DISN IP services, like Private IP, NIPR, SIPR (via SAM Architecture), Pseudowire, NFG, etc.. With E-lines from the CSP, the DISN is not implementing any protocols or exchanging routing with CSP. This is similar to any tail circuits we terminate today on DISN PE routers. It is configured the same as if it were an 802.1Q trunk to one of our customers. See Figure 2-2 for a high-level diagram of this option.

The E-Line option provides connection to the NIPRNet Virtual Routing and Forwarding (VRF) and requires Connection Approval Process (CAP) Approval. Customers submit CAPs to DISA. Connection to the JRSS is also possible using this method although a layer-3 VPN solution to access JRSS may scale better, because many customers can access JRSS via the same VPN.

Figure 2-2 E-Lines from CSP over NNI

MPLS Layer-3 VPN Peering over NNI to CSP:

MPLS Layer-3 VPN peering is expected to be a less common type of VLAN from the CSP. In this case, we are External Border Gateway Protocol (E-BGP) peering a Layer-3 VPN on the DISN network with a Layer-3 VPN on CSP’s network. Customers that connect directly to a CSP’s Layer-3 VPN service can connect with a TDM interface or an Ethernet Interface, assuming the CSP is still supporting TDM interfaces. A couple of our customers have existing large MPLS Layer-3 VPNs on a CSP’s network. These customers are looking to connect their existing Layer-3 VPNs on the CSP’s network to the DISN. Technical details of the peering relationship are below:

a. Internet Engineering Task Force (IETF) RFC 4364 Option 10a VRF-to-VRF (This is the most secure method to peer with CSP for Layer-3.)

b. No MPLS labels exchanged between DISN and SPs.

c. Each VPN/VRF is on separate VLAN between DISN and Commercial SPs.

d. Separate E-BGP peering on each VPN.

e. Strict BGP route filtering will be done on each E-BGP session. (Done ingress on J-PE)

f. No default routes accepted from CSPs. (J-PE drops default route. CSP doesn’t drop)

g. No connection directly to NIPRNet VRF via this method. (E-Line only)

h. This method can connect directly to JRSS.

Figure 2-3 MPLS Layer-3 VPN peering with CSP over NNI VLAN Assignments:

The VLAN assignments on the NNI interface to the CSP are the same as they are for any customer interface. Table 2-4 summarizes the VLAN standard. See DISA standard in separate document for details.

Table 2-4 VLAN Assignments for NNIs on CEG Gateways

Maximum Transmission Unit (MTU) MTU for Trunk NNI with CSP The minimum MTU we can accept for the Trunk NNI is 4400. The objective MTU is 9000 since the DISN is moving to make this the standard for all leases.

a. Threshold MTU for Trunk NNI: 4400

b. Objective MTU for Trunk NNI: 9000

MTU for Customer Circuits (Individual VLANs) In almost all cases, the MTU for customer access circuits is 1500 (1518 including Ethernet header). NIPR, SIPR, Private ISP and NFG can never exceed 1500. The DISN currently has a few special Layer-3 VPNs that have requested an MTU of 4000. This could also be supported for CEG, assuming the CSP can support an MTU of 4000 of their side.

Converged CE Router using CSP service to access DISN It is generally very economical for the customer to access more than one service utilizing the same physical interface. A Converged CE Router is a router that has a forwarding path for both encrypted classified traffic and unclassified traffic which is kept separated using VRFs and VLANs. Figure 2-5 depicts this implementation. Notice, the VLANs separate the different service on the access link and the VRFs separate the service within the CE router. This method of separation has received blanket approval by the DISN Security Accreditation Working Group, as long as the classified data is Type I encrypted in addition to the VRF and VLAN separation.

This same architecture applies to Converged CE routers connected to E-Line service on the CSPs network. Customers are responsible for deploying Unclass Converged CE routers (Not the CSP) A separate E-line is needed for each VLAN leaving Converged CE router. Note: for SAM, the CSP may have a SAM Layer-3 VPN which should be used instead of an E-Line.

Figure 2-5 Converged CE router

Performance Specifications for CSP Performance requirements consist of latency, packet loss, availability and jitter.

Performance of the NNI The NNI will typically be a local fiber connection between the J-PE and a co-located commercial handoff node. The NNI needs to be completely error free. (i.e., zero packet loss)

End-to-End Latency (Delay) Specification The round-trip latency is dependent on which theater the service resides. See latency numbers below specified in milliseconds:

Intra-CONUS ≤100 ms Intra-EUR ≤100 ms Intra-PAC ≤150 ms Intra-SWA ≤100 ms

End-to-End Packet loss, Availability, and Jitter Not all requirements are the same. This design is calling for (3) different Service Characteristics which may or may not be available from all CSPs. The specifications below may need to change based on the realities of what SLAs we can get from the CSPs. DISN trunks (J-PE to J-PEx) will need to use the Premium service due to the services (i.e. circuit emulation) offered on the J-PE routers. If a Premium performance level is not available for J-PEx trunks, then OTU-2 leases will be used instead.

Assured #1: (Premium) Packet loss not to exceed: 0.001% (99.999) (1 out of 100,000 packets can be dropped) Availability (End-to-End): 99.9% Jitter: 5ms

Assured #2: (Silver) Packet loss not to exceed: 0 .02% (99.98) (1 out of 5,000 packets can be dropped) Availability (End-to-End): 99.75% Jitter: 15ms

Assured #3: (Basic) Packet loss not to exceed: 0.1% (99.9) (1 out of 1,000 packets can be dropped) Availability (End-to-End): 99.5% Jitter: 25ms

Interface Specifications for CSP Interface Specification for NNI Trunk Service Type: Network to Network Interface (NNI) Physical Media Type: Fiber (Single mode) Interface Type: 1GE, 10GE, or 100GE Tagged/Untagged: Tagged (802.1Q) Interface Specification for customer connection to CSP Service Type: Layer 2 E-line or Layer 3 Private IP Type Media: Customer defined (i.e., Single mode, Multimode, copper (CAT5E or better)) Interface type/rate: Customer defined (i.e., Optical 10GE, 1GE, Fast Ethernet, 10/100) Tagged/Untagged: Both Types (Customer defined). Some CSPs may require customers to use 802.1Q tagging.

Note: For Customer getting Layer-3 VPN from CSP, TDM may be the only option for some locations. This has no effect on the DISN because the NNI will always be Ethernet.

Security Control and Data Plane Security All the security controls that we currently put on customer interfaces will still apply to the NNI interfaces. For example, for each VLAN on the NNI we still must apply Control Plane Policing and Data Plane ACL policies. Per-VLAN ingress policing and per-VLAN egress shaping will be applied to each VLAN to insure the availability of each VLAN.

Unlike customer interfaces, Strict Route Filtering for Layer-3 VPN BGP Sessions must be applied for each Layer-3 VPN. This is an extra security precaution because the DISN is peering directly with a CSP.

The security mechanisms applied on the J-PE to the NNI are summarized below:

a. Control Plane Policing (Ingress J-PE)

b. Data Plane ACLs (Ingress J-PE)

c. QoS Controls (Ingress Policing per VLAN, Egress Shaping per VLAN)

d. Strict Route Filtering for Layer-3 VPN Peering with CSP (Ingress J-PE)

Ethernet Layer-2 Type-3 Encryption (Optional for CEG locations)

This is an end-to-end Ethernet (Layer-2) Encryption solution for E-Lines. The Aggregation Encryptor can terminate multiple E-Lines from different customers and is also referred to as an Gateway Encryptor. See figure 2-6 for a high-level overview of the Ethernet Encryption architecture. The Ethernet Encryptors must be configured to pass the VLAN (from red to black side) without encryption of VLAN tag and MAC Header. The Aggregation Encryptor supports a Selective Bypass option which means it can provide bypass for some E-Lines that don’t require encryption. Customers decide if their E-Lines require encryption. They also have to procure a matching encryptor for their side of the E-Line. J-PEx trunks (OCONUS) require encryption.

This is a book-end solution that may be deployed at some CEG locations. It is called book-end because both ends of the E-Lines must use the same type of encryptors, until “standards based” interoperability between different Layer-2 Ethernet Encryptor vendors is available.

The following are the (2) modes that must be supported by the Aggregation encryptors:

1. N:N , VLAN Pass Thru, TRANSEC=On

2. N:N , VLAN Pass Thru, TRANSEC=Off

Figure 2-6 Layer-2 Type-3 E-Line Encryption Quality of Service

With multiple customers sharing the same physical interface, implementing QoS is very important to insure that one customer’s traffic cannot impact another customer. This will require a hardware implementation of per-VLAN QoS with shaping and queuing. The same DSCP plan and QoS profiles that apply to customer interfaces will be used for the NNI trunks connecting to CSPs.

Redundancy for MPLS Layer-3 VPN and E-Line

Redundancy can be achieved with Layer-3 VPN or (2) separate E-Lines from the CSP.

0. Gateway Redundancy for MPLS Layer-3 VPNs CEG can achieve Gateway redundancy for a Layer-3 VPN if there are (2) or more NNIs between the DISN and the CSP. BGP metrics determine the Preferred and Standby paths. See figure 2-7 for an illustration of how this works.

Figure 2-7 Example of Gateway redundancy for SAM over (2) CEG Gateways

0. Achieving Redundancy with (2) Separate E-Lines Redundancy can be achieved with CEG if (2) separate E-Lines are used and each E-Line goes over a separate NNI. See figure 2-8 for an illustration of how this works.

Figure 2-8 Achieving redundancy using (2) E-Lines

CSP owned Aggregation Device at Off-Campus Lease Locations

Since the CSP must put some type device at all the applicable off-campus locations, it makes sense for the CSP to aggregate all the Ethernet circuits from the multiple customers/circuits on that base. For this design, the CSP will aggregate all the circuits from the base routers/devices. See figure 2-9 for high level diagram of this concept.

Figure 2-9 CSP Aggregation Device

FOUO

image3.emf image4.emf image5.emf image6.emf image7.emf image8.emf image9.emf image10.emf image11.emf image1.jpeg image2.emf