Cisco EIGRP OTP Connects Networks Across Provider Infrastructure

Cisco Systems has been hoisting the EIGRP flag up high of late. Enhanced Interior Gateway Routing Protocol--a routing protocol, of all things (yawn)--wouldn't seem like anything to get too excited about, what with all the hoopla surrounding software-defined networking, overlays and network virtualization. And yet Cisco is clearly committed to EIGRP.

First of all, much of EIGRP has moved from proprietary status to open. Open EIGRP was announced earlier this year, when Cisco released significant portions of the EIGRP specification to the open source community in the form of an IETF draft. This move met with mixed reviews and some sideways glances, but it demonstrates Cisco's desire to see EIGRP used even more broadly in the industry. With the widespread acceptance of the OSPF routing protocol, some might wonder what the point is, but EIGRP fans can attest to its flexibility and scalability--EIGRP really stands up.

Cisco is giving EIGRP users an interesting feature, which was announced in June at Cisco Live. EIGRP Over the Top (OTP) allows EIGRP routers to peer across a service provider infrastructure without the SP's involvement. In fact, with OTP, the provider won't see customer routes at all. EIGRP OTP acts as a provider-independent overlay that transports customer data between the customer's routers.

To the customer, the EIGRP domain is contiguous. A customer's EIGRP router sits at the edge of the provider cloud and peers with another EIGRP router a different location across the cloud. Learned routes feature a next hop of the customer router--not the provider. Good news for service providers is that customers can deploy EIGRP OTP with their involvement.

Inside EIGRP OTP

OTP is a genuinely new feature that Cisco has created for EIGRP, so let's peek under the hood at how it works. There are a few key elements:

•Neighbors are discovered statically. There's no auto-discovery mechanism here. For customers thinking about trying to build an EIGRP mesh across a provider cloud and shuddering at the thought of manually configuring n-1 relationships per router, it's not as bad as all that. OTP does not require a mesh of peering relationships to support a full mesh topology. (See the next point.)

•An OTP mesh scales by use of a route reflector (RR). When designing the EIGRP OTP overlay, a customer selects a router to be a RR. When additional customer routers are added to the OTP overlay, EIGRP is configured on that new router to peer with the RR. The RR takes route advertisements in, and reflects them out to all other EIGRP customer routers in the OTP overlay. The RR preserves the next hop attribute of the route, which is critical. This means that the RR is not going to be the hub of a hub and spoke forwarding topology. Instead, a full forwarding mesh is formed. For example, let's say we've got three routers: R1, R2 and R3. R1 is the RR. R1 is peered with R2 and R3. When R2 advertises a route, let's say 10.2.2.0/24, to R1, R1 reflects 10.2.2.0/24 to R3, preserving R2 as the next hop of that route. When R3 needs to talk to 10.2.2.0/24, it'll connect directly to R2, and not through R1, which reflected the route to it.

•Metrics are preserved across the service provider cloud. In other words, the EIGRP domain treats these neighbors and links just like any other EIGRP neighbors and links. Therefore with OTP, a customer ends up with a contiguous EIGRP domain across the SP cloud. That eliminates a common scenario of isolated EIGRP domains at each customer office being redistributed into the SP cloud, and then redistributed again from the SP cloud into remote office EIGRP domains. OTP also eliminates the scenario of multipoint GRE tunnels (for example, with DMVPN and NHRP) being nailed up as a manually created overlay that runs EIGRP across it. An OTP configuration is comparatively simple to code in IOS.

If you're thinking this through, you might be wondering how the EIGRP customer routers can push traffic for remote subnets across the SP cloud, if the customer routers are not advertising those subnets to the SP. The answer is that traffic between the customer EIGRP routers is encapsulated in LISP packets. Therefore, all the SP needs to know is how to get from customer router to customer router to carry the LISP packets flowing between them; the SP will know this by virtue of the directly connected routes used to uplink the customer routers to the SP cloud.

[What's holding back TRILL adoption? Find out in "TRILL's Hidden Cost."]

Cisco chose LISP to encapsulate traffic between the customer routers because it provided a critical design feature that enables OTP: stateless tunneling in an any-to-any mesh topology. It's worth pointing out that LISP is only used as a data plane transport. In other words, Cisco is not saying that EIGRP OTP has a dependency on LISP. Instead, Cisco just happens to be using LISP as the encapsulation format to smuggle traffic across the SP cloud between customer routers.

A consideration for secure environments is data privacy across the SP cloud, and LISP has no native encryption function. However, OTP can be configured to work with GET VPN to encrypt the LISP traffic flowing between the customer EIGRP routers.

For shops interested in multitenancy, EIGRP OTP supports this via VRFs or Easy Virtual Network (EVN), mapped to (as best as I can tell) LISP multitenancy. LISP packets have header fields that support tagging certain packets to belong to certain tenants (think VPNs). I'm speculating a little on this detail, as I don't have any hard data on exactly how it works.

NEXT: OTP's Shortcomings

My first thought here is that OTP simply gives network designers a new, useful tool in their toolbox. There's not a lot to condemn this approach, but let's consider a few things.

Like any overlay, LISP introduces packet overhead (i.e., you're taking a packet and sticking a wrapper around it, making the packet bigger than it was). Therefore, the packet maximum transmission unit (MTU) size must be considered. However, as the interfaces carrying the encapsulated traffic are SP-facing, this might not be a huge concern. Ask your provider to support oversized packets, and off you go--well, at least in theory it's that easy. In reality, it could be a bit of a challenge to accomplish this with your service provider.

Alternatively, a designer might need to implement path MTU discovery or rewrite TCP maximum segment size (MSS) to avoid fragmentation issues at the customer router sitting at the edge of the SP cloud.

Another concern that popped in my head is that the EIGRP OTP route reflector appears to be a single point of failure, but documentation is scant on OTP thus far so I don't know for sure. As the RR is such a key component of the OTP design as well as a critical component for scaling the solution, my gut feeling is that there's probably a mechanism for RR redundancy. I just didn't happen to see any evidence of such a thing in the material I reviewed.

An additional drawback could be that EIGRP OTP doesn't support multicast traffic at this point. That's not going to be a roadblock to OTP adoption for many organizations, but since large multicast trees (including ones that traverse a SP cloud in some way) are increasingly common, this could cause OTP to fail the requirements of some networks in the short term. Multicast support is road mapped for OTP.

Another point to consider: To scale to very high throughput requirements, encapsulation and decapsulation operations should be performed in hardware. Customers evaluating EIGRP OTP will need to consider whether the combination of hardware and software they want to run OTP on will support the data rates they require. It's not yet clear to me which of Cisco's myriad platforms will eventually support OTP, and of those platforms, which of them could support LISP encapsulation in hardware.

That's not to say that OTP is a non-starter if the encapsulation operation is punted to the device CPU on a given platform. If there's enough CPU in the device to handle the required throughput without impacting the device negatively, then that's a valid solution.

Cisco is shipping OTP now on IOS XE on the ASR 1K platform. In November, Cisco expects to ship OTP on classic IOS in the Catalyst 3K, 4K and 6K products. I have no information on shipping OTP for the ISR router platform, but my best guess is that it will be a Cisco priority, as many customers position the ISR platform as a WAN edge router. Speculating again, I suspect adding this feature to NX-OS is a somewhat lower priority for Cisco, as Nexus hardware is less likely to be positioned at the WAN edge. However, there are use cases I can conceive of for OTP on Nexus, and interestingly, there's at least one Nexus 7000 series line card that can handle LISP in hardware.

What do you think of OTP? Is it something your organization could use? Share your thoughts in the comments section below.