Inside Flat Networks

Get the full-length
Analytics Report on flat networks

>> See all of our reports <<

Virtualization causes significant changes in network traffic, forcing IT to reconsider the traditional three-tier network approach. Tiered networks were designed to route traffic flows from the edge of the network through the core and back, which introduces choke points and delays while providing only rudimentary redundancy.

Enter the flat network. Also called a fabric, this approach allows for more paths through the network and is better suited to the data center's requirements, including the need to support virtualized networking and virtual machine mobility. A flat network aims to minimize delay and maximize available bandwidth while providing the multitude of network paths demanded in a virtual world.

But a flat network also requires some trade-offs, including the need to rearchitect your data center LAN and adopt either new standards such as TRILL (Transparent Interconnection of Lots of Links) and SPB (Shortest Path Bridging) or proprietary, vendor-specific approaches. We'll look at how a flat network differs from a traditional tiered infrastructure and examine potential shortcomings. (For a look at flat network security, see "How To Secure Your Flat Network".)

How We Got Here

Ethernet won the battle for the LAN more than a decade ago, but it still suffers significant limitations. One prominent problem is the forwarding mechanism. When an Ethernet switch doesn't have a MAC address and interface pair in its forwarding table, or if it receives a broadcast Ethernet frame, the switch makes a copy of the frame and forwards the copy to all interfaces. Because Ethernet has no Time To Live (TTL) header field to keep a frame from being forwarded indefinitely, if there's a physical loop in the network these frames will be copied and propagated repeatedly throughout the network until it crashes.

Radia Perlman, a renowned network engineer and an Intel Fellow at Intel Labs, created the Spanning Tree algorithm, which became part of the Spanning Tree Protocol (STP), to solve this issue (among others). And it works. In at least 40% of the networks I see, Spanning Tree has never been changed from its default settings, but it keeps the network up, while providing some redundancy.

However, while STP solves significant problems, it also forces a network design that isn't optimized for many of today's data center requirements. For instance, STP paths are determined in a north-south tree, which forces traffic to flow from a top-of-rack switch out to a distribution switch and then back in again to another top-of-rack switch. By contrast, an east-west path directly between the two top-of-rack switches would be more efficient, but STP doesn't allow that path.

InformationWeek: Mar. 12, 2012 Issue

Download a free PDF of InformationWeek magazine
(registration required)

Get Flat

This is where flat networks come in. They can lower latency, provide access to more bandwidth, and increase the return on investment of a data center's network infrastructure.

There are several ways to design a flat network or fabric. You can use TRILL, SPB, or several proprietary approaches from infrastructure vendors. I separate these approaches into two groups.

Group one, including TRILL (from the IETF), SPB (from the IEEE), Cisco's FabricPath, and Brocade's Virtual Chassis Switching (VCS), aims to make LAN switches smarter. At a high level, these technologies all address the learning and distribution of forwarding paths using a link state routing protocol (IS-IS, which plots the best path through the network) and the forwarding of traffic from Host A to Host B across multiple paths. However, they differ significantly in their execution.

TRILL adds a new frame type (a TRILL frame) to normal Ethernet frames and forwards them between ingress and egress RBridges. Think of RBridges as ZIP codes. The final street address, say, the destination MAC of Host B, doesn't matter until you are in the right ZIP code (egress RBridge). The ingress RBridge maintains a mapping of final MAC addresses to ZIP codes (RBridges) and then all intermediate switches move traffic from one ZIP code to the other. This additional encapsulation header also includes a TTL field to help stop flooding produced by physical loops.

Unlike TRILL, SPB doesn't encapsulate traffic in a new frame type. Instead, it uses either Q-in-Q or MAC-in-MAC encapsulation, which is available on many switch models. Once it learns its mapping of destination address-to-egress switch through IS-IS, it encapsulates the original frame in a new Q-in-Q or MAC-in-MAC frame and readdresses the new frame to the egress switch.

SPB and TRILL both support multiple paths, faster failover than Spanning Tree, and increased reachability. SPB proponents point to its support of legacy ASICs because no new frame format has to be added to the frame. In other words, you may only need a software upgrade to support SPB, while TRILL will likely require new hardware to handle the TRILL frame format.

Note that while TRILL and SPB solve some of the limitations that are introduced by STP, these technologies also have potential downsides. For example, their topologies may require significant security configuration to ensure that switches are communicating only with other authorized switches.

Cisco FabricPath and Brocade VCS are similar to TRILL in that they encapsulate traffic for transport across a fabric backbone. However, these are proprietary technologies that don't interoperate with TRILL. In fact, they both work differently with existing Spanning Tree networks. VCS either passes your Spanning Tree info through its fabric or drops it entirely. FabricPath terminates Spanning Tree domains at the edge of its fabric.

The second group of technologies tries to eliminate Spanning Tree's inefficiencies by eliminating the tiered design altogether. This approach treats all switches as one giant switch or fabric. Juniper's QFabric is an example. Essentially, the physical switches act as blades within a giant chassis. In Juniper's model, the QF/Director handles the control plane, coordinating the individual switches. The entire fabric not only looks like one giant switch to connected hosts, as with TRILL or SPB, but is also managed as one giant switch.

Another alternative is Multi-Chassis Link Aggregation, variations of which are offered by most major switch vendors. MLAG switches act as a single switch for downstream STP bridges to eliminate Spanning Tree blocking of redundant paths. This allows for better utilization of links and simplifies management. Most MLAG implementations are limited to two aggregated switches.

A flat LAN architecture holds a lot of promise, but we don't recommend blind adoption. If your engineers don't have time to set up a Spanning Tree root bridge priority in order for the Spanning Tree algorithm to choose the best possible tree, can you really expect that TRILL or SPB or any other more complicated setup will be tuned properly?

The LAN stability provided by Spanning Tree shouldn't be taken for granted; you must clearly understand the potential ramifications of new flattening technologies. There's a reason tiered network architectures are so prevalent: They work. If you decide to go flat, apply these new approaches judiciously.

Jeremy Littlejohn is president of consulting firm RISC Networks. Write to us at [email protected].