Multihoming With BGP
If you work for an ISP, you probably have an appreciation of the BGP (Border Gateway Protocol)'s built-in flexibility and protection. BGP was designed for multiple communicating networks with no single administrative entity, so it's tailor-made for the Internet. On the other hand, if you work at an organization with an Internet connection, you may not even have to bother learning how to spell BGP, much less how it works. You can get along by using a default route that points all external traffic to your ISP and let its staff work out the details. However, if you're thinking of adding another Internet connection to a second ISP for backup purposes or load-balancing, you'd better familiarize yourself with BGP. Note: We highly recommend that you work closely with your ISP before experimenting with features that propagate outside your network.
What Is BGP?
BGP is the only widely implemented EGP (Exterior Gateway Protocol) and the only routing protocol used to link networks to one another on the Internet. BGP was first specified in 1989 in RFC 1105. Version 4 was specified in 1994 in RFC 1654 and updated in RFC 1771. There have also been a number of documented extensions. Version 4's most significant contribution is its ability to aggregate advertisements from multiple contiguous routes in one routing-table entry. This is known as CIDR (Classless Interdomain Routing). BGP4 was implemented when big routing tables began overwhelming routers. What CIDR does is protect you from many potential outages and instability on the Internet, and provide great relief for address depletion by more efficiently dividing addresses.
When enabled, BGP4 establishes relationships with adjacent routers, referred to as neighbors. Unlike OSPF (Open Shortest Path First) and EIGRP (Enhanced Interior Gateway Routing Protocol), which will automatically discover routing neighbors, BGP won't exchange routing-table information until both routers have configured one another's IP addresses and ASNs (Autonomous System Numbers) on their corresponding interfaces. Once this is completed, the neighboring routers are considered peers.
Neighboring routers send small "keep-alive" messages to one another. If a neighbor stops receiving keep-alive messages for a predefined "hold time," it will update its routing table to reflect the loss in available routes. BGP also sends incremental updates when routes become unavailable. Otherwise, the full routing tables are exchanged only when two routers first establish or re-establish a peering relationship as described above.
BGP is a Path Vector Protocol, which is similar to a Distance Vector Protocol, but with a key difference. A Distance Vector Protocol chooses routes based on the hop count (or routers traversed) and link speeds. BGP, in contrast, chooses a route that traverses the least number of ASes (Autonomous Systems). As a routing advertisement passes through an AS, it prepends (adjusts the path length advertised) the ASN of the origin AS to the path of other ASes it has traversed. By default, the path with the fewest ASNs is stored in the routing table as the optimal path to a destination network. One AS can contain multiple routers, so it's possible for the actual hop count to be higher than the AS path indicates.
BGP Optimization
With BGP's built-in flexibility, however, you can enhance this default behavior. For instance, you may want to control the path traffic takes as it leaves your network. When peering with multiple neighbors in an external AS, or in different external ASes, there will be multiple paths to the same destination network. By default, BGP determines the optimal path by picking the route that traverses the fewest number of ASes. However, BGP does not take link speed or network load into consideration when computing paths, so the shortest path may not be the optimal one.
You can get around this by using BGP's Local-Pref attribute, which forces BGP to take a particular next-hop route in a scenario with multiple choices. You simply tell the router that all, or even some, of the routes advertised to one of your router interfaces should receive a higher Local-Pref weight than the same routes advertised to another interface. Because Local-Pref is always considered before the computed path distance, the interface you designate with the highest Local-Pref will be chosen as the best route.
Controlling traffic as it comes back into your network is more difficult. With geographically diverse networks, where one ISP connection is a lot closer to one part of the network than to another, you may want to use the MED (multiexit discriminator) attribute, which specifies the path external traffic should use when destined for one of your internal networks. Although the MED attribute is a simple way to control incoming traffic, it will work only if both Internet connections go to the same ISP, since it won't be propagated outside that ISP's AS. Of course, prepending is another way to control incoming traffic.
BGP routing can also be controlled through the community attribute that puts a predefined code on a group or community of routes so the receiving router takes a predefined action based on the value of that code. This code can be user-defined, but the most common is a reserved or well-known community, called No-Export. When a BGP router sees a route come in with the No-Export community, it will not advertise the route outside its own AS. This can be handy for balancing incoming traffic.
The ISP Balancing Act
You'll most likely use BGP if you have multiple connections to the Internet. But if you're interested only in load-balancing, you'll want to stick with one ISP. For redundant ISPs, you'll need two ISPs, and balancing the load will be more difficult.
When you have one ISP and one router at your end, balancing outgoing traffic can be simple, and you have the most control over the paths your packets traverse. In fact, if both interfaces are on the same router at your end, you can even avoid BGP. For example, Cisco Systems gives you the ability to load-balance between two static routes. It's possible that your ISP could do the same for the traffic entering your network.
As we discussed above, you can use the MED attribute to control how traffic enters your network, as long as both connections go to the same AS. If both connections go to the same ISP, there's a very good chance both connections will go to the same AS. For traffic leaving your network, you can also use the Local-Pref attribute to control how traffic leaves your network.
BGP should automatically take care of failing over from one ISP to another and thus provide redundancy with its default settings. However, load-balancing with multiple ISPs can be very challenging. The best approach is to start with the default BGP settings and monitor the transmitted and received traffic. Once you get a baseline, use the Local-Pref attribute to fine-tune traffic leaving your network based on the destination external network or on the AS. There are more than 60,000 networks advertised on the Internet, so target routes with the highest traffic levels. You can always experiment by switching preferred routes from one ISP to another if you get response-time complaints.
You may also need to balance the traffic coming back into your network. One common way to do this is to insert extra AS numbers in the advertisements as discussed above. This works best if you're advertising multiple, internal networks. You can then make the advertisements for one network look better than those for another and force the traffic back into your network via the corresponding ISP pipes.
Another approach is to limit the traffic on one ISP connection to that ISP--and all its customers--and let the rest of your traffic use the other ISP. This works well if much of your traffic rests with the customers of one of your ISPs. The approach also works well if you're using VPNs (virtual private networks) to connect your branch offices or partners on the same ISP. You can use the No-Export community feature to make sure your network isn't advertised beyond that ISP's network. This will take care of incoming traffic. You can also use the Local-Pref attribute to control outgoing traffic by making sure outgoing traffic is restricted by the ISP's ASN.
The drawback to this approach is that you sacrifice some redundancy. If the connection to your ISP that talks to the whole Internet goes down, your other connection can't take over unless you turn off the No-Export option. In this way, the only traffic that will know how to reach your network will still be restricted to that ISP's customers.
BGP provides a lot of control over the destiny of your Internet traffic. But be sure to work closely with your ISP when implementing these options. When you communicate with the Internet that intimately, you must make sure you don't cause any problems. Moreover, if you'll be accepting routes from the Internet, you'll need to invest in lots of memory and CPU for your routers. You should have at least 64 MB available, if you're accepting a full routing table from one ISP. You'll need to double that if you're accepting full routing tables from two ISPs in one router. Also, before you sign up with an ISP, be aware that there's no guarantee the ISP will even advertise a network that's been provided by another ISP. Make sure your new ISP can provide the redundancy and load-balancing you expect.
Editor's Note: Next week, we'll wrap up our discussion of IP routing with a discussion of multihoming, routemaps, and QoS.
Peter Morrissey is a faculty member of Syracuse University's School of Information Studies, and a Contributing Editor.
Send your comments on this article to him at ppmorris@syr.edu.
RSS
