In a data center that has deployed Cisco Nexus 5000 or 7000 switches, Cisco Nexus 2000 series fabric extenders (FEX) are commonly used for top of rack (ToR) connectivity. FEX units are priced attractively, as they serve a limited purpose and are not feature-rich switches. In fact, a FEX is not a switch, in that a FEX cannot switch traffic locally or be managed independently. A FEX functions only when connected to a Cisco Nexus 5K or 7K series. All traffic flowing into a FEX will be sent down to the parent 5K or 7K for forwarding, even if the destination is on the originating FEX. If you think of a FEX as a remote line card with no local switching capabilities, you've got the idea.
Having established that a FEX is not a switch, let's take a look at the process of installing a new FEX and bringing it online.
1. Rack the FEX. The official Cisco documentation demonstrates racking a FEX in a four-post rack, using the supplied rack ears and rail slides. I have successfully racked FEXs in open two-post racks using only the rack ears. If you choose to do this, be sure to mount the ears further back along the chassis for better weight balance. However you mount the FEX, provide enough clearance for the fiber uplink cables, as they stick out of the front of the chassis. Tight tolerances could prevent the rack door from shutting.
Do not power on the FEX until all cabling and uplink port provisioning on the uplinked 5K or 7K has been completed.
2. Install & cable the uplinks. When uplinking to Nexus 5Ks, one topology option is to dual-home the physical Fabric Extender. Dual-homing a FEX provides path redundancy, but cuts in half the total number of Fabric Extenders you can deploy. A Nexus 5K supports 24 total connected FEX devices, meaning that two 5Ks could support 48 total single-homed physical FEX. When dual-homing, only 24 total FEX are supported between the two 5Ks.
If you choose to single-home the FEX, you lose uplink path redundancy for single-attached hosts. If single-attached hosts are not a concern, a common scenario is to deploy two FEX to ToR, each single-homed to a Nexus 5K or 7K. Multiattached servers then spread their uplinks across the two ToR FEX devices, and in that way enjoy uplink path redundancy. As of this writing, only Nexus 5Ks support dual-homing of FEX.
If you choose to dual-home the FEX, the uplink ports you select must match on both 5K-1 and 5K-2. For example, if you chose Eth1/1 on 5K-1, you must also use Eth1/1 on 5K-2. Another wise design choice is to spread your uplinks over multiple 5500 ASICs (the silicon inside the switch responsible for forwarding traffic). In the Nexus 5500s, ASICs are mapped to groups of 8 consecutive Ethernet ports on the front to the switch, 1-8, 9-16, etc. Therefore, spreading FEX uplinks over ports 1 and 9 is smarter than 1 and 2. If the ASIC servicing ports 1-8 fails, you won't lose both FEX uplinks.
Although options vary by model, FEX can be uplinked with a variety of media. When ordering, be aware of Cisco SKUs that bundle the FEX with uplink media, such as Fabric Extender Transceivers (FETs) or Twinax. Each option has a different price and distance constraint, so research this carefully to be sure you meet your installation’s requirements.
Another consideration is the number of uplinks to use. For example, a 2248TP FEX has 48 10/100/1000 access ports and four 10-Gbps uplink ports. If uplinking all four 10G ports, the oversubscription ratio of access to uplink ports is 1.2:1--very low. At the same time, you’ve used up four expensive 10G ports on the Nexus switch on the other end. Do your traffic patterns warrant using all four uplink ports, or can you get by with just two? Note that from a technical standpoint, the FEX will function correctly with only a single working uplink, but a sensible design uses at least two in a production environment.
The remainder of this example assumes a dual-homed FEX using all four uplinks, connected to a Nexus 5500 pair configured in a virtual port-channel domain.
3. Configure a virtual port-channel and add physical interfaces. Two of the FEX uplinks will homed to one 5K, and two to the other 5K. Then, all four of the FEX uplinks will be combined into a single virtual port channel. Each FEX is assigned a number from 100-199.
The physical interface requires two specific commands to tell the hosting 5K that the interface is servicing a FEX. The command "switchport mode fex-fabric" lets the Nexus switch know that the device on the other end of the link is fabric extender. Note that if you use FETs as uplink media, the switch can’t use these optical modules until this command is in place.
The command "fex associate " tells the Nexus switch which specific FEX is being uplinked to that port. The number selected must match for all uplink ports.
4. Apply the code below to both 5Ks. Once this step is complete, you can optionally prepare your console session to watch the messages that will scroll as the FEX comes up for the first time by typing "term mon".
interface Po101
description UPLINK FEX-01
vpc 101
switchport mode fex-fabric
fex associate 101
no shut
interface Ethernet1/1
description UPLINK FEX-01
switchport mode fex-fabric
fex associate 101
channel-group 101
no shut
interface Ethernet1/9
description UPLINK FEX-01
switchport mode fex-fabric
fex associate 101
channel-group 101
no shut
Next Page: Powering Up
