Networking

04:18 PM
Connect Directly
Twitter
LinkedIn
Google+
RSS
E-Mail
50%
50%
Repost This

Cisco Nexus 6000: First Impressions

I got a sneak peek of the Cisco Nexus 6000 switch series at Cisco Live. It's packed with 40-GbE ports, multicast support and solid core Ethernet capabilities that will translate into speed, simplicity and scale. The Nexus 6000 means Cisco finally "gets" cloud networking.

While attending Cisco Live in the London, I went to a session exploring the architectural details of the forthcoming Nexus 6000 data center switch. The Cisco Nexus 6000 closes Cisco's product gap in the cloud networking market and addresses the need for a leaf/spine networking architecture built around high-speed, low-latency Ethernet networking.

Cisco positioned the Nexus as a mid-range switch for aggregation, but I can't see who would buy it for that reason. That said, here are details.

The 6000 series includes line-rate forwarding for all ports at any packet size. Cisco claims 1 microsecond port-to-port latency via cut-through switching for both 10 GbE and 40 GbE. The series offers 31 concurrent SPAN ports (which I find awesome because it provides a much needed boost to network visibility and analysis).

It has a 256K entry host table shared for MAC, ARP, IPv6 ND, (*,G) multicast and /32 host route (which is good but not great, as Brocade does 384K/384K in its VDX chassis). The 6000 series also includes Equal Cost Multipath support for up to 1,000 paths.

The Nexus 6000 has a surprising amount of multicast support, including PIM Bi-Directional. It seems likely that this was intended for VXLAN support for overlay networking. For more on the relationship between VXLAN and multicast, see my post VMware's Dilemma: VXLAN or Nicira?.

The 6000 series comes in two models, the 6001 and 6004. The Nexus 6001 offers 48 10-GbE ports, plus an optional four 40-GbE ports. The 6001 comes in a 1 RU form factor with redundant power.

The Nexus 6004 is a pitched as an aggregation-layer switch in 4RU form factor with redundant power. It has 48 40-GbE ports in a fixed form factor. It can support a total of 96 40-GbE ports via four additional line cards. Cisco says future upgrades will include 100 GbE via line cards.

If you are new to 40 GbE, you may want to know that the QSFP interface is used for 40 GbE transceivers. The 40 Gb standard uses 4 x 10 GbE "lanes" that are multiplexed at the SERDES physical layer. Therefore, you can obtain a QSFP transceiver that acts a 4 x 10 GbE port and connect to existing Nexus 22xx series ToR switches.

40 GbE also uses a new type of connector, with eight fiber optic cores per interface. This will change how you cable your data center in the future.

Use Cases

The Nexus 6000 fits between the Nexus 7000, which is oriented toward service providers, and the 5500/2200 switches.

Today, the Nexus 7000 is the "kitchen sink" services platform. The NX-OS software on the Nexus 7000 provides MPLS, LISP, vPC/FEX, and other fancy but often buggy software features. It's expensive to run in terms of power consumption and maintenance, and very hard to upgrade and maintain due to its complex internal architecture.

Service providers and large-scale cloud providers have found that the Nexus 7000 isn't well suited for their needs and have turned to other providers. Cisco hopes to address this with the Nexus 6000. It will appeal to providers and large cloud implementations for fast, simple Ethernet switching. The Nexus 7000 will continue to be attractive to enterprises that perceive safety in buckets of features).

Cisco claims that the Nexus 6000 is the fixed format unit for 10G/40G/100G and returns Cisco's focus to core networking features. The Nexus 6000 has smaller rack space, low power, low latency and jitter, and a limited feature set.

While the maximum throughput of the Nexus 6000 is a hefty 7.68 Tbps, the Nexus 7000 could, one day, reach 36 Tbps. Most companies can't afford the 40 G line cards on the Nexus 7000 because of the fancy silicon that offers fancy features (and lead to high pricing).

Annoyances

While the Nexus 6000 series has some compelling features, I did find a few nits to pick. First is the airflow, which is back to front. Cisco says reverse airflow will come in a future release. Second, 100 GbE isn't expected to ship until mid-2014, which is a long delay.

Finally, its IPv6 scalability is limited. The TCAM must be using older designs for 32-bit addressing of IPv4, because IPv6 consumes multiple TCAM slots and reduces scalability to just 50,000 hosts. It's 2013 and IPv6 is still a second-rate technology at Cisco when it comes to hardware support.

First Impression

The Nexus 6000 series is aimed at customers that need high-speed Ethernet with a limited set of network services. For these customers it's a much better alternative than the 7000, which is too complex and unreliable for core Ethernet (though desirable for its comprehensive services and extended features such as QoS, MPLS, VDC and OTV). I'd relegate the Nexus 7000 to the WAN or campus edge and put the 6000 in the core for my networks.

What I see is Cisco slowly adapting to the new era of networking. While the Nexus 6000 makes the extended life support of the decade-old Catalyst 6500 even more confusing, equally, the Nexus 6000 will be welcomed by a significant segment of the market that is building public and hyper-scale cloud-based networks and needs speed, simplicity and scale instead of massive feature sets and fancy functions. Greg has nearly 30 years of experience as an IT infrastructure engineer and has been focused on data networking for about 20, including 12 years as Cisco CCIE. He has worked in Asia and Europe as a network engineer and architect for a wide range of large and small firms in ... View Full Bio

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
EtherealMind
50%
50%
EtherealMind,
User Rank: Apprentice
2/5/2013 | 7:36:23 PM
re: Cisco Nexus 6000: First Impressions
Flexibility is yesterdays requirement. Today it's reliability.
David Klebanov
50%
50%
David Klebanov,
User Rank: Apprentice
2/4/2013 | 11:17:00 PM
re: Cisco Nexus 6000: First Impressions
Greg and David,

Thank you for your comments. Let me reply to both of you at the same time.

Cisco's footprint in Data Center switching fabrics is so huge, that you can imagine how many new features and feature enhancement requests are being put forth by our customers. The more creative our customers get in deploying Cisco Nexus products, the more versatility they expect from them. Service providers expect more speeds and feeds, financials want ultra low latency, enterprises want feature-richness with great deployment flexibility and the list goes on. Many of them want their requests to be acted upon yesterday :-) We listen to all our customers and prioritize accordingly.

Our annual R&D spend is more than some of our competitors' net worth and that should indicate how serious we are and how much we push ourselves on developing quality hardware and software products. There is no such thing on this planet as "risk free code", I wish there was. Cisco software is no exception to that rule. We put our software through months of rigorous testing for both specific features and holistic solutions. With NX-OS we also instituted approach of short and long lived software releases where customers have an option to go with either safer and more field-baked code (long lived) or the one that introduces the most new features and newer hardware (short lived). Features from short lived releases eventually get into the long lived release once we have confidence that they have reached enough maturity and stability. Each organization should develop for themselves a switch/router code upgrade strategy, which is aligned with their business objectives. If you want to take advantage of the most recent software features and new hardware, you will have to be more aggressive in your upgrade strategy and there is always a certain amount of risk associated with that.

Cisco bakes enough redundancy into its products to minimize any chance of catastrophic failure and by exercising architectural approach to building your Data Center switching fabric, we drive these chances even lower. When you architect your solution the right way, you can definitely trust your network. As a customer, you can decide if you want to make a heavier investment with a more modular design or to be satisfied with more collapsed approach. For example, some customers decide to co-locate functionality such as Routing, services, L2 DCI and so on in the Spine of their network, leaving Leaf nodes purely for end-host connectivity. Other customers segment routing, services, L2 DCI and so on into their own Leaf nodes while keeping Spine for high speed, low latency transport. Each option comes with its own set of pros and cons, so I guess "trust in the network" is a subjective point of view.

Lastly, scale. I agree that designing your network for the right scale is not always an easy task, especially if your solution scale today is not what you would expect it to be 3 years from now. Even worse, you might not even know what the target scale will be. By distributing intelligence throughout the switching fabric, we eliminate the single point of resource contention, which gives you the certainty that your design can accommodate future scale growth. It is not just about TCAMs, in fact, it is a combination of quite a few considerations some of which might be obvious and some are rather obscure. As far as buying cheap, I think it's sufficient to say that one is "not rich enough to by cheap things". Cheap hardware is not always bad, but in today's economy cheap also means less tested, less validated, less experienced and so on. Most of the businesses cannot afford to wait for failed hardware replacement, because they invested in equipment that does not have sufficient redundancy. What good will monitoring do in this case? I have a friend who once told me "think about your children when you make strategic investments for your company". It might sound a bit harsh, but the reality is you want to invest in products that will keep you sleeping at night (and employed :-)).

Thank you for reading.
David
@DavidKlebanov
David Berlind
50%
50%
David Berlind,
User Rank: Apprentice
2/4/2013 | 6:43:29 PM
re: Cisco Nexus 6000: First Impressions
This is quite the conversation David... seeing you and Greg go back and forth with each other. I hate to attempt to summarize .. especially given my limited understanding of the incredibly technical specifics. But in my experience (and I think Greg is trying to say this), flexibility comes at the price of complexity. Going back to this quote..."Nexus 7000 has an unprecedented versatility and what one may perceive as complexity, is in fact a creative way of achieving deployment flexibility,"...you seem to brush-off the perception of complexity. This could be interpreted as saying... "well, there's unprecedented complexity to match" but look what you get; flexibility (which is a bit redundant with versatility).

Larry Ellison used to talk about the war on complexity. He would argue that you should take Oracles solutions out of the box and not customize them or you risk complexity and significant cost down the line. The more complex and custom your systems, the more brittle they become at points of upgrade and migration. It's hard enough getting everything to work together.

Also, if a customer says that the complexity leads to instability, I don't think the answer is essentially, "well, disable the broken parts, keep only what you need and see what happens."

Confidence plays a role in the helping guys like Greg sleep at night. Customers might be more confident if they knew that the product was rock solid, regardless of the degree to which its flexibility gets exercised.
EtherealMind
50%
50%
EtherealMind,
User Rank: Apprentice
2/4/2013 | 3:43:23 PM
re: Cisco Nexus 6000: First Impressions
You still haven't given me "risk free code", or even the confidence to upgrade often.

You still haven't given me "trust in the network". Simple services in the core, complex edge work for data centres too.

You still haven't given me "certainty in scale". Flexible TCAMs or variable performance is expensive & complex to operate. I'd rather buy cheap OEM switches and use the money saved to DevOp a monitoring system.

David Klebanov
50%
50%
David Klebanov,
User Rank: Apprentice
2/3/2013 | 10:06:32 PM
re: Cisco Nexus 6000: First Impressions
Hi Greg,

Thank you for providing insight. Let me comment further:

# Nexus 7000 complexity

Nexus 7000 platform is ground-up Data Center switch. Features introduced into Nexus 7000 family are all driven by customers' requirements. If a feature is there, there enough customer interest for it. NX-OS software is highly modular. You have a choice of enabling or disabling specific features, which causes kernel processes to be started or stopped. It dramatically decreases the software failure domain footprint and is operationally superior to having multiple software families with various functionality. Simply put, don't enable features you don't plan to use and you won't have to worry about it.

EPLD upgrades are not always necessary and are done in a rolling fashion, which does take more time, but minimizes possible disruptions, preventing multiple I/O modules from being taken down at the same time. My opinion is to never perform maintenance work without proper change window and should unexpected behavior occur during ISSU, it can be addressed with proper Cisco TAC/AS support. In my previous experience by working with numerous large customers, ISSU feature performed very well during vast majority of cases resulting in absolute zero packet loss and zero disruption. Each unique case, where ISSU encountered unexpected behavior should be individually examined and addressed.

# MAC Scalability

Many vendors proudly present large MAC address tables with ridiculously low ARP/ND table sizes. What good does 128K MAC scale do on the router, if it can accommodate only 8K ARP/ND entries? One should understand the true meaning of published numbers when looking at the datasheets...

TCAMs have certain efficiency numbers and it has nothing to do with Cisco. 128K MAC addresses should be calculated with 10% hash collision, so safe number to assume is 115K and you should not do anything further than that. If you want to monitor TCAM consumption, it is perfectly possible through the CLI or XML and will take you minimal effort.

Nexus 6000 can scale to more than 128K MAC address when positioned as Layer 2 only switch (for example FabricPath Leaf node). Stay tuned during 2013...

# IPv6

As I mentioned before, there is no IPv6 design we cannot accommodate scale-wise. Layer 2 scale for both IPv4 and IPv6 on Nexus 6000 switches is the same, since both share the same MAC address and consume one entry out of TCAM table. Most of the modern Data Center traffic is east-west and conversational learning feature does wonders to overall solution MAC scale. It is not marketing, it is hard facts.

For Layer 2/Layer 3 boundary, we take distributed approach of scaling IPv4/IPv6 to practically infinite amount of MAC addresses and host routes. If one wanted to build a single device which has a capacity to accommodate switching fabrics hosting 10s or 100s of thousands of VMs, this device would run out of other resources (such as control plane load for example), before reaching TCAM limits. Distributed approach is the way to go and for that, Nexus 6000 excels.

Thank you for listening and I LOVE debates like these :-)

David
@DavidKlebanov
EtherealMind
50%
50%
EtherealMind,
User Rank: Apprentice
2/2/2013 | 11:11:11 PM
re: Cisco Nexus 6000: First Impressions
Hi Omar

Thanks for your comment.

On VXLAN & Multicast - as I noted in my previous article - http://www.networkcomputing.co... - the use of Multicast technology is also risky. Maintaining stable multicast trees is challenging, and securing them in a multi-tenant network is complex and operationally failure prone.

So I welcome the new features of the Nexus 1000V to perform the packet replication in software at the edge of the network and to improve the reliability and trust I can place in my Ethernet core.

greg
EtherealMind
50%
50%
EtherealMind,
User Rank: Apprentice
2/2/2013 | 11:05:54 PM
re: Cisco Nexus 6000: First Impressions
Hi David, thanks for your comment. I'll respond to those of your topics that aren't marketing noise.

My opinions are coloured by my professional experiences with the Nexus 7000 over the last 4/5 years.

# Nexus 7000 complexity

Cisco has chosen to pile features onto the NX7K creating a progressively more unreliable and untrustworthy system. The corporate focus on new services and fancy features means that the core of my data centre is now a risky and challenging environment. EPLD updates take an entire change window. ISSU reduces impact but does not reduce operational risk of route flaps, loop detection or multicast failures. Recent software experiences are that most Cisco software, across all business lines, ships with bugs, memory leaks and this translates into serious business risk. Cisco operates an ship early, ship often software model that encourages customers to find software faults. Under ITIL methodology, this is a serious change risk.

Further, my reseller/s and Cisco will not and do not provide assurances that the upgrade process is reliable and trustworthy. Professional services offering to conduct testing by both reseller & Cisco are poorly constructed, and come with no additional assurance in spite of the astonishing costs.

Therefore, the complexity of NX7K platform is a liability to businesses focussed on meeting stringent SLA. More simply, I can't rely or trust it and I'm looking for alternate solutions in the years ahead. Here hoping that the simplicity of the NX6K offers a stable Ethernet core, and I'll move the NX7K to the edge of the network as a "Fancy Services" block with service modules.

# MAC Scalability

The TCAM configuration is "flexible". There is no certainty into the actual scalability of the NX6K since TCAM are allocated according to configuration, or consumption. This indeterminate condition undermines serious network planning. Thus claims for 128kMAC depends on the actual network, and less for IPv6. I find that ridiculous in 2013.

It's really past time that Cisco started investing in better silicon design & planning to implement the sorts of TCAMs, CPUs & Memory that are needed in backbone routers and core switches and stop using nasty hacks to make marketing math that looks good on the brochure.

I agree that customers can mitigate vendors 'going cheap' on the silicon with good designs - but at the price of constant operational vigilance. This breeds uncertainty and loss of faith in the product. It costs me serious amounts of money to analyse the limits of the NX6K TCAM capacity and to institute a monitoring program for long term risk reduction.

# IPv6

I find ludicrous the fact that, in 2013, IPv6 capacity is 40% that of IPv4. Cisco has had 10 years to plan new silicon and products. TCAM designs exist today that have much more capacity than this and would remove this problem. Why then does Cisco choose what appears to be cheap and lazy instead planned and growth oriented ? Is it because the products have spent so long in development that the silicon design is more than 5 years old ? Is the software development process at Cisco really that problematic (as I'm often hearing) ?

Cisco promotes the concept that DC switching should last for decades, and price products in a 5-7 year depreciation. It seems reasonable to expect Cisco to offer a product that could last that long.

Thanks for your comments. I trust that you will take the message back to Cisco engineering that reliability is the future of networking as much, if not more so, than fancy features.

greg
omarsultan
50%
50%
omarsultan,
User Rank: Apprentice
2/2/2013 | 9:33:28 PM
re: Cisco Nexus 6000: First Impressions
Greg:

Thanks for taking the time to review our latest member of the Nexus Family. Positioning of a given switch is always an interesting conversation. With the N6K, we see it as a pretty flexible switch--beyond the use cases you noted, we expect a lot of interest for the switches in building a scalable access layer. Both FabricPath support and FEX support open up a number of design options. I think customers will find a number of interesting things to do with the density, performance and L2/L3 capability.

On a slightly different note, with regards to VXLAN and multicast, last week we announced/showed a couple of approaches to VXLAN without the need for multicast, which should be available in the next quarter or so (more details here: http://blogs.cisco.com/datacen... )

Regards,

Omar (@omarsultan)
Cisco
David Klebanov
50%
50%
David Klebanov,
User Rank: Apprentice
2/2/2013 | 4:33:59 AM
re: Cisco Nexus 6000: First Impressions
Hi Greg,

Disclaimer: I work for Cisco.

Good review, however, I have to disagree on the few points you made. Sorry, but this will be long :-)

1. Nexus 7000 is not oriented towards service providers, it is all around feature-reach Data Center platform for business of all kind. That is why it exists in four form factors for various deployment models.

2. "Service providers and large-scale cloud providers have found that the Nexus 7000 isn't well suited for their needs and have turned to other providers". Some of the world's largest service and application providers run Nexus 7000. I can't disclose any names of course, but trust me that your packets pass through Nexus 7000 when you shop and socialize around :-)

3. True that, as any other new platform, Nexus 7000 had its fair share of stability issues, but over the last several years Cisco made a tremendous progress in making the switch a perfect fit for both data and storage networks with the highest degree of reliability.

4. Nexus 7000 is complex? Can you please elaborate in what way? Nexus 7000 has an unprecedented versatility and what one may perceive as complexity, is in fact a creative way of achieving deployment flexibility.

5. Nexus 6000 has many use cases and yes, one of them would be at Agg layer of traditional DC multi-layer design, but I wouldn't say that it would be my first priority. N6K perfectly fits Cisco's Clos design providing flexibility in deploying it as Layer2 only or Layer2-Layer3 leaf node. Clos designs are becoming increasingly popular as customers realize true potential of true Layer 2 and Layer 3 spine-leaf architectures.

6. There are many scale discussions that get locked on MAC address scalability. Many believe that the more MAC addresses you can support, the better platform it is (your comparison to Brocade VDX). In reality, scale of a solution is far from just counting MAC address. Sorry to pick on Brocade, but look at VDX 8770 datasheet for things like maximum switches in VCS fabric, ECMP paths in VCS fabric, maximum VLANs, maximum trunk member per VCS fabric port and compare that to Nexus 6000 maximum limits in FabricPath category. There you will see true scale differentiation. As far as MAC scale, conversational learning in FabricPath network can go looong way to scale MAC addresses beyond TCAM table size comparison.

7. IPv6 is NOT a second-rate technology for Cisco! The reason we take several TCAM entries when programing IPv6 entries has to do with overall table efficiency to accommodate IPv4 and IPv6 at the same time. True, that IPv6 requires more TCAM space for such programming, but we can absolutely accommodate ANY size IPv6 deployment with Nexus 6000 using architectural approach with Clos switching fabric design.

Your comments are welcome!
David
@DavidKlebanov
Hot Topics
6
IT Certification Exam Success In 4 Steps
Amy Arnold, CCNP/DP/Voice,  4/22/2014
6
Edge Devices Are The Brains Of The Network
Orhan Ergun, Network Architect,  4/23/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
2014 Private Cloud Survey
2014 Private Cloud Survey
Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% ­extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.
Video
Slideshows
Twitter Feed