Did Brocade Bungle Cisco Test?

Brocade Communications Systems Inc. (Nasdaq: BRCD) has been privately circulating test results it claims shows Cisco Systems Inc.'s (Nasdaq: CSCO) MDS 9509 Fibre Channel director switch drops a small, but significant, number of frames -- which can be devastating to overall system performance.

The allegations are a potential bombshell, coming just as Cisco starts shipping its first FC switch into the market through its major reseller partners, EMC Corp. (NYSE: EMC), Hewlett-Packard Co. (NYSE: HPQ), Hitachi Data Systems (HDS), and IBM Corp. (NYSE: IBM) (see HP Moves Hard on Cisco and Cisco Gets Set).

Is it possible Brocade found an Achilles' heel in the Cisco switch that everyone else seems to have missed?

In short, it doesn't look like it. First, each of Cisco's partners have spent months testing the switch -- and all of them have certified it for use with their storage. Miercom, the firm that has performed paid performance tests for both Brocade and Cisco switches, says it did not encounter the problem Brocade describes, although Brocade points out Miercom did not run a frame-loss test.

And after reviewing Brocade's test results -- which have not been validated by a third party -- Cisco discovered that frame-loss errors are reported if the Spirent Communications Smartbits testing equipment Brocade was using isn't properly configured with a synchronization cable."We have found that there is absolutely no issue with the switch," says Tom Nosella, senior manager of technical marketing at Cisco. "When you find something like this, you need a second tool to verify your results."

Brocade, trying to bolster its case that the Cisco switch has critical performance issues, shared its internal test results last week with Byte and Switch. It tested an MDS 9509 with version 1.1(1) firmware and configured with 16-port line cards. Using Spirent's SmartFabric test application, Brocade ran a 48-port, full-mesh test with 2,148-byte frames, testing for out-of-order delivery (OOD) and frame loss. It spread the 48 ports randomly across the seven blades in the 9509's chassis.

With in-order delivery (IOD) turned on and tested with a single crossbar, Brocade found that the MDS 9509 returned the following error rates:

Table 1: Brocade Test Results for Cisco MDS 9509, Single Crossbar With In-Order Delivery Turned On

Note that if a Fibre Channel switch drops any frames at all, it's cause for concern because the condition can impair performance.Brocade, which says it only recently discovered this issue, also shared the results of its testing of the MDS 9509 with Merrill Lynch & Co. Inc. analyst John Roy. Based in part on that information, Roy upgraded his rating on Brocade from Neutral to Buy, which helped push up the stock 20 percent last week Tuesday (see Brocade & Cisco: Who's Out of Order?).

The problem is, those test results may be invalid.

Nosella says Cisco replicated Brocade's test using Spirent's SmartFabric application, and discovered that on roughly 15 percent of the tests SmartFabric reported dropped frames. Then, Cisco looked at data from the switch's low-level ASIC counters and found that, over the course of several days, the MDS 9509 did not lose a single frame along the path. "The MDS is clean," he says. "It doesn't drop frames."

He adds that Cisco subsequently ran the same SmartFabric test against the Brocade SilkWorm 12000 and found that in four tests out of 11, dropped-frame errors showed up.

Spirent and Cisco say they have since isolated what was causing dropped-frame errors to be reported incorrectly by the SmartFabric application: A synchronization cable between the two Smartbits chassis must be in place for the test to work properly.Mark Fishburn, VP of technical strategy at Spirent, says once the sync cable was put into place neither Cisco nor Spirent was able to find any dropped-frame errors. "However," he adds, "we're still investigating it because Brocade is claiming their chassis were synchronized."

Indeed, Brocade insists it's still seeing dropped-frame problems on the MDS 9509. "We always run our tests with the cables synchronized, and we still see the intermittent issue on Cisco with the latest code," says Ron Totah, senior technical marketing manager at Brocade.

Nevertheless, Fishburn expresses dismay that Brocade publicized the results of its MDS 9509 test without first getting a second opinion: "That's really an unfortunate thing, that they've done this test in private that wasn't validated by anyone outside of themselves."

Cisco's key distribution partners, meanwhile, say they haven't seen any performance issues with the MDS 9509 that haven't already been resolved. "EMC has been running the Cisco MDS 9509 through extensive qual[ification] testing," says EMC spokesman Dave Farmer. "The system performs to customer expectations, and we're on schedule to ship this quarter." (See EMC, Cisco Do the Deed.)

EMC, in particular, is well known for thoroughly testing third-party products for potential compatibility or performance issues. Recently, for example, it issued a note saying it will not support McData Corp.'s (Nasdaq: MCDTA) Open Trunking feature because it found an unacceptably high rate of out-of-order frames in its internal testing (see EMC: McData's Trunking Needs Work).Roger Archibald, VP of infrastructure and NAS in HP's Network Storage Solutions group, says HP did find some issues early on in its testing of the MDS 9000 having to do with protocol timing between the Cisco switch and HP's arrays. "They fixed the issues," he says. "Otherwise, we wouldn't be shipping the product."

He adds, "As we do testing, our focus is working with the partner, not with one of their competitors."

Such aggressive marketing campaigns by Brocade against its competitors -- also known in the industry as spreading fear, uncertainty, and doubt, or FUD, about a rival's products or technology -- appear to be a standard part of its business practices.

In August 2002, for example, Brocade issued a "competitive brief" to its sales staff, alleging that sending 2-Gbit/s traffic to McData's Intrepid 6064 director caused the switch to crash. No such issues have ever been reported by McData, its partners, or customers pertaining to the 6064.

But Brocade makes no apologies for its competitive marketing tactics, saying it's just good business. "The research we do is meant to help our field sales force win deals by highlighting our strengths over our competitors, and is usually not meant for public consumption but rather just customer discussions," says Totah.Whether that research is totally accurate or not, though, seems to be another matter.

Todd Spangler, US Editor, Byte and Switch