Firm Swaps WAN Optimizers, Learns Tough Lessons

When Kansas City, Mo.-based 360 Architecture centralized its SAN in 2006, the IT staff didn't realize they were in for a crash course in the realities of WAN optimizers.

Centralization seemed like a great move. "Our storage network used to be very distributed," explains IT manager Jeff Hamer. "We had a number of network attached boxes, and we used to keep storage at the branch offices. We experienced the normal challenges, having to back up data at a remote location with no IT staff... You cant be sure someone isn't going to swap a tape."

To get all its data in one place and make it more manageable, 360 Architecture opted for an EMC CLARiiOn SAN with a Celerra front end for NAS functionality. The storage network it chose was iSCSI, and the SAN links were configured over a dedicated VLAN on a Cisco Catalyst 6509 core switch.

The systems were installed at the 130-employee-company's headquarters in Kansas City, Mo., which is linked to remote offices in San Francisco, Dallas, and Columbus, Ohio. The company's MPLS data center network runs at 3 Mbit/s and links from there to remote branches operate at 1.5 Mbit/s each.

Data migration was the next step. "Once that storage system was implemented, we brought back all the data, so we could more efficiently back it up and protect it," Hamer says. He used EMC's Replistor application, which he says was simple enough to use, if time consuming. "We used DFS [Distributed File System] as a layer of abstraction between the actual network location and the end user’s drive letter mapping, so the cutover changing the share paths was seamless. Total relocation of branch data to the core spanned a couple of weeks."But the migration brought another issue to the forefront. "All of the branch offices needed access to the data," Hamer says, adding that network links to remote sites were insufficient to offer adequately fast delivery of CAD files and other strategic data to the branches. What's more, the group realized they required a product that ensured data integrity along with performance.

"We knew from the start that a collaboration solution needed to be a major part of the data centralization. Prior to migrating all data to the new SAN, we tested different replication solutions, software based file-locking solutions... None of them offered the functionality or performance we were looking for. At that point we realized a full WAN optimization solution is what we needed."

The group looked at wares from Cisco, Packeteer, and Riverbed. They finally decided Cisco’s would be the best to meet the data collaboration issues as well as the WAN optimization, and they installed that vendor's Wide Area Application Services (WAAS) solution in May 2006.

It was a decision Hamer would regret.

To Page 2"The first six months were quite challenging. We had performance issues from the start," Hamer says.

It was tough to find the source of the problem, but at least part of it was due to a mismatch between Cisco's engineering approach in this case and 360 Architecture's data requirements.

In the spring of 2006, Cisco did not yet offer a WAAS solution that worked in band, or in the network path. Though it would release one in March 2007, Cisco's WAAS solution for Hamer worked alongside the catalyst switch. Its optimization was based on use of Web Cache Communication Protocol (WCCP), a Cisco protocol designed to redirect selected types of traffic among routers to caches on the Web. CAD files, for instance, would be redirected to a Web cache in order to speed up response times.

The approach required 360 Architecture to adjust its entire Layer 3 network. "The Cisco WAAS solution was deployed in a WCCP configuration at Cisco’s request because that is the configuration they recommended for optimum performance," Hamer says. "Then we had to change configuration on every router in the network... Changes on all our network routers [was] a difficult process for a company with multiple branch offices."

While Cisco did not comment on the case study for this article, one source there says many Cisco customers prefer an out-of-band solution. "Enterprise customers unanimously select out of band," says Feng Meng, product marketing manager for WAAS at Cisco. The reason, he says, is that an in-band solution always presents an unacceptable point of failure for large companies. If all traffic is actually passing through the optimizer, a failure could stall data center activities."In-path is more appealing to smaller organizations, you don't need WCCP... But long term, it presents a single point of failure," Meng says. Cisco mitigates this by providing automatic failover for its in-line solutions in the event of hardware, software, or power failure. "Still, if you have a box directly in the data path, resiliency is not as good as out of path."

Another issue was 360 Architecture's reliance on computer-aided design (CAD) files. Autodesk, a popular CAD application, accounts for nearly 60 percent of the architectural firm's data traffic to remote sites. "We saw the worst performance with that," says Hamer. He says Autodesk applications utilize digital signatures, which "Cisco didn’t play well with." The result was slowed-down access.

But the biggest problem was that Cisco's WAAS appliance was dropping alternate data stream (ADS) requests. These requests, which are a normal part of CIFS traffic on any Windows-based network, carry information required to access files and folders.

After six months of troubleshooting, the problems had not been solved. "The client systems expected a response to these ADS requests, and not receiving one, they would continue to issue the request dozens of times before giving up, causing the substantial delays in file browsing," Hamer recalls. "We worked through many levels of Cisco TAC, including a member of the WAAS development team, who finally determined the product was working as expected and there was nothing else they could do. At that point we had to make the decision to pull the Cisco solution."

Clearly, Hamer and his group were experiencing firsthand that not all WAN optimizers support particular applications in the same way. Best practices demand a balance among applications, user expertise, and the supplier's bent. (And most optimizer suppliers tend to have specialties.)"Cisco did not seem have enough experience with CAD usage over optimized WANs to anticipate or identify issues with digital signatures on CAD files," Hamer states. "Six months of daily troubleshooting and providing countless network traces to Cisco was not enough to yield a simple recommendation of disabling digital signatures for Autodesk applications."

The architectural firm decided it was time to check other solutions. "I had started to talk to Packeteer and Riverbed so we could test their functionality," he explains. Both companies sent out demo boxes in early 2007, and 360 Architecture installed them both.

Neither Packeteer nor Riverbed had the problems of the Cisco solution. For one thing, both companies offered boxes that worked "in path," or within the network, instead of alongside it at the switch. The WAN optimization would work with the existing router settings, instead of requiring modifications.

Riverbed also recommended that the firm disable the digital signatures in Autodesk. It worked like a charm. The Riverbed system is now able to handle the company’s Voice over IP, file sharing traffic, and all of its applications without requiring changes to the network. The Riverbed appliance also contains fail-to-wire NICs that address the single point-of-failure issue that surfaces with in-band solutions, Hamer says.

Presently, Hamer says he gets "upwards of 4X improvement in WAN speeds." For every 40 Gbytes of data requested over a WAN link from a branch office, for instance, only 10 Gbytes actually traverse the connection. This makes the 1.5-Mbyte links "feel like" 6-Mbit/s connections, he maintains.Ultimately, Riverbed won the three-horse race for 360 Architecture's WAN optimization setup. Hamer liked the management interface on the Riverbed Steelhead appliance because he felt he could see better what was going on compared with the competition -- what traffic was optimized and what kinds of results were being obtained. In the end, the group paid "marginally" more for the new unit than Cisco's price; the deployment cost 360 Architecture about $36,000, Hamer says.

Hamer's group has established an environment in which people in all of the firm's offices can collaborate using the same files and applications at high speeds. "Before, we didn’t have a way of collaborating and using the same set of data," says Hamer. "Now we really have one company."Have a comment on this story? Please click "Discuss" below. If you'd like to contact Byte and Switch's editors directly, send us a message.