Kindred Healthcare

NEW YORK -- Storage Decisions -- While moving from McData to Cisco switches, Kindred Healthcare proved forklift upgrades don't have to be painful. They might even benefit a business if done with enough planning.

In a presentation here today, Kindred storage manager Tim Hesson said that in mid-2005 he decided to move from a core-to-edge SAN to a more consolidated switching structure. The decision led him to replace 26 McData switches with four Cisco MDS 9509 directors.

Hesson refers to his topology before the upgrade as "Switch City," with a variety of devices and fabrics accumulated over several years as the Lousiville, Kentucky-based firm acquired companies. Kindred consists of hospitals, nursing centers, rehab centers, and pharmacies.

"We had strictly McData -- core directors, switches, pseudo directors, and fabrics all over from acquisition after acquisition," Hesson says. "We built this fabric over here, this fabric over there."

For Hesson's group, the problem was more the messy topology than McData switches. But Cisco had a couple of advantages that prompted him to switch vendors:

Hesson says he planned the project around improving the firm's business processes by solving a few problems. First off, Kindred's legacy SAN design was hard to manage. It was down to about 10 available storage ports, because more than 100 ports were tied up with interswitch links (ISLs) that connected switches. Also, the healthcare provider was using direct-attached tape for disaster recovery, and Hesson wanted to tie the tape into the SAN.

Hesson says Kindred has more than 200 Tbytes on EMC Clariion and Symmetrix SANs, and he expects that to more than double this year.

"Not only am I going to try to solve today's issue, but I want to look ahead at least three years out if not five years," he says. "If you have good leverage with your vendors and are getting competitive pricing, you're not going to get that leverage two years from now when you're just adding something.

"I said 'If we're going to spend this much money, why don't we see what we can do to add value?' I wanted to get away from ISLs. I want to improve performance, but I don't want to tie up ports, because ports are money. We also needed to connect to our disaster recovery site in Philadelphia."

Still, doing forklift switch upgrades can be a daunting task. Hesson says it required a great deal of planning, testing, and training to pull it off in seven months -- 75 days to plan the migration from McData to Cisco switches, 45 days to install the switches, and 90 days to migrate. He says the migration took place with only one instance of downtime, and that instance involved a development system, not a production one. Hesson says the cleaner topology also helped save Kindred $200 a port.Hesson added two more MDS 9509 directors early this year. "It took about two weeks with no downtime," he says. "I just can't stress enough the difference [that] using a modular design brings. I can switch any of these blades out at any time. You can add storage ports and host ports. When new blades come out with better capacity, I can pop the old ones out and pop these in."

Hesson attributes the smooth migration to several factors. One of them is putting together a team dedicated to doing the upgrade without scheduling any other major projects at that time.

"You're going to have your whole fabric broken apart," he says. "That's not a time to introduce another major change. So we did a freeze on major changes -- no major changes until we were out the other side of this. It was very good for us to have no other moving parts, less things to break."

Another factor was picking the right professional services partner, he says. In his case, he went with Cisco.

"Vendors are great before the check is cut, but when something goes wrong at one o'clock in the morning they're nowhere to be found," he says. "Or they tell you its the HBA or the switch. Look for a vendor you can count on. We chose Cisco even though we were an EMC shop because I felt I could count on them and I was pleasantly surprised."He says Cisco came through when Kindred had a problem with the supervisor control processor in one of its switches. "These are the points when your vendor can walk away," he says. "Cisco not only replaced the supervisor they found at fault, but replaced every supervisor in our infrastructure."

Next up, Hesson's looking to add data classification and perhaps block virtualization. He says he uses Cisco's VSANs (virtual storage area networks), which lets him partition traffic over the switches. (See Cisco Flaunts Fancy SAN Features.) The good thing is he can isolate his tape drives and patient care system for better performance.

"The downside is, you're locking that array down," he says. "If you want to squeeze every nickel out of your arrays, you can't get there if they're in separate VSANs. We lost some of the flexibility of tiered storage. I'm excited about the next level of products [intelligent switches] with block virtualization."

— Dave Raffo, News Editor, Byte and Switch