eBay

ORLANDO, Fla. -- Storage Networking World (SNW) -- A rapidly growing business is obviously a CEO's dream, but it's not necessarily a lot of fun for the IT staff, as e-commerce site eBay's Paul Strong found out when he joined the Website a year ago.

Strong, eBay's distinguished research scientist, told attendees at SNW today: "We experience what every business wants -- exponential growth. Businesses love when the graph goes up and to the right, but it's every IT person's nightmare. Our scale may be extreme, but most of you will eventually feel this type of pain."

Extreme is putting it mildly. eBay has 212 million registered users and draws 1 billion page views and 26 billion SQL queries in a typical day. The site averages nearly 6,000 SQL transactions per second per database and hits 16,000 during peak times.

To handle that, eBay has 13 discrete SANs -- each dedicated to a particular quality of service. Its data centers have 2 Pbytes of raw storage, 8,000 SAN ports, 700 SAN-connected hosts, and more than 100 database clusters. eBay adds 10 Tbytes and 75 LUNs per week.

Ninety-five percent of eBay's storage is on SANs, with email, messaging, and fixed content on NAS.Strong wouldn't name any vendors, but eBay is a big Sun shop and he came over from Sun. (See Sun Powers eBay.)

eBay uses a grid to manage servers, databases, and storage tools in aggregate pools, and the company develops most of its storage management tools in house.

"We treat our entire data center as a grid," Strong says. "We integrate all of our tools into the grid. That includes storage and data management."

Why build applications to manage all that storage?

"We have to," he says. "We get commercial off-the-shelf solutions and do horrible things to them. Commercial storage management tools are great at managing heterogeneous environments with the lowest common denominator. We break them. If there's a lot of customization involved, we just build it. Why pay licensing fees if we have to customize it anyway?"eBay's in-house tools help the IT staff build scalability and redundancy into its storage, which Strong fondly calls "rotating rust." Its data centers are set up with "multiple SAN paths, up to four paths per LUN for highest performing storage. We maximize our ports, cache, and disks per array," he says. He keeps up to four copies of each database. To split traffic evenly, eBay carves its database clusters into around 600 partitions.

Because eBay's site is its business, Strong's goal is to keep the site up and running at close to 100 percent at all times. That means code changes and anything else that affects the databases must be done while the site stays up.

"It's like changing the engine on a turbo jet at 36,000 feet," he says. "We can't stop, we can't land, and we have to do it with minimum impact on users."

eBay has grown from 1 million outbound emails and 54 page views per day in June of 1999 to 35 million outbound emails and 874 million page views by the end of last year. Yet availability rose from just under 97 percent to 99.94 percent.

Still, Strong feels there is room for improvement -- or at least room to do as well with a lot less manual intervention. He's a big proponent of grid standards as a way to automate processes."We partition databases, create LUNs, then the DBAs do their thing, then the server group," he says. "We want to get rid of those steps and automate the flow."

That's where standards and good resource management products come in.

"We'd replace our storage management tools with standards-based vendor tools if we could," he says "We consider storage management outside our core competence. We would rather use commercial products to manage. That's why standards matter to us."

Cost matters, too. Strong says cheaper SANs head his wish list. He maintains cheaper systems and storage is offset by larger per port infrastructure costs. "That constrains our ability to scale," he says.

Dave Raffo, Senior Editor, Byte and Switch