Safely Purchasing Second-Hand Storage Gear

Our anonymous IT director acknowledges that buying used has its drawbacks, including longer wait times for replacement parts. With the "phone home" technology built into some new products, replacement parts are at the door sometimes before you know you need

them. With used gear, you'll wait a few days. Still, it's an inconvenience he'll put up with to get the cost savings--as long as the used gear comes from a reputable supplier.

"Once you find a reliable source," he says, "you stay with him."

Sales of used storage gear sales total about $50 billion annually, according to the Association of Service and Computer Dealers International, a 35-year-old nonprofit organization of 220 members. Institutionalizing secondary-market practices has boosted consumer confidence in the used channel, says associate chairman Joe Marion, who cites the sharp increase in the number of deals ASCDI members are reporting. In 1995, he says, equipment sales totaled $25 billion to $30 billion.

Not Created EqualDepending on whom you ask, we're not talking about the "gray market" for storage technology. In fact, just mentioning the term "gray" to a used-equipment dealer bought us a lengthy introduction to what can best be described as a caste system in the secondary market.

Gray-market equipment is actually new equipment that has been purchased through indirect channels, usually on terms the vendor hasn't formally sanctioned. End-of-quarter inventory dumping by distributors--and some vendors--floods the market, but with one major gotcha: It may be difficult to get a warranty or service agreement on gray-market gear because a simple check of serial numbers will alert the vendor that the equipment was not sold through approved channels.

"If General Motors buys gear from a distributor, then turns around and sells it as used gear, that's OK with the manufacturer," Marion says. "If the distributor holds a fire sale at the end of the quarter and dumps new equipment onto the gray market, that's not OK. And you may not get to recertify the gear for maintenance."

Gray marketers are often accused of deceptive business practices, both by vendors who don't like competing with their own new-in-the-box equipment and by used-gear vendors who say gray marketing taints the entire secondary market. While slashing acquisition costs, gray-market equipment can dramatically increase operational costs over time. Without a warranty, even the smallest problem can become a maintenance nightmare. And without valid licenses on installed software, customers may have no access to vendor patches and upgrades, and may actually be breaking the law.

In contrast, secondary-market resellers deal in pre-owned gear, pure and simple, says Paul Leeber, a vice president at Hanover, Mass., reseller BL Trading. Their products consist of demo equipment, off-lease gear and operational equipment pulled off the line when a company either decides to upgrade or goes out of business. The top dogs in this market, which include BL Trading and Zerowait as well as N-1 Technologies in Westchester, Ill., are often both resellers of new and purveyors of used. Leeber's company, for example, sells new and used EMC gear.Aware that his used-equipment business is a source of annoyance for EMC, based down the road in Hopkinton, Mass., Leeber tries to lead with new EMC gear.

Not all vendors have a problem with secondhand sales. IBM, for one, offers an aggressive hardware recertification and rewarranty program. By getting to know who's who at Big Blue, Greg Morgan, director of equipment trading at N-1, landed some attractive deals for his company--like used Shark disk packs at 10 percent off list price and blessed by IBM for either its own or third-party maintenance.

Zerowait is an exception to the new and used sales model. Formerly a reseller of new Network Appliance gear, the company now focuses on providing used NetApp hardware, together with private maintenance and warranty support. Zerowait CEO Michael Linett says his company, which has the engineering expertise to do its own maintenance, delivers better value to consumers than the original equipment vendor. To bolster his case, he hosts a blog site, zerowait.blogspot.com, that addresses the foibles of NetApp gear.

"I just can't see how a high-availability storage platform becomes less highly available because the vendor has released a new model," Linett says. "Some vendors take their equipment off support in 18 months when they begin shipping new products, and they try to force consumers to upgrade to their latest gear when what they've already installed still has a lot of value to offer."

Despite his occasional lambasting of NetApp, Linett has allies at the vendor. For example, our anonymous IT director says he turned to Zerowait for used gear on the recommendation of a NetApp sales rep. But it is Zerowait's performance, not the glowing reference, that keeps the IT director's voicemail company coming back.

So how can you tell a top-notch reseller from a fly-by-night? Look for companies with expertise in the used equipment they sell. Top resellers will cherry-pick used gear that comes on the market for its conformance with the recertification criteria used by vendors or large third-party warranty outfits, like IBM Global Services or DecisionOne, and they often offer their own support staff and maintenance agreements on the type of gear they sell. Either way, these resellers offer a solid value proposition.

The top dogs also know the ins and outs of the recert and software-relicensing side of the business, Leeber observes. "Legally, you can't license something that, if it is absent, makes a machine nonoperational, like the BIOS on a PC," he says. "But you can license, and restrict the transfer of licenses on, software that makes the machine productive."

That's one reason you rarely see the resale of an EMC SYM 5 Symmetrix. "After SYM 5 came out, EMC required the use of proprietary tools to maintain its gear," Leeber says. "There are only a couple of major players that handle third-party maintenance on this platform, so when you see an order for one used, it's an odd duck."

Beneath the top strata of the secondary market is a group that Leeber calls "Bedroom Brokers" or "Tweeners." These are middlemen who contact large consumers or major resellers, find out what they're looking for and hunt up a supply. The good ones, Leeber says, have good contacts. Others simply monitor listing services to see what's available, then buy and sell the gear like brokers, taking a small cut on the transfer.Recent years have seen the consolidation of used-equipment listing services into large portals, like processor.com and powersourceonline.com, making it easier for major resellers to circumvent these secondary brokers. The portals provide a location for used-equipment suppliers to list their inventory of products and parts. They also provide a location where resellers can interact with the "bottom feeders" of the secondary market, often referred to as chop shops.

Chop shops buy old gear and harvest the parts. Their main interest lies in providing parts for maintenance service providers. Occasionally, a lower-tier broker will cobble together a full array from a collection of parts and represent it as a complete platform. However, such compilations generally can't be recertified, and these suppliers may find themselves blacklisted by reputable dealers.

Is used gear right for your company? The obvious pitfalls include equipment that can't be recertified or can't be operated to its full potential because of software-license restrictions. Other gotchas have to do with the quality of after-sale service and support provided by the seller, OEM or third-party maintenance service provider.

Marion touts the ASCDI as a good starting point for exploring the used option. ASCDI members subscribe to a code of ethics that is "actively policed" by the organization, he says.

However, ASCDI membership is no guarantee of success in the secondary market. As Leeber points out, a reseller's 20-year warranty is worthless if the company goes out of business tomorrow.Do the usual due diligence, Leeber says. Look up the reseller on Dun & Bradstreet. Check out its financials. Talk to customers. Find out how long the vendor has been in business and whether it has always operated under the same name or used a collection of aliases. Look for guarantees that the product is eligible for a manufacturer maintenance contract, or check that the supplier has staff technicians qualified to maintain the product. Support staff should have certifications, where they are available, on the gear the secondary vendor is offering, though these may be less important if the support staff has a long history of high-quality service that can be confirmed by customers.

Will Used Ever Go Mainstream?

The market for used equipment is cyclical, according to insiders. When the economy is good, business falls off, Leeber says. When the economy slows, business heats up. After 20-odd years in the business, Leeber can see the signs.

"Along with the economic cycle is the cycle of technology itself," Leeber says. "First, we brokered PCs, then LANs, now storage. In the case of LANs, everyone bought equipment from a single vendor. Then administrators started adding pieces from third parties. And when interoperability was assured, the used-equipment vendor came into the picture."

The same thing is starting to happen in storage. "Monolithic is starting to peak," Leeber says. "There are a lot of crumbs falling off the table, and brokers are there to take advantage of the situation."Linett agrees, referring to the current situation as the "democratization of storage." Soon, he says, you'll be able to get the kind of storage you need instead of what the vendor tells you to buy. Increasing acceptance of used gear is a harbinger of commoditization, Linett says, noting that the real money will be made not in new or used gear, but in service and support.

In the absence of data on product quality, there is still some pushback from prospective customers on the idea of buying used, Linett says. The reseller's only means for addressing this concern is its customer list.

"Quality is measured in terms of service," he says. "Any vendor who has been in business for 10 years will have a golden Roledex of customers it can offer as references." This is the same strategy a new-equipment vendor typically takes to convince customers about the quality of its latest products, which haven't been in the market long enough to garner feedback.

Leeber is cautious about predicting when this democratization will occur. Used-equipment sales accounted for only 20 percent of his business in 2004. He says the holdup, in the final analysis, is the IT manager: "I find customers saying, 'Why should I stick my neck out to save the company a few bucks? If it doesn't work, I may lose my job for having suggested the used option.' It's a balancing act, and it still isn't as cut-and-dried as the cost-savings arguments would make it out to be."

Then there's the problem of tech decision-making. Often, it isn't up to the IT manager to decide what to buy or from whom. One secondary marketer, who asked not to be named, reported that a big sale of used Sun Microsystems storage gear was just squelched by Sun itself. Upon receiving a request to recertify the used gear that an IT manager in the Pacific Northwest was preparing to buy, Sun descended "like a hammer" on the company's senior management. Sales reps told the CFO horror stories about used gear and convinced him that buying new equipment, still expensive even with the substantial discounts the vendor offered, was the better strategy.According to the secondary-market vendors we interviewed, Cisco Systems and Sun are high on the list of vendors most resistant to used-equipment sales. EMC and NetApp have so much proprietary software that relicensing can be a bear. The friendliest vendors, from the perspective of many used-equipment resellers, are Hewlett-Packard/Compaq and IBM.

"Big Blue is gold," says ASCDI chairman Marion. "If the equipment is up to spec, it has no problem putting the gear back under maintenance. It even sells its own used gear on eBay."

•FYI Caveat Emptor
One IT manager we spoke with said that prior to finding a good used-gear vendor, he placed an order through a broker who delivered 20 boxes that were not identical and clearly a kludge of used parts. Only after he stopped payment on the shipment did the vendor make things right.

•FYI server rotation
According to a Gartner survey, 45 percent of respondents reported storage server-replacement cycles of four years or longer.•FYI favorable forecast
The NAS and unified storage market will grow to $2.1 billion in 2008, up from $1.47 billion in 2004, Gartner projects.

Jon William Toigo is CEO of storage consultancy Toigo Partners International, founder and chairman of the Data Management Institute, and author of 13 books, including Disaster Recovery Planning: Preparing for the Unthinkable (Pearson Education, 2002) and The Holy Grail of Network Storage Management (Prentice Hall PTR, 2003). Write to him at [email protected].

To a certain extent, buying used gear runs counter to the conventional wisdom of storage acquisitions encouraged by the vendor community. In a recent news conference, a manager from IBM reported that more than 80 percent of Big Blue's customers came to the company for "integrated storage solutions." That's code for hardware and software preassembled and preintegrated by the vendor. Buying used gear often means stepping outside this comfort zone, an unsettling thought for CIOs who are looking to reduce complexity or who are short on tech staff to perform integration tasks.

However, many companies are learning that outsourcing their storage strategy-development process to a vendor has significant downsides.

Vendor product lock-in is one problem that may not become apparent until the IT manager seeks to bring in innovative technology from a third-party vendor. Of more immediate concern is the haphazard job of integration the vendor has made of its hardware and software. Storage hardware vendors typically use software to prop up otherwise unjustifiable pricing models for commodity hardware components. But unfortunately, the software is sometimes poorly integrated or missing important functions.A cellular telephone company recently deployed an "integrated, one-stop-shop" storage package from a prominent vendor, only to discover that many parts of the promised software functionality were still under development or didn't work as advertised. The company told us that even though it needed only 40 percent of the feature set provided on the expensive platform, it was required to pay for 100 percent of the software licenses.

This kind of experience has prompted a growing number of companies to consider the used-equipment alternative as part of a plan to develop a cost-effective storage infrastructure that supports specific application requirements rather than going with a one-size-fits-most setup from a brand-name vendor.

"Most of the CIOs and CTOs we deal with look at their data infrastructure in the same way that distribution companies look at the purchase of a long-haul truck," says Michael Linett, CEO of secondary-market vendor Zerowait Corp. "The first purchase is through the Peterbilt or Mack dealer, but the maintenance and upgrade, and replacement engines, are done by third-party maintenance organizations for reasons of price and long-term service relationships. Storage manufacturers push new models every 18 months. Their ROI is 18 months to three years, while the customer looks at IT investments to return results for four to seven years. This is the gap that secondary vendors fill."

In the final analysis, the decision to roll your own infrastructure must take into account many factors, ranging from convenience and application fit to the politics of budget approval and an honest assessment of on-staff capabilities.

Keep in mind that the current problem of oversubscription with underutilization--a condition that plagues most IT shops--usually can be traced to the way new equipment is packaged by primary vendors and to the poor quality of management across the infrastructure as a whole. Buying used can't fix this problem, but it can provide greater flexibility by freeing up capital resources to invest in better management technology.We all know people who wouldn't think of buying a brand-new car--folks who are quick to point out how many thousands of dollars a shiny new vehicle loses in value the moment you drive it off the dealer's lot. Then there are those who repeat the "buying someone else's problems" mantra and don't mind paying for that new-car smell.

Likewise, the potential cost savings of buying secondhand storage gear is not enough to lure all IT pros. Some see value in having a trusted vendor like EMC, Hewlett-Packard or Sun Microsystems spec out, install and maintain their systems, despite the price premium, and aren't unduly concerned with lock-in or the requirement to buy some functionality they may not need.

But if money's tight, in-house technical chops are plentiful and you're willing to do your homework, you can get serviceable used equipment at a fraction of the cost of new gear. The key to success is choosing your used-gear vendor wisely.

The nonprofit Association of Service and Computer Dealers International seeks to "promote and enforce high ethical standards of business conduct within the industry," according to its chairman, Joe Marion. The ASCDI Web site, www.ascdi.com, lets potential customers search its database of 220 members based on location, specialization and manufacturer.