How is Performance Measured?

Measuring bits and bytes related to your CIRs (committed information rates) is fine for ISPs, but measuring human-based performance is much more subjective. You must define goals, deadlines and other deliverables, such as performance benchmarks. Some good service metrics to define include initial call-response time, call-resolution time, end-user satisfaction via surveys, numbers of calls or incidents handled, server uptime, server-response time, and total network availability. This is a lot to track, but if you set it up right, your provider will do the tracking, and you will do spot-checks.

We can't overemphasize the importance of not only collecting the data but making sure a pair of intelligent eyes scans it. Like a cell-phone bill that doesn't get reviewed, a performance report that doesn't come in human contact could contain large problems. Set up the particulars for this when you devise your service contract. (For more on performance measurement and making sure those levels are met, see "Let's Make a Deal.")

Is the Contract Monolithic or Flexible?

Is the service time-based? Or can the ultimate objective be broken down into small deliverables, such as "migrate a department from NDS to ADS in X time." Sometimes you're better off purchasing these small deliverables ý la carte, rather than committing to a huge monolithic fee that covers a large project or period of time. On the other hand, economies often are built in to large purchases, and if you are comfortable with the provider, the monolithic contract makes sense.

Vendor Viability Analysis Click here to enlarge

The benefit of going á la carte is that you can back out of a relationship quickly with a provider you're test-driving. Sometimes, though, because of organizational restraints, you'll have no alternative to large contracts. Richard Zimmerman, an IT manager at DaimlerChrysler, says that many organizations, including his, need to have their 2003 budgets solidified by April of this year. Work with your purchasing department to achieve maximum flexibility within the rules of your organization.

However, be careful about buying blocks of hourly service, even though you'll likely get a discount when "buying in bulk." Do your homework before you commit to a service firm that collects prepaid amounts for blocks of service time. Scams still abound, says Steven R. Bryant, network manager for law firm Henderson, Daily, Withrow & DeVoe. "It was, by all appearances, a qualified and reputable firm," Bryant says about one service provider. "It then closed down with no warning, leaving clients with no service and less cash to buy service. While I've seen nothing that compares to 'Enron scale,' I have seen this scenario a couple of times and do suspect it happens with lots of smaller consulting/service operations everywhere." Lesson: Ask around, contact the Better Business Bureau, and talk to other businesses that have used the companies you are considering.

Are There Remedies for Failing to Meet Expectations?

Without counter-measures built into a contract, you can expect service complaints to go nowhere quickly. Although most of the service providers we sampled offer monetary compensation as a remedy for failing to meet service levels, we were particularly impressed with IBM Global Services and NetSolve. Both have money-back satisfaction guarantees. NetSolve offers a clever monthly money-back guarantee: It forces IT managers to identify problems early, which is better for IT and for the provider.

This is a good point to emphasize: When you get into a service-provider relationship, you give up hands-on technology management in favor of relationship management, which will require plenty of attention too. Taking a hands-off approach to technology management would lead to disaster; by the same token, if you ignore your outsourcing vendor for some time, or until things are totally fouled up, you'll be partially to blame.

Are Reliable References Available?

Providers, naturally, will give you only those references sure to check out well. One way to combat this is to go on site visits, even if they're to "golden child" sites. Bring along more than one person from your organization, and try to corner someone other than the primary reference contact at the site while your co-worker keeps the primary contact busy. This plan can be resource-intensive, at least for remote sites, but it's worth doing for long-term or high-dollar projects: It's amazing what dirty laundry this technique can expose. For instance, one of Network Computing's partner labs, when considering a storage-area network, learned from a reference's employee that the SAN being pushed by the vendor and reference hadn't even gone into production yet.