We've long realized that a perceived lack of commercial support has put a damper on Linux adoption in the enterprise. Even though most distribution vendors, such as Red Hat and SuSE, have support offerings, they are distribution-specific, possibly even hardware-specific (for details on vendor support offerings, see the chart, "Price Range and Pricing Method by Vendor/Offering). But what if you're running more than one distribution, plan on switching distributions or are unhappy with the vendor's support? Who ya gonna call?
We set out to find the answer. First, we identified five vendors--Caldera, Linuxcare, Hewlett-Packard, MissionCritical Linux and IBM Global Services--that offer distribution-neutral technical support services; we also invited Linux Support Services to submit its paid enterprise offering. Despite a number of e-mail and phone messages, MissionCritical and LSS did not respond--not a good sign for a support provider (we did, however, look at LSS' free service; see "You Really Do Get What You Pay For," below).
First, we sent out a detailed RFI for our fictional company, Surreal Systems (read the vendor's responses to the RFI, below). Second, to test the services, we came up with a two-pronged methodology. We asked PR contacts at the participating vendors to set up accounts for Surreal. We also threatened the contacts with all kinds of nastiness if they flagged the accounts, and we believe we stayed anonymous. We then devised a set of questions (see "The $10,000 [Linux Support] Question," below) and called the vendors' technical support lines at predetermined times of day.
In our report card, we have a numerical tie between Caldera and Linuxcare. The factor that pushed Caldera ahead, though, was the company's aptly named TEAM support model. A single point of contact was assigned to Surreal, and we came to feel that our technical account manager was, indeed, part of our team. Linuxcare was a strong challenger thanks to its wide distribution and application support and the depth of knowledge and commitment to education shown by its staffers. HP came in third, but its reasonable pricing model snagged it our Best Value award. Bringing up the rear was IBM, whose pricey service left us cold. Although the IBM staffers answered our questions correctly, we spent way too much time on hold and verifying our contract, name and phone number.
How We Tested: Linux Support Services
We weren't satisfied with reviewing distribution-neutral Linux support services based solely on responses to our RFI, so we devised a plan to test the services directly. We created a fictional company called Surreal Systems, made up some contacts and basic system information, and asked each vendor to set up an unflagged account for Surreal. Over the course of a month, we called technical support to assist us with a variety of problems and configuration issues.
Each vendor heard the same set of concerns, ranging from fundamental configuration troubles to "Omigod! I can't log in because I lost my root password." Not only did we time the calls to compare how long we sat on hold and how long it took to resolve the issue at hand, we also took notes on the responsiveness of the technical staffs and on their apparent competence.
While on the phone with support staff, we walked through the proposed solutions by performing whatever commands were necessary on systems in our Green Bay, Wis., Real-World Labs® and made sure the solutions worked correctly. For some situations, we mocked up machines with errors to ensure that all information given to the support staff was technically accurate.
The remainder of our evaluation was based on the replies to our RFI. We compared the vendors' claims about staff experience, hours of operation, after-hours support and options for contacting vendors as part of the grading process.
The $10,000 (Linux Support) Question
When we set out to review Linux support services, we wanted to do more than just compare RFIs. We wanted to experience what you experience. So we reached into our bag of tricks and conjured up a company, Surreal Systems. We then discreetly talked to each vendor's PR department and requested that an account be created for our company and its fictitious support contacts.
We had to pull out our thumbscrews for IBM, which initially set up an account for Network Computing. We also warned the participants that if we suspected their support staffs had been tipped off, we'd say so in print. After we finished our calls, we fessed up to support staffers and tried to find out if they'd been tipped off. Linuxcare's Bill (who rocks) said he figured it out after the second call. Caldera TEAM Services' Jim, our assigned support engineer, responded with "Really?! I didn't know!" IBM's and HP's services were so far from personal that we never talked to the same support engineer twice, and considering both companies' impersonal approaches we're fairly certain that both IBM and HP staffers were in the dark.
After reviewing the responses and service, we're confident that you would get the same level of service as our fictitious company. While Linuxcare saw through our charade, its attitude is one of dedicated customer service, and we don't believe the discovery tainted our review.
Here are the questions and answers, and why we asked them.
Q. How do we stop the apmd service from running permanently?
A. In other words, How do we prevent a service from starting when the machine boots? To answer this question, it's necessary to understand runlevels and startup scripts and how the two interact. Before you can stop any service, you need to know what runlevel you boot into, and that means you need to know the Red Hat command to tell you that info, or you need to know what file the info is stored in (/etc/inittab). You also have to know how Linux determines what scripts to run at what boot level (/etc/rc.d/rc.X/, where X is the runlevel) and the naming convention of the scripts contained in each directory. Stopping the service is simple because Red Hat distributions provide a "setup" utility that lets you manage, among other things, what services are started. Only Linuxcare offered this as a solution; the others asked us to rename various scripts or even remove them entirely.
Q. How do we mount a remote SAMBA share automatically at boot time?
A. We asked this question to determine how well the support staff understood mounting of remote file systems and use of SAMBA. Using SAMBA for file sharing is common in a heterogeneous server environment, and it's tedious to mount a remote share every time you boot manually. To pass the correct arguments, you must understand SAMBA and how file systems are mounted in Linux. This question gave our support contacts more problems than we had expected. In the end, every vendor correctly instructed us to add the remote share to /etc/fstab with the proper user name and password as arguments.
We also waited for each vendor to help us in securing /etc/fstab after putting a clear-text password in the file. HP and Linuxcare support staffers mentioned the security risk and assisted us in changing the permissions on the file to keep the password hidden from regular users.
Q. Help! We lost our root password!
A. There are several methods to retrieve lost or changed passwords, and we were given two completely different solutions. We asked this question primarily to see if any vendor would question us as to how we lost our root password. Not one support engineer asked us about a possible intrusion on the machine, even though this would be a valid reason for our password not working.
This question requires a knowledge of runlevels and how to manage those runlevels--booting into single user versus multiuser as well as how to change runlevels from the command line. Caldera ignored the ability to boot into single-user mode to change the root password and instead suggested a series of steps that required us to mount file systems as well as change the permissions on various file systems to get at the passwd utility. Because we, of course, didn't know which partition the root and /usr file systems were supposed to be mounted on, we forced the support engineer to explain where to find this information.
We also wanted to determine if support would explain or encourage how to secure single-user mode. We were pleased when HP and Linuxcare provided info on how to lock down the Linux boot options as well as secure lilo.conf from prying eyes.
You Really Do Get What You Pay for
We figured, hey, Linux is free, why not check out a free support option? Linux Support Services offers two levels of distribution-neutral Linux support: free (as in beer) and enterprise. The free support is truly free, but before you rush to the support site, you need to know a few things.
First, the support site is entirely Web-based and staffed by volunteers. There are no guarantees on the level, speed and quality of the support you'll receive, so caveat emptor is the phrase of the day.
With this in mind, we kept our expectations low; we essentially aimed to find out whether the free service would be at all helpful. The first question we submitted was answered satisfactorily via e-mail in less than 24 hours. This raised our hopes that perhaps the service would be good for issues that are not time sensitive. Unfortunately, our next question--a relatively simple inquiry--remained unanswered for the remainder of our review period. We submitted a much more difficult question, just to be thorough, and, as expected, never received a response.
We concluded that the free support offered by this organization may be helpful to those who aren't in a hurry to receive answers. But for the enterprise, depending on a free service for support is almost like not having any support at all. So if you plan to use a free service or newsgroups as your primary support option, when you head for the wiring closet, you'd better bring a pillow. And maybe a beer.
We sent out a detailed RFI for our fictional company, Surreal Systems to the four companies - IBM, Caldera LinuxCare and Hewlett-Packard. Their responses to our questionnaire are below in PDF format.
REPORTS
Analyize In-Line NAC strategies and products.
ANALYTICS Plan and design your enterprise blade server deployments
InformationWeek U.S. IT Salary Survey 2008
Salaries for business technology professionals are falling. Here's what you need to know in order to make good hiring decisions and personal career choices. Purchase Today: $299
REPORTS
ANALYTICS
Follow 
