Survey research is a science, a tool for gathering accurate information about the attitudes and behavior of a very large population based on a fairly small sample. A well-designed and properly administered survey can gauge the sentiments of a population of millions based on a sample of approximately 1,500 respondents. Many IT managers search, often in vain, for techniques and strategies that will let them accurately measure their success, or lack thereof, in meeting the needs of their customers. Some use surveys to achieve this goal, while others use ad-hoc focus groups; advisory committees are common as well. But are they getting any valuable information in the process?

Tell Me What I Want to Hear Most IT managers, and a fair number of independent research organizations, enter into a survey-research exercise with preconceived notions. They think they know the answers before they ask the questions. It's not a conspiracy, it's just human nature. Unfortunately, designing a survey instrument that is not contaminated by this bias isn't easy, even for professionals. It is particularly difficult when the questions elicit answers that are cast in shades of gray. It's one thing to ask people which candidate they supported in the most recent presidential election. It's another thing altogether to ask them to rate the effectiveness of their IT organization in addressing their need for access to information.

It's also common to see these projects spin out of control, resulting in a lengthy survey that demands too much of the respondent's precious time. This problem is particularly acute when the survey is produced by a committee, with each member contributing a few dozen questions he or she would like to include. It is much more effective to make user surveys short and focused. Both the response rate and the quality of the responses will be greatly improved.

When it comes time to administer the survey, project leaders often have a hard time deciding how to present the questions to individuals. The most traditional method is a paper survey mailed to a sample of customers. Alternatives include telephone, in-person and electronic versions. There are pros and cons to each method, but I favor the phone survey, because it usually produces the best response rate and also because it forces the survey designer to be clear and concise.

While surveys have their place, most are poorly designed and administered. But even high-quality surveys can fall victim to improper analysis. The biggest mistake occurs when IT managers attempt to generalize the survey results taken from a sample of respondents to the population as a whole. While a 60 percent response rate may sound great, it really misses the point. While professionals can accurately survey populations of millions by drawing samples of a few thousand, the proportional size of the sample required grows dramatically as the size of the population decreases. Thus, if you are trying to get an accurate assessment of satisfaction with system services from a population of 2,000 users, a sample of 5 percent or 10 percent or even 25 percent will not yield statistically valid results, even if the response rate is 100 percent. That's not to say that such a survey has no value, but the rigorous statistical methods used by pollsters simply cannot be applied to small populations.