Defining Our Terms

ROI is a simple concept: It's what you get in return for the money you spend. IDC says ROI is the net return on the investment (a total of cost reductions and additional revenues) divided by the amount of the investment. The result is usually expressed in a percentage ("payback for the companies surveyed averaged 614 percent," for example), though you have to look at a several additional factors to understand where that percentage comes from.

ROI tends to be a factor of time--the return over a certain number of years. One of the statistics you'll see in ROI reports is time to payback, which is the length of time taken for the combined savings and revenues resulting from the purchase to equal the purchase price. This is generally expressed in a number of days and is within the scope of the total time of investment, a figure often expressed in years.

IRR (internal rate of return) is ROI expressed as a compound annual percentage rate. Some companies describe ROI in terms of IRR, while others provide the ROI across the entire length of time considered in the study--generally in the three-to-five-year range. For the same return, IRR will be a lower number since it only looks at an annual rate instead of the rate covering multiple years. When comparing figures across products, check to see whether the analyst is expressing the results in ROI or in IRR. The Microsoft study on Bridge Information Systems, for example, quotes a five-month payback with an IRR of 277 percent. Enterasys' study of network solutions shows an average payback of 109.1 days with a 704 percent ROI. If you were looking at both studies, you would have to keep in mind the difference in time periods expressed by IRR and ROI when comparing their relative effects on finances.

The Long And Short of It

If an analyst is looking at ROI over a multiyear period, long-term issues like inflation become necessary considerations. In addition, some analysts will look at the opportunity cost of investing the money in networking hardware or software rather than putting it in the stock market or some other investment. These are worthy attempts to take ROI out of a "best of all possible worlds" setting and make it more realistic. The problem is that the analyst has to make some assumptions about what the world will look like over the time of the investment.

In the United States today, inflation is not the concern it was in the 1970s, but there's no guarantee it will remain in the 2 percent range forever. As for opportunity costs, everyone who tracked his or her 401(k) portfolios from the late 1990s into the new century understands that investment returns will vary year to year. Make sure you understand the analyst's assumptions when you look at NPV (net present value)--the ROI over the total study period expressed in today's dollars--and that you're clear on how NPV was applied to a particular ROI.

REJ (rapid economic justification) is designed to look at the return on investment across a shorter span than that of a traditional ROI study. Microsoft developed and is the main proponent of REJ, devoting considerable Web space to explanations of the methodology and why it's better than traditional ROI.

REJ differs from traditional ROI in two ways. First, the "rapid" component means that the analyst doesn't consider heavily the time value of money or relating a dollar three years from now to a dollar today. Second, the REJ method attempts to quantify some issues that are difficult to account for in a traditional ROI study, such as reduced time for developing marketing collateral and enabling better collaboration and communications.

Microsoft argues that for office productivity tools, REJ may offer a better time horizon than ROI. For categories like network infrastructure, on the other hand, the product's lifetime will likely far exceed the window provided by REJ. (Check out "Business Value of Microsoft Solutions" to figure out whether REJ could be useful to your company's situation.) However, we haven't noticed anyone other than Microsoft using REJ.

Do the Numbers Add Up?

Let's use the IDC study of network solutions sponsored by Enterasys networks to illustrate how these key concepts relate. In this study, IDC interviewed a number of customers, so the financial results are averages. IDC projected the numbers over three years, a reasonable life span for a specific network-infrastructure component.

Over the three-year period, the average amount invested in the technology was $8.3 million (see "Running the Numbers"). The average annual cost savings and revenue increase was $27.9 million. These two numbers can be used to generate one of the first conclusions of the study. Divide the benefit--$27.9 million--by the investment--$8.3 million--and you get 3.35. Then, to calculate the time to payback, divide 365 by 3.35, and (with some rounding up) you get 109.1 days--IDC's payback-period figure.

Running the Numbers Click here to enlarge

If you multiply the $27.9 million annual benefit by three years you get $83.7 million. You won't see that number in the ROI study, though, because it fails to take into account the opportunity cost of the investment. IDC calculates that cost based on a 12 percent return for the mythical "other opportunity," and shows an NPV of the three-year results as $66.9 million. Now, subtract the initial $8.3 million investment from the $66.9 million NPV, and you get the number that allows IDC to show a three-year ROI of 704 percent.

The figure of 704 percent for this technology is defensible mathematically, but it seems pretty fantastic when you look at it from the perspective of the real world, and here's why: The numbers based on the payback period--a relatively short period of time--are extended without great modification through the length of the study. Whenever any effect of technology, whether positive or negative, is extended through a period longer than the typical revision cycle for the technology, the conclusions become wobbly. In this case, the study's period covers at least a couple of revision cycles, making the conclusions uncertain.

How do the other studies stack up in terms of real-world information?

The Computer Associatesısponsored study performed by IDC on infrastructure management uses similar numerical methods and provides details similar to the Enterasys study. Of the remaining three studies, the REJ studies published by Microsoft include some figures on IRR and other measurements, but the details required to duplicate the studies' results just aren't there. The Delphi Group study for CA contains the most narrative and the fewest numbers--its conclusions aren't couched primarily in terms of percentages, and there is no way to derive numbers from the information provided.

Comparisons Count

In addition to comparing vendors' ROI studies of particular technologies or products with one another, compare them to your own ROI study. If the study is based on a single company, as is the Bridge Information Systems study from Microsoft, comparisons can be difficult if your company is in a different line of business.

Studies based on samples and averages can make for easier comparisons, though you have to be careful because a single huge company or government organization can significantly skew the results of a study. The IDC study we evaluated included details about the companies that participated, including any that might skew results. In the final analysis, you must rely on your judgment--there are no hard-and-fast rules on taking the results from a study and applying them to your situation.

Some vendors have recognized this difficulty and provided ROI calculators to help nonaccountants generate key numbers. For example, Computer Associates posts a Microsoft Excel spreadsheet that generates numbers regarding single-sign-on implementations. While the spreadsheet is easy to use and will generate a number that purports to show how much money a customer can save, it does not create a figure that could be used to compare directly with the numbers in a study such as the Enterasys-sponsored IDC report.

Microsoft has an online ROI calculator for Windows XP, Office XP and a combination of the two products. This calculator is more complete than the Computer Associates spreadsheet and deals with some of the "softer" issues, like REJ, but its numbers still don't compare directly with those of a traditional ROI study.

Who You Gonna Trust?

As we've seen, there are many ways to express standard measures of return. If these standards don't tell the story the vendor wants, you're likely to see great emphasis on soft factors like employee satisfaction and happiness, with no attempt to put a dollar figure on the smiley faces. In a similar vein, if a vendor presents an entirely new way of measuring economic benefit, you should ask yourself how bad the product or technology performed under the standard ROI measures.

An ROI report that's heavy with numbers and filled with explanations can still be a bad study, but at least you can figure out what's bad about--perhaps the basic assumptions differ wildly from your circumstances or the calculations produce results that don't add up. Most important, though, having the numbers in front of you will let you answer the questions that corporate executives are bound to ask about the study. Those queries usually deal with the validity of the assumptions and calculations, along with concerns about comparisons between the participating companies and your company.

Even if a report is laden with numbers, do two things before taking the report with you into a meeting. First, go to the back of the report, where you'll find hidden most of the details about how numbers were generated. Many reports give great details there on the companies involved in the study, the assumptions made by the analysts and the formulas used to generate results. Next, check the math in the report. I've found many simple arithmetic errors in research reports, and it's far better to correct them at your desk than under the watchful eye of the CIO.