Market Analysis: Real-Time User Monitoring

Our reader survey for this article showed a keen interest in this granular look at the user experience. We asked which groups rely most on performance information and, not surprisingly, IT management topped the list. But business managers were in a virtual tie for second--with IT admins--in paying attention to IT performance reports. Are they watching bits and bytes? Not likely. Are they looking for a clue about customer and user satisfaction?Of course.

Management isn't focusing on IT infrastructure performance solely because of the SLA (service-level agreement) fines levied for performance failures. Although some enterprises do have availability and throughput goals against which their IT shops are measured, most don't. Readers overwhelmingly said a tool that gathers and reports user performance data in real time would be valuable.

RTUM products collect data on end-to-end response time, errors and throughput of user-app sessions, including those for Web and business apps. This data is stored as specific app sessions, then aggregated and linked by server, location, page and object so specific sessions and global performance trends are visible.

Choosing among the myriad methods and technologies available to achieve the RTUM goal can be difficult, however. Agents can gather a mass of data, but if deploying agents is problematic because the nodes to be monitored aren't under your control, the alternative is agentless technology. Services like those from BMC Software, Gomez, Keynote Systems and Mercury Interactive, which fire robotic transactions, also measure the end-user experience. We'll evaluate three products in "Real Users, Real Insight" but first let's examine the various technologies.

It's helpful to map out the pros and cons of the processes available for performance management in general, and for RTUM specifically.

Performance management stretches across network, system and application. Network approaches have been around for years, even before the SNMP RMON (Remote Monitoring) standard. RMON uses computers or probes connected to the network in promiscuous mode to count packets as they pass. This approach is the basis for the RTUM products we tested.

But even though the RMON standard went through extensive interoperability trials, and management vendors developed useful applications in the early 1990s, RMON never took off, in part because of the cost of the probes, and in part because LAN bandwidth was cheaper to buy than to manage. In addition, though RMON and proprietary approaches like those from Net QoS Inc. and Network Physics have advanced what network probes can gather, their application knowledge is limited to average/min/max-type usage statistics. They cannot dive into the specifics of a user's experience.

System agents have been cranking just as long as RMON-like probes, and provide OSs, application servers, Web servers and enterprise applications with performance-management information that these products don't natively produce. There are agents such as those from Hewlett-Packard, for example, that can monitor the end-user experience. They insert headers into the transaction, from an agent on a remote client machine or a server in the machine room, and track the performance of transactions across n-Tier application systems.

Getting an agent on a system--especially a system you don't own--isn't easy. But the political difficulties of inserting tags in a transaction make the organizational hassles of systems agents seem like molehills. We can hear the conversation now: "So, Mr. User, this code will watch everything you do on your PC so when you call with problems ... hello? Mr. User. are you there?"Fortunately, some performance-management products, including those from BMC Software, CA, Empirix and ProactiveNet, are agentless. They use telnet, SSH (Secure Shell) or WMI (Windows Management Instrumentation) to monitor performance from a remote computer--not a new idea, but a good one, especially where ownership is an issue. But remote monitoring doesn't gather a lot of data because it relies on whatever remote interface is integrated by the OS, application server or application vendor. Good enough for availability, throughput and basic load. Not good enough for RTUM. It's not that built-in performance monitoring of operating and application platforms isn't useful. But the specifics of how an application is monitored and how users interact with the application are beyond its ken.

Analytics services, grown from Web log processing, view users' interactions with Web applications. Examples are BEA Systems' WebLogic, Omniture's SiteCatalyst and WebSideStory's HBX Analytics. They insert Java scripts on Web pages following user clicks, sessions, aborts and, most important, responses to promotions, advertising and merchandising on Web sites.

These products report some end-user performance information, including average/min/max number of users, throughput on pages, object download times, page popularity and the errors users receive. But performance-wise, they don't average/min/max the end user experience across network, server and application tiers, and they don't monitor specific user transactions.

All performance approaches--agent, agentless and service-based--claim to monitor apps. The approaches they use to do this tend to fall across three categories: port, standards-based protocol and proprietary application protocol. Many inexpensive network-monitoring applications are limited to TCP and UDP (User Datagram Protocol) port identification. Not a bad approach, but it's sort of like monitoring the door of an Italian restaurant--you're nearly certain that any given patron has eaten Italian food, but you're not sure what he or she ate, or how good it was.A more comprehensive approach relies on standards-based protocols, such as HTTP, HTTPS and SOAP (Simple Object Access Protocol). Performance vendors use specific knowledge about the delivery of applications over these protocols to understand the user experience. This is like being inside the restaurant, knowing the menu and watching user reactions. If you're around long enough you can find out the average/min/max time it takes to get the eggplant parmigiana. If you write it down, patrons' order can be tracked and stored.

The proprietary app-protocol approach is similar, but vendor-specific. Performance vendors that know Oracle, PeopleSoft and Siebel transactions, for instance, can track the specifics of each transaction. This is like knowing in advance what will be on the menu. RTUM performance products do all this too, and also monitor availability and intake. It's about tracking what users do on Web sites--how and when they come and go, and what the performance metrics tell us about their experience.

To switch metaphors, analytic products are the psychiatrists--they try to figure out why a user behaves in a particular way on a Web site. Not that RTUM products don't get any of that information--they can, for example, track how long a user stays (a long stay indicates that the user is getting the content he or she wants); track aborts and stops (stopping a page load is usually a performance, not content, problem) and track access differences (dial-up users experience a Web site differently from broadband users).

Finally, some RTUM products will show and track transactions. Operationally, pages are the atom--the smallest sub-particle of a user's experience. Pages are not what we have Web sites for, of course, and are not the reason users go to Web sites. Transactions--whether buying, learning or getting a service--are the motivators. But pages are the elements with which we interact, so they're the base RTUM unit.

Transactions have value for mapping behavior. You'll need transaction data if you're mapping load using a service like Gomez's GPN or Keynote Systems' Web Site Perspective, where transactions are an important part of performance feedback. Finally, the response of users to promotional and merchandizing efforts lies outside RTUM and in the analytic camp.

The main cost of RTUM is in the hours needed to analyze data and define what should be watched, not in purchase or implementation. The products we tested cost $45,000 to $125,000, not hard to swallow when compared with the six to seven figures often quoted for other performance products. The savings come from not having to implement agents and/or coordinate access to servers. However, if you don't accurately configure your RTUM product, you'll waste time and money.

Note that it's unusual for management products, even performance management products, not to have some training or professional services included. But none of the vendors in our test provided these niceties. They do spend some on-site time to get us started, but the products are meant to be easy to implement and use, and that is a long-term cost savings.

RTUM should take off this year. Customers are getting more sophisticated and expect top-notch performance; if you don't give it to them, your competitors will. End users also expect their critical apps to work, and drains on productivity cost real dollars. IT is not doing enough--74 percent of problem alerts come from end users complaining to the service desk about performance, not from IT monitoring infrastructure components, according to Forrester Research.

The RTUM market landscape is also getting interesting, with Mercury Interactive's September acquisition of BeatBox Technologies for $14 million in cold cash and Compuware's May acquisition of Adlex.RTUM is the next best thing to being in the customer's living room, monitoring your site's performance and eating some pasta. So get real ... real time, that is.

IT can monitor users and their experiences in many ways. All have pros and cons:

Why do some people happily pay $6 for a screwdriver at their local hardware store when they could get the same tool at Home Depot for $3.99? Service, that's why. Customers want a pleasant shopping experience, with individual attention, whether in person or online. And end users want to do their jobs without being bogged down by the IT infrastructure. But that requires monitoring, preferably of real users in real time. Agents are problematic, and getting only simple averages smooths over outlying performance aberrations. In "Nerd's Eye View" we discuss the benefits of RTUM (real-time user monitoring) and help you decide if it's right for your organization.

For "Real Users, Real Insight," we tested Compuware's Vantage, Coradiant's TrueSight and Quest Software's User Experience Monitor (UEM) RTUM offerings in our Syracuse University Real-World Labs®. For Vantage and UEM, we created transactions (prescribed sequences of browsing, clicks, object downloads and data input)--steps a user would go through to check a bank account balance and buy a bonsai tree. For TrueSight, we set "Watchpoints" for each page to see the specific experience of every user hitting these pages.

We sent controlled Web transactions over a background load, simulated users having problems accessing specific applications, and checked how each product captured and reported these issues. When we found unusual performance, we fearlessly dived into the event to see what was causing our bonsai buyers to abandon their trees, for example.

We were happy with the insight provided by all three products, but TrueSight took our Editor's Choice award. It's easy to use, and its complete URL parsing let us track down a variety of problems. Quest UEM won our Best Value award, thanks to its low price and good scalability.