Keynote Systems KeyReadiness
KeyReadiness delivers the most thorough analysis. It not only defines where and when a site has performance problems, but also identifies the site's current capacity and the boundaries that identify the gray area between satisfactory and poor performance. For example, only its report gave us insight into projected user satisfaction as load increased. KeyReadiness' "user experience" metric takes user satisfaction, dissatisfaction and site abandonment into account, based partially on Zona Research's 1999 finding that a Web server has about eight seconds to deliver the goods before losing the user.
In CountryWatch's case, the site could scale up to 12 times its current capacity with only moderate changes in the user experience and to nine times current load with no impact. The CountryWatch site handles about 700 sessions per hour, and Keynote's testing showed that the site didn't begin to falter till the load hit 7,500 sessions per hour. At that point, mean-page-response times doubled and the server began to show errors, though projected user satisfaction and abandonment rates stayed in a satisfactory range. At 9,000 sessions per hour, response time increased fivefold, and user dissatisfaction and abandonment rates hit a dismal 35 percent.
"KeyNote KeyReadiness Session Completion" (see graphic, below) shows completed and abandoned sessions by load and tells the whole story. With this graph, we could ascertain when users would leave the CountryWatch site, given its current configuration. A second graph showed us that even in peak-load situations, the shopping-cart transaction continued to be processed -- good news for CountryWatch.
By examining the server logs, Keynote determined that the server's ASP and Microsoft SQL back end were bottlenecks. However, the vendor didn't go as far as Mercury, which made cause-and-effect determinations about ASP and SQL functions on the site. Keynote recommended making the changes, and then retesting to validate. Keynote also performed a separate analysis on these two back-end services. That analysis, which would have cost CountryWatch an extra $8,000, yielded both granular technical results and next steps that were consistent with Mercury's. However, our Keynote contact said customers such as CountryWatch would get the most bang for the buck from the KeyReadiness service alone, without paying for the additional analysis.
Network recommendations, such as those Keynote made about the CountryWatch server, were not as definitive as Mercury's, but they did correctly point out that in CountryWatch's case the network and its infrastructure were not problems. Here, Keynote monitored the overlay of time to first byte and mean-page response to determine if the server or the network were becoming the bottleneck. If the first byte slows but the overall page download remains the same -- as was the case in our test -- the network is not likely to be the problem. But Keynote's analysis pointed out that ASP-delivered content is often at fault.
Keynote was the only vendor to make a revenue projection (see "Revenue Projections vs. Load Level," above) and loss recommendation, based on the frequency of scripted product-purchase transactions and the resulting failure rate of those purchases as attempted session loads increased to the point of failure.
Keynote was up-front about the assumptions it made for our test, but this is typical of the kind of information KeyReadiness delivers.
Get Real!
Preparing for a test is most of the work, and Keynote leaves nothing to chance. The company spent more time and had more parameters to make the scripting of simulated users as real as possible. That preparation paid off with results we trust.
Initially, Keynote analyzed one month's worth of CountryWatch's Web logs, using the frequently used software from WebTrends for Web site log analysis in conjunction with a proprietary analysis tool. Keynote also spent in-depth time interviewing CountryWatch regarding its site's performance, both historically and projected.
The analysis gave Keynote an idea of the normal load on CountryWatch's site. The scripts were designed to roam the site, controlling the order and frequency with which simulated sessions would be controlled. Interestingly, only Keynote identified and backed out the traffic created by the monitoring of CountryWatch's site undertaken by Ipswitch's What'sUpGold and Web site monitoring services from Gomez.
The scripts create the started sessions that define the key difference between Keynote's testing approach and those of the other vendors. Keynote builds a session-distribution signature, which is carefully constructed from averages, standard deviations and factors such as what impact marketing promotion is likely to have on the site.
Keynote's concept of user abandonment and its flip side, user satisfaction, are measurements of how well or how poorly a site is doing under load. When a site slows down, users leave. Keynote's scripts emulate user abandonment of a slowing Web site. If you're thinking that's one less user, and therefore reduced server load, you're right. The point is that it changes the performance of the Web site.
All the vendors have ways to account for users' thinking time while reading a Web page, but Keynote's method is the most granular. Interaction speed, or the speed at which a user reads a page, is the metric that accounts for an experienced user being faster than a new user. Latency tolerance -- a user's willingness to put up with slow page delivery -- is also factored into the scripts. Finally, user tenacity considers how much of a wait a user will put up with, given that it varies by transaction type. For instance, a user will likely wait out a slow stock trade, but a slow-loading home page would be clicked into history. All these factors are weighted differently given the load being generated on the server. Now that's granularity.
All in all, Keynote has a story worth listening to. Money spent on one of its tests will yield useful results.
KeyReadiness, starts at $9,900, Keynote Systems, (650) 522-1000, (888) 539-7978. www.keynote.com
|
 |
REPORTS
ANALYTICS
Follow 
