Keynote Systems' Streaming Perspective 3.0: A Reality Check for Streaming Media

Keynote provides high-level reports via a real-time Web interface, and can have reports and alerts e-mailed to you on a regular basis.

The Setup

I tested SP Monitoring service in our Network Computing Real-World Labs® at Syracuse University. Keynote included a URL to an audio stream from a public radio station so I could get a feel for the depth of information available. In addition, I previewed Keynote's SP Diagnostic service by creating stream checks for RealPlayer media from our own NWC Radio program.

I established a contract with Keynote to set up my account and begin to monitor the URLs I submitted. The URLs can support any stream supported by RealPlayer or Windows Media Player, such as RM or ASF files. This includes live and on-demand clips and slide shows with bit rates up to 300 Kbps. Higher bit rates are available by special order. You can monitor four types of metafiles (ASX, WAX, SMI and RAM) and CGI-processed HTTP links that result in streaming media.

SP Phone Home

SP Diagnostic service lets you check the availability of the URLs you assign to a channel for monitoring. I selected a number of streams from the NWC Radio site to diagnose. But because I had only one channel, I had to reassign it to different URLs and check them in one at a time. I wanted to focus on the monitoring service and measure the quality of an end user's streaming media experience, so I set the SP's distributed scanners--software modules that simulate both RealPlayer and Windows Media Player--to check 10 times a day and to alert me by e-mail if the streams were unavailable.

During 17 days of monitoring, SP's summary data for the public radio station showed that it had more than 90 percent availability and that it was delivered with an average bit rate of 15.3 Kbps. It also showed the average times to connect (1.2 seconds), buffer (2.6 seconds) and rebuffer (1.1 seconds).

I was able to reproduce most of these numbers from my home office using RealPlayer over a cable modem. Home scanners, connected to the Internet using broadband cable/DSL modems, provide a picture of streaming media presented to the end user in the last mile. Other Internet access through fixed or 3G wireless and satellite is marginal.

Measuring Quality

One metric you can use to determine stream quality is SP's StreamQ, a measure of your streaming media that's viewable from the Web site and delivered to you in a regular report. StreamQ takes the various measurements the scanners make during their 60-second media checks and assigns them grades ranging from A+ to F. Grades are based on the time spent waiting for a connection, including time for DNS resolution, metafile action and server/player handshakes; the initial buffering time for the player to receive the requisite bytes for initial playback; and the rebuffering time when bit rates fall below the encoded playback rate. The scanners also consider the stream's total playing time.In my tests, StreamQ gave the public radio station stream an A during peak work hours and an A+ at other times. I drilled down by date and time to pinpoint the location of the scanner, the original URL of the file or metafile requested, the server type, and the number of packets received and lost during playback. Information was available on the actual bit rate of delivery and buffering and rebuffering times. I could find the DNS resolution time and the results of a trace route to the network path used by the player and the server, including round-trip delay time in milliseconds. I could also review the error code the player received.

In most cases when a scanner couldn't connect with the server, the scanner reported RealPlayer Error code 800400C4 ("Server has reached its capacity"). When the StreamQ rating dropped from A+ to A, the trace-route data showed problematic network delays.

Sean Doherty is a technology editor based at our Syracuse University Real-World Labs®. Write to him at [email protected].

Post a comment or question on this story.

Many variables affect the video quality as perceived by end users on the desktop. Most of these depend on the encoding and decoding process and the frame rate delivered to the user. The frame rate differs based on the codec used and the decoding process, and that will vary depending on whether the process is done in hardware or software and what video player is implemented on the desktop. All these variables can lead away from an objective metric of quality to a subjective determination of quality; they can even be misleading. For example, RealPlayer's decoder provides information on the frame rate delivered to the desktop. However, it polls the frame rate after it interpolates frames. This can result in a frame rate higher than the encoded rate. And when you start comparing frame rates, you quickly digress to a subjective preference for a codec that delivers fewer frames with high resolution and color depth or more frames with less resolution and color. It is important to note that Keynote's Streaming Perspective measures the frame rate of some streams and includes the metric in the stream details, but this is not included in its StreamQ metric.