Unfortunately for the vendors, with so few publications doing testing,
it's sometimes impossible to coordinate product release schedules with
review opportunities. When that happens and a vendor is looking to
create a little buzz, it's not uncommon for them to turn to a
professional test lab for "independent validation." While there are many
such labs active today, the Tolly Group is widely regarded as the market
leader.
Vendors pay Tolly to perform tests that they know in advance will cast
them in a positive light. Tolly employs experienced testers, and the
company follows a set of published ethical guidelines. The group openly
discloses its testing methodology for all to see. That should provide
readers of Tolly reports with assurance that its evaluators aren't
cooking the numbers. Experienced IT professionals know what they are
getting with a Tolly report: Accurate information presented in a manner
that is favorable toward the vendor that paid for the test.
However, while Tolly's validation reports provide a service by
empirically validating vendor claims, the company crosses the line when
it engages in competitive testing. That's because competitive testing is
all about the test plan itself. If Vendor A pays for the test plan, how
do YOU think the test is going to turn out for Vendor B?
Tolly's most recent effort in this area involved a competitive test of
wireless VoIP security commissioned by Aruba Networks and designed to
highlight its product features against market-leader Airespace. Tolly
independently acquired Airespace equipment and, as part of their "fair
testing" process, advised Airespace that the wireless company could
either actively support the test or not. Airespace executives took one
look at the test plan, which was obviously designed to highlight Aruba's
integrated firewall capabilities, and said "No way!" In fact, Airespace
pointed to restrictions in its license agreement and threatened
litigation if Tolly tested without Airespace's approval. Tolly backed
down, but published the report anyway--minus competitive
testing--commenting on Airespace's deficiencies throughout the report.
The report itself provided little insight into the fundamental issues of
whether WLAN switches should include integrated firewall capabilities or
whether encryption should be handled in the AP (access point) or the
switch. It was simply an Aruba promotional white paper, the kind of
thing you might expect from a vendor but not an independent test lab.
