Networking

04:29 PM
Lee Badman
Lee Badman
Commentary
Connect Directly
Twitter
LinkedIn
RSS
E-Mail
50%
50%
Repost This

WLAN Stress Tests: What’s the Use?

Wireless hardware testing can shed much-needed light on Wi-Fi products when every vendor claims to have the best stuff. But you have to pay attention to the testing details.

Independent testing of vendor products is a rare and valuable thing. That's certainly true in the wireless networking market, as WLAN makers are pushing out ever more access points and associated hardware, and wireless architectures are becoming staggeringly complex. How do potential customers know what product is right for them, or if their current vendor is delivering the goods as advertised?

Independent test results can help--as long as they are kept in proper context. One of the more interesting independent tests comes from Keith Parsons, of Wireless LAN Professionals.

Parsons' "Wi-Fi Stress Test" is touted as a repeatable, vendor-independent access point analysis. The goal of the test was simple: pit an increasing number of Apple iPads against a single AP until the AP crumbled, and measure the same data points along the way for each unit under test.

APs from Aerohive, Aruba, Cisco, HP, Juniper, Linksys, Meraki, Ruckus, Ubiquity and Xirrus were provided by each manufacturer, and the rest of the test environment was composed of client devices and test gear owned by Wireless LAN Professionals.

Parsons' testing team included representatives from seven WLAN vendors and two dozen volunteers unaffiliated with the vendors who were eager to learn from some of the best in the industry. Two weeks ago, I spent time with Parsons and fellow industry experts and analysts at Wireless Field Day, and the Stress Test got a lot of attention.

Rather than regurgitate the results, I want to share my impressions of what I liked about Parsons' approach, and what I'm not so keen on.

The Ups and the Downs

If you read Parsons' report, you'll see he goes to great pains to explain the parameters of his test, as well as its limitations. In other words, he's not making this test out to be anything more than what it is.

His goal was to see what APs could stand up to the max load, measured in iPads pushing known traffic. That's it. He wasn't crowning anyone with the title of The Best Wireless System. He was clear with his methods, he had a great team of testers, and he kept an eagle-eye on the vendors that participated. No configuration or performance-optimizing was tolerated where vendors had an active hand in testing. At the end of it, Ruckus did well in this exercise, with Cisco close behind.

[ Join us at Interop Las Vegas for access to 125+ IT sessions and 300+ exhibiting companies. Register today! ]

What didn't I care for in this analysis? Though Parsons made it crystal clear that he doesn't consider this a "real-world" test of WLAN products, I was puzzled that all testing used 20 MHz wide channels in the 5 GHz band of 802.11n. I don't know of a single network that is configured for other than 40 MHz channels, as that was one of the prime drivers for migrating to 11n from legacy 11a/g technology. I also feel conflicted about a iPad-only testbed, as in my own network experiences I find Apple products to be maddeningly inconsistent depending on their own OS version combined with the specific code found on the infrastructure APs and controllers.

Parsons also points out that his analysis excluded many other critical functions of modern business WLAN: feature sets, management interfaces, location services and all of the other pieces that add up to TCO. Put another way, even though this stress test was not billed as real world (and yes, everyone's version of that notion varies), it was a bit too far away from real world for my liking. I learned what APs best stand up to huge volumes of traffic as delivered in the testing under specific parameters, but that is arguably a relatively minor data point in the bigger story of WLAN ownership.

At the same time, I truly appreciate Parsons' efforts to undertake this task and to gather the resulting data. I look forward to how Parsons will build on it in future tests, and wouldn't mind participating in the next round.

Comment  | 
Print  | 
More Insights
Comments
Newest First  |  Oldest First  |  Threaded View
LeeBadman
50%
50%
LeeBadman,
User Rank: Apprentice
3/1/2013 | 2:55:11 PM
re: WLAN Stress Tests: What’s the Use?
Dave,

I'm right there with the pack in appreciating what Kieth does for the WLAN space, and was thrilled to be able to meet him at Wireless Field Day. Thanks for the comments.

-Lee
HalifaxDave
50%
50%
HalifaxDave,
User Rank: Apprentice
2/28/2013 | 11:02:38 PM
re: WLAN Stress Tests: What’s the Use?
As always, I like your articles and insights! what I appreciate the most about the stress test, is the genuine attempt at vendor neutrality. Many of us hold Keith in very high regard for his expertise in WiFi. I am confident that this is only a taste of what is to come from Keith and others. He has basically laid out an approach and tool-set that is achievable by many of us (that are willing to put in the time).
I agree with your pros and cons, and would like to see similar testing with multiple APs.

Dave
LeeBadman
50%
50%
LeeBadman,
User Rank: Apprentice
2/27/2013 | 2:07:39 PM
re: WLAN Stress Tests: What’s the Use?
Hi Kieth,

Thanks for the information and explanation, and I am envious of the fun that must have been had during this testing given the crew involved. Where schools are asking for this sort of test, wouldn't adjacent classrooms also have APs, and is the premise that some mechanism would be invoked to keep one room's worth of devices on a specific AP?

I would imagine you've also drummed up some business for the tool makers as a result of this, and rightly so- I'm glad you showed how you used them in detail.

-Lee
kparsons840
50%
50%
kparsons840,
User Rank: Apprentice
2/27/2013 | 7:08:35 AM
re: WLAN Stress Tests: What’s the Use?
Lee,

Thanks for the comments, both pro and con on this Wi-Fi Stress Test.

The reason for the 20MHz channels in 5GHz was two-fold. First, our iPads were .v4's - and *could* support 40MHz channels... but we wanted to emulate the majority of the hand-held devices, both smartphones and tablets - the large majority of which *only* support 20MHz channels.

The second reason was to force the failures to be within the resources of our test bed. If we'd had 60+ devices - then 40MHz channels might have been used. Since we only had 30 iPads, we needed the 'breakage' of Wi-Fi to occur sooner, rather than later. Having all Access Points work with more than our test bed resources wouldn't have had very educational of results.

Actually - we have had multiple K-12 IT departments ask specifically for this test - because it reflects what their needs are in their 'Real-World'. What was not 'real-world' was the size of the 1080p HD Video stream, and the HUGE FTP transfer going on at the same time. Again, this was to induce failure - to show 802.11 has a very limited resource in time and frequency. These must be allocated for, and designed around, in order to maximize client traffic on your Wireless LAN.

We look forward to doing more testing, perhaps multiple AP testing, in the future.

Regards,

Keith Parsons
Managing Director
Wireless LAN Professionals, Inc.
keith at WLANPros.com
More Blogs from Commentary
Infrastructure Challenge: Build Your Community
Network Computing provides the platform; help us make it your community.
Edge Devices Are The Brains Of The Network
In any type of network, the edge is where all the action takes place. Think of the edge as the brains of the network, while the core is just the dumb muscle.
Fight Software Piracy With SaaS
SaaS makes application deployment easy and effective. It could eliminate software piracy once and for all.
SDN: Waiting For The Trickle-Down Effect
Like server virtualization and 10 Gigabit Ethernet, SDN will eventually become a technology that small and midsized enterprises can use. But it's going to require some new packaging.
IT Certification Exam Success In 4 Steps
There are no shortcuts to obtaining passing scores, but focusing on key fundamentals of proper study and preparation will help you master the art of certification.
Hot Topics
6
IT Certification Exam Success In 4 Steps
Amy Arnold, CCNP/DP/Voice,  4/22/2014
6
Edge Devices Are The Brains Of The Network
Orhan Ergun, Network Architect,  4/23/2014
White Papers
Register for Network Computing Newsletters
Cartoon
Current Issue
2014 Private Cloud Survey
2014 Private Cloud Survey
Respondents are on a roll: 53% brought their private clouds from concept to production in less than one year, and 60% ­extend their clouds across multiple datacenters. But expertise is scarce, with 51% saying acquiring skilled employees is a roadblock.
Video
Slideshows
Twitter Feed