On Location: American Airlines Center's Managed Wireless

Just as a professional athlete's skill is based on a solid grounding in the fundamentals of the sport, exceptional wireless networks are built on smartly engineered wired infrastructures. The AA Center is no exception. Its groundwork comprises a hub-and-spoke switched network with a Nortel Networks Passport 8600 Layer 3 switch at the core, supporting 1,200 wired ports and about 27 closet switches, with three firewalls and two Internet connections to provide fault tolerance for a critical InfoGenesis POS (point-of-sale) application hosted by a California-based application service provider.

The AA Center's other applications are hosted locally on Wintel servers connected over Fibre Channel HBAs (host bus adapters) to two Dell-EMC CX500 SAN units, one in the arena's data center, another in a spare data center across the street. There are enough multimode fiber runs in place to support both the SAN and the Gigabit Ethernet backbone. The switched Ethernet network is carved into multiple VLANs (virtual LANs)/IP subnets using standard 802.1Q tagging based on functional domains; major divisions separate corporate and "game-day" applications. Game-day functions are tightly controlled to ensure uptime. The IT staff set up wired VLANs for each team the center hosts, the building operating team, the POS system, video editing and the PCs located in spectator suites, plus one for telecommunications and one for TicketMaster. The media can access the suite VLAN, which is restricted to Internet use.

When we asked why the IT team is so bullish on wired VLANs, network engineer Louis Yuan told us that when the W32.Welchia worm hit the corporate network, one VLAN segment went down. But during that night's game, all applications worked flawlessly, thanks to the VLAN separation. NetIQ's MailMarshall and WebMarshall have since been installed to help prevent a reoccurrence of even one VLAN outage.

The InfoGenesis POS system provides supply-chain and real-time inventory features that have become business-critical to the center (for more on the business justification and the AA Center's end-user applications, see "Nothing but Air," ). To guard against a leased-line outage that could cripple this system, Yuan deployed a primary circuit to Southwestern Bell (part of SBC Communications), a fractional DS-3 running at 10 Mbps. This circuit is backed up by a T1 microwave radio connection to XO Communications (see network diagram, below).

From a routing perspective, fault tolerance is achieved by the creation of multiple default routes in the core Nortel switch with different priorities. Because the goal is to have outbound ASP traffic up during game time, and because NAT (network address translation) is in use, this type of fault tolerance works nicely and is much simpler to implement than exterior routing protocols, such as BGP (Border Gateway Protocol), Yuan says. This makes sense. BGP involves overhead--both in training and troubleshooting--that the center has avoided with its fault-tolerance scheme. Although SBC has redundant fibers to the building, both leading to separate COs, the XO circuit bypasses SBC entirely, making it the provider of choice in case of SBC downtime. As is appropriate with any fault-tolerance setup, Yuan has done a "fire drill" to make sure that the routes cut over correctly and that services keep on trucking.

The AA Center's existing Nortel gear doesn't have Power over Ethernet capability, so Joe Heinlein, the center's IT director, purchased PowerDesign 6001 units to provide juice to AC-bereft AP (access point) locations, such as over the stadium's catwalks. Aruba Network's professional services division handled the site survey and initial configuration, then spent about four days training the center's staff. Power levels were tweaked, and APs were moved to what the group felt were optimal locations. The deployment consists of 35 APs throughout the arena, 33 of them active. Aruba's Web management platform has been invaluable in showing coverage areas and in visualizing power levels.

Game day finds Yuan, Heinlein and desktop technician Derric Paige in the operations center keeping an eye on a large screen displaying their monitoring software. NetIQ's AppManager is used to track critical services on Windows servers. IBM's Tivoli NetView lets the staff monitor network reachability. Heinlein, Yuan and the rest of the crew keep an eye on the monitoring systems so they can react before they get user calls. Wireless technology isn't new to the AA Center, but the staff says the current wireless system's ease of management and secure nature are a pleasant change. Both Cisco Systems and Aruba wireless LANs are in use; the standalone Cisco infrastructure consists of AirPort 350s, installed and managed by TicketMaster. In some portions of the arena, the Aruba system detects the Cisco gear as interference, but so far, Heinlein and crew say that while they haven't formally tested for a performance variance, they haven't experienced any trouble with the network or applications. The staff has no plans to switch the Cisco wireless network over to the Aruba system, not only because it's not broken, but because TicketMaster runs and manages this function. The TicketMaster VLAN on the Nortel core is essentially isolated, used only for TicketMaster functions; the TicketMaster WLAN uses this transport to get back to the TicketMaster router.

Ticket takers are equipped with Symbol 4810 PDAs running PalmOS that connect to the TicketMaster wireless network and let them scan customers' tickets in real time. Legit tickets, whether purchased online and printed at home or bought through conventional outlets, are greeted with a cheerful beep. Duplicate tickets cause the PDA to emit a buzz that spells game over for forgers.

Since installing Aruba wireless technology, the AA Center's technical staff has been busy adding wireless applications. The choice of Aruba was driven more by a business-alliance decision than by technical requirements (Aruba donates professional services in exchange for ad space), but the IT staff says Aruba's technical chops are ready for prime time. In fact, a few of the WLAN's bells and whistles are still adding depth to the bench. One example: Aruba supports 802.11i, but the practical considerations of using older multivendor client hardware has led the center to choose VPN authentication to the wireless network as the methodology for securing traffic. The Aruba switch acts as the VPN termination point, and so far, it has worked flawlessly with a variety of gear.

The ordering system, for example, lets wait staff swipe credit cards seat-side--and thus calls for a high level of security; this app runs on Cassiopeia E700 handhelds. The E700s don't support 802.11i, but work fine with Certicom's Movian VPN client. There are trade-offs, of course. For instance, requiring end users to sign on to both the VPN and the application occasionally causes confusion and necessitates some hand-holding. Still, the IT staff is happy with its decision. In addition, though the center's Aruba AP-60 APs support 802.11a/b/g PHY (physical layer) specifications, all end-node equipment supports 802.11b.

"We were faced with replacing 32 handhelds at a cost of about $1,200 to $1,500 per handheld or finding a solution for the Casio," Yuan says. They ended up installing the VPN client for about $100 per device. We asked about using WPA (Wi-Fi Protected Access) or WEP (Wired Equivalent Privacy), but in fact, neither of these is supported on the E700 units. On the other side of the 802.11 compatibility spectrum, security personnel at the AA Center are testing a handheld video-surveillance system that runs on iPaq h5400 units (the security app was created by local company DVDallas). These units do support advanced Wi-Fi authentication and encryption, but to keep things standard, they also are set up for VPN authentication. Surveillance feeds run about a second behind because of latency, but so far, this has not been a problem for the officers involved.

Network access is based on user identity; the Aruba switch accesses the center's Windows Active Directory RADIUS server for authentication. At the moment, all authenticated users have access to the entire wireless network. Although Aruba's much-touted policy-based firewall feature can block various types of traffic by user logon, the staff isn't yet limiting access, simply because it does not yet perceive a need to do so. Wireless VLANs for the inside building staff, events, the ordering system and the planned public VLAN are in place. At the moment, VLAN access-control lists in the Nortel routing switch regulate traffic between VLANs. The VLANs on the wireless infrastructure are based on ESSIDs (Extended Service Set Identifiers); while QoS (quality of service) is available, the center hasn't needed to go live with it as yet.

The AA Center staff does plan to implement user-based internal firewalling to gain greater control--currently, anybody can log in from any internal network segment. Yuan sees a potential problem with this, and he's looking forward to locking down users to specific apps and sites while letting IT do administrator work from anywhere on the network. He wryly brings up the Welchia incident as an example of segmentation benefits.

Although Aruba supports VRRP (Virtual Router Redundancy Protocol), the system has been very reliable thus far, particularly during the critical "game day" window--those hours when fans are in the arena--so funding for the redundant hardware required for fault tolerance of the wireless system hasn't been a priority. Still, Heinlein says that redundant hardware is on his wish list, and at press time, we found out that plans for adding this redundancy have been finalized; the necessary equipment will be arriving in a month or so. When asked whether the center would consider adding VoWLAN (voice over WLAN) phones, given that redundancy is on the way, he says it's on the radar but not a must-have at this time.

One function that requires an optimum mix of compatibility and security is providing WLAN access to reporters and photographers on the stadium floor, with their wide range of gear--equipment running everything from Mac OS 9 to Windows ME. The AA Center's answer: WEP, locked down to specific MAC (Media Access Control) addresses. The key benefit, according to desktop technician Byron Sharp, is that this method prevents media representatives from sharing access. Because each rep pays a fee, a conventional user name/password system would make sharing access tempting. Locking down to the MAC address handily deals with this concern. Because the events VLAN is used, the users are restricted to the DHCP server, the DNS server and the Internet. A public-access portal is planned, with not only a separate VLAN, but also bandwidth limitations using features of the Aruba switch. Because bandwidth control is possible, Yuan and Heinlein are contemplating whether an additional Internet line is necessary. "To be able to turn them on, turn them off, turn a particular VLAN off on any AP at any given time" is a big win, Heinlein says.

In fact, the Aruba console gives Heinlein and staff an enviable level of control, but management features are not Dick-and-Jane simple--even the technically astute Yuan says he wishes he could have been in on Aruba's training (he was on walkabout in New Zealand during this early stage of the project).

What We Like About You

When we asked AA Center IT staffers what they like best about the Wi-Fi network, we got a number of responses. Heinlein cites security benefits as well as ease of deployment and configuration. In addition, the staff didn't miss having to manually tweak power levels. Although the advanced system features can be complex, the wireless tuning is automatic. "The system does configure itself," Heinlein says. "You don't have to be that site survey accurate."

In our recent review of WLAN gear (see "Update: Wireless LAN Battle Plan,"), we confirmed that Aruba does indeed shine in security and configuration functionality. The company's dense wireless deployment scheme, for example, lets the main switch easily make power-level and monitoring decisions. The main Aruba switch receives feedback from all APs and adjusts AP power levels automatically for optimum coverage. At the AA Center, most AP power levels are pegged at 20 mW--creating a small and dense grid, just like the marketing literature says. As new APs are deployed, the system determines if the coverage is too dense. If so, it chooses an AP to turn into an air monitor; these provide feedback for autotuning. The arena map and floor plan were downloaded into the switch as a JPEG graphic at the time of configuration (an AutoCAD drawing will be imported soon). Heinlein and staff can view coverage areas, override the automatic power levels if they wish (they haven't needed to so far) and keep track of which users are associated with which AP, with a quick resolution as to the user's physical location on the map. While at the arena, we logged into the public network, and the center's staff was able to triangulate to our laptop as we walked around. Although the staff downplays its triangulation abilities, the tracking is good enough to send someone to a specific area to investigate a rogue AP or misbehaving laptop.

Another thing Heinlein likes about the Aruba system is its point-and-click ease of setting up new APs. He cites provisioning a venue for a big event, such as an NCAA game, which usually involves a number of trailers in the parking lot, all of which need connectivity.

"It's a helluva lot easier giving them access outside wirelessly than it ever would be running phone lines or Ethernet wires all over the place," Heinlein says. "We've lived through that."

Finally, with third parties using the WLAN, the ability to troubleshoot, monitor and deny is critical. One feature Yuan likes is Aruba's ability to do live packet capture from any node on the network; he's successfully used Ethereal to decode the packet captures. Aruba's blacklist ability is also a hit.

"We've got it set up so that if you screw up a login a set period of times, it throws the device into a blacklist and starts a DoS [denial-of-service] attack against the device," Heinlein says. That rivals having a couple of NHL enforcers at your beck and call. Game point for IT. Jonathan Feldman is director of information services for the city of Asheville, N.C., and a contributing editor to Network Computing. Previously, he was director of professional services at Entre Solutions, an infrastructure consulting company based in Savannah, Ga. Write to him at [email protected].

Title: IT Director

At Work: Responsible for all aspects of IT, including data and telecom

At Home: 52 years old. Married, no children. Hobby: home improvement Alma Mater: University of Toledo; B.S. in business administration

HOW HE GOT HERE: 2000 to present: IT director, American Airlines Center

1997 to 1999: Marketing representative, The St. Paul Companies

MOUTHING OFF: Best part of the center's wireless network: "Getting it up and seeing that it works."

Worst part of the center's wireless network: "Putting it up and seeing that it doesn't work." I work at American Airlines Center because: "They let me come in every day and play with their toys."

Worst moment of downtime in your career: "A couple of years ago, we got hit by the W32. Welchia and W32.Blaster worms, and we were down for about 48 hours. That was the 48 hours from hell."

Funniest comment ever heard from a user: "Oh, is that where you turn it on?"

If only I had a bigger IT budget, I would: "Finish the wireless mesh across the building."

Biggest nontechnical challenge: "Managing my staff's schedule." Most misunderstood aspect of my job: "People think I know more than I do."

If I had the wireless project to do all over again, I would: "Have a better appreciation for the wide range of devices that will connect to it."

I love technology when: "It has value and it's used properly."

I hate technology when: "It ought to work and doesn't."

My next career: "Retirement." Favorite Dallas Maverick: "Michael Finley--he's been through the bad times and the good times."

Title: Enterprise Engineer

At Work: Responsible for network planning, design and management at American Airlines Center

At Home: 31 years old. Single. Hobby: travel Alma Mater: University of Texas at Austin; B.S. in computer science

HOW HE GOT HERE: 2000 to present: Enterprise engineer, AA Center

1999 to 2000: Senior consultant, UGotACall

MOUTHING OFF: Best part of implementing a wireless network: "Control, control, control."

Worst part of implementing a wireless network: "Too many records out there." Worst moment of downtime: "When we got hit by the [Welchia] virus two years ago. I was in China on vacation, and I got a call."

If only I had a bigger IT budget, I would: "Get the latest and earliest of everything. Then I'd get a plasma screen for myself--a bigger one."

Greatest business challenge: "Finding new, out-of-the-box technology opportunities for our sponsors."

The most misunderstood aspect of my job: "There's a perception that [IT people] don't do anything. If people don't see you, they think you're not working, but there's more stuff going on in the back end than anybody ever knows or realizes."

If I had the Aruba network project to do over again, I would: "Take the training classes, both before and after implementation." When I retire, I will: "Be on a beach somewhere."

Favorite Dallas Maverick: "Dirk Nowitzki--he's the best player on the team."