Multimedia Rocks Virgin Entertainment Group's WAN

If customers at the Virgin Megastore in Times Square are sampling music or video clips at the same time a store employee places a voice-over-IP call to Virgin Entertainment Group North America's corporate headquarters in Los Angeles, the voice on the other end of the line sometimes comes across choppy--or not at all.

Adding a second T1 line to the New York City store didn't help the voice problems there. The reason? Virgin's increasingly popular digital listening stations, which pull music and video files from an online service, simply need more bandwidth than the retailer allocated. So Virgin is upgrading its 3-Mbps MPLS WAN link at headquarters to 10 Mbps to ensure that voice calls aren't trampled by the Black Eyed Peas or Kelly Clarkson.

Although the company did some basic monitoring of store traffic to ensure it had sufficient bandwidth for VoIP, it hadn't anticipated the success of the listening stations. "Times Square was a surprise," says Robert Fort, director of IT for the music and entertainment company, a subsidiary of the U.K.-based Virgin Group. "We underestimated the number of listening stations that would get [used] at the same time. We thought 3 Mbps could handle it. And, with the high level of activity at that store, there was much more phone traffic, too."

The New York City store has 150 Virgin Vault digital listening stations, which are IBM Anyplace Kiosks running a Virgin-developed Microsoft .Net application. The new listening stations provide faster access to music and far more features than CD-based stations. When a customer wants to sample a song, Virgin Vault first accesses the company's SQL Server at its Los Angeles data center for title metadata, then its Internet Information Server (IIS) for the music clips. The Virgin Vault application grabs the 350KB audio samples from an onsite Virgin Megastore server or off the Internet from online music and video sample services. The New York and Hollywood, Calif., stores--the only ones with Virgin Vaults so far--also run Cisco Application and Content Networking System (ACNS) appliances in-house, to cache some clips and help minimize WAN traffic.

Virgin had previously given voice priority on the WAN pipe, to avoid latency and delay, but even with queuing and prioritization on the MPLS-based WAN, the VoIP call quality suffered when it competed with the Virgin Vaults. "There's only so much you can do once the [pipe] is being overutilized," says Ken McNeil, senior manager of IT administration and support.

With the WAN upgrade under way, Virgin is also working with its MPLS WAN provider, AT&T-SBC Communications, to reconfigure the QoS (quality of service) settings on its Cisco routers. "Once we add the bandwidth, we can come back and re-look at how we're going to carve it out, whether it's 40 percent to voice or whatever," McNeil says.

Virgin went with the MPLS-based WAN service from then-SBC Communications last year to cut its telecommunications costs. The MPLS service, which AT&T-SBC manages, integrated Virgin's voice and data infrastructure, saving the retailer 40 percent to 60 percent in network costs per location. However, the need to add a 10-Mbps link will reduce the previously expected savings by about $40,000. Virgin converted its headquarters and the stores in Hollywood and Times Square--its largest sites--in August to MPLS and VoIP. At the same time, Virgin upgraded its old stand-alone, CD-based listening kiosks to the networked multimedia Virgin Vaults.

Overall, Fort says, Virgin has invested about $1 million in equipment and software, including Cisco 1700 and 2600 series routers, VoIP Call Managers and IP handsets, but excluding the MPLS WAN service itself. That number is confidential, he says. Virgin is gradually adding the remaining 15 stores to the MPLS WAN with VoIP and, possibly, Virgin Vaults. "We still haven't gotten the full benefit" of the MPLS infrastructure, Fort says. He anticipates additional cost savings as more stores switch to VoIP.Virgin's network infrastructure is under more pressure than most retailers because it must support both its operational data as well as entertainment and sampling for its customers. "Most [retailers] have applications that are calling in at night to get sales and pushing down price files," Fort says. "Our infrastructure has a higher demand on it. When we took the leap from broadband to T1, it at first seemed excessive, but now that we're doing real-time polling" and VoIP, it's crucial, he says.

The retailer runs a SQL Server-based data warehouse application that polls the stores every 15 minutes for about 5-MBs' worth of sales and other information. With this real-time data, Virgin can, for instance, adjust the positioning of a new CD or DVD in the store if it isn't moving fast enough. The same application also generates reports for the stores and home office, including key performance indicators.

Virgin also may add real-time video traffic over the network for customer promotions and business operations. For its customers, Virgin may offer videocasts from in-store events--Sheryl Crow and Mariah Carey have recently performed at the New York and Hollywood Virgin Megastores, respectively. "We can extend that type of event to a videocast to each of our locations," Fort says.

Internally, Virgin may add videoconferencing for sales and executive meetings among employees from its geographically dispersed stores. That would mean adding cameras to laptops, and possibly more bandwidth. "But our immediate issue is the voice side," Fort says. "We need to get that in place first. We've learned that you have to have some real-world experience to know the exact bandwidth needs of each station, and so on."

Fort offers some hard-won advice on how to plan bandwidth requirements for VoIP: Keep a good inventory of your current call volume; know in detail your bandwidth utilization by app, location and peak times; and don't forget to factor in future bandwidth-heavy projects.Meanwhile, Fort and his department are exploring whether to add instant messaging with Microsoft's Live Communications Server. Heavy instant messaging would mean adding more bandwidth, too, he says. "We're investigating the business improvements [IM would bring] and its network impact now."

Virgin Entertainment Group, North America

[15 minutes]

<="" span="">Robert Fort

46, is director of information technology for Virgin Entertainment Group, North America. He's responsible for all things IT at Virgin, including multimedia applications and strategy for its MPLS backbone. Fort has been with Virgin for two and half years and in IT for 28.What's cool about working for a music store chain: "The culture. I'm a musician myself, and it's fun having exposure to all the new music and DVDs and all the special artist in-store events we have at Virgin. Plus, being a part of the Virgin brand...everyone, it seems, knows about Virgin, and it leads to great conversations."

How it's not always so glamorous: "There is a lot of hard work behind the scenes at a record store. With over 400,000 SKUs, you can imagine the amount of work involved in purchasing, receiving, pricing and merchandising. It's often a real eye-opener for our new store personnel."

What stinks about VoIP: "It doesn't do dishes."

Why Virgin's MPLS WAN deployment has taken so long: "The AT&T-SBC merger occurred around the same time, and some people we had been working with were rolling off the account altogether. It took a while for the new AT&T team to be established, so that slowed things for us in Phase 2 of the deployment. It set us back a couple of months."

What Fort's co-workers don't know about him: "I can be talkative at times. I'm sure they know more than they want to by now."What keeps him awake at night: "Besides crickets, barking dogs and drag racers, the magnitude of day-to-day technologies that need to operate correctly and the vast number of opportunities we are eager to pursue."

Subject that makes him rant: "Only one?"

Favorite hangout: "The Guitar Center. I have to cut back, though...I already own enough keyboards and guitars."

Comfort food: "Dr Pepper."

Favorite team: "Lakers, baby!"Wheels: "Mazda Tribute. Done with the need for a status symbol, and it's reasonably priced and big enough to tote my kids, music equipment and golf clubs."

In his music player: "The Secret Machines, Pink Floyd, U2, R.E.M., Mark Knopfler and Joe Satriani."

Must-see TV: "ER, Boston Legal, CSI, PGA golf."

After hours: "I take weekend trips with my teenage daughters, play/record music and hack away at golf."

First career: "Stock boy."Next career: "I haven't given it much thought ... I'm enjoying this one."

The Hard Sell

When Robert Fort pitched the MPLS WAN and VoIP to upper management last year, he emphasized the savings Virgin Entertainment Group would realize by ditching its carrier-based VPN voice service for the integrated voice and data MPLS.

"I didn't elaborate excessively on building the platform" for MPLS or VoIP, says Fort, director of IT for the LA-based music and entertainment company, a subsidiary of the U.K.-based Virgin Group that owns Virgin Megastores. "They were not interested in that, but in the costs."

Virgin execs had the usual reservations about VoIP--whether it really worked and about voice quality over the IP network--but Fort eventually sold them on the numbers. "We knew VoIP would never fly if we were to say, 'It's cool, so let's do it,'" Fort says. "You don't look at it as a techie, fun thing to do."By staggering its capital investment payments to AT&T-SBC, the company would come out in the black, he says. "We showed a positive impact on our P&L," Fort says. "That's a great message to a CEO."

Fort convinced management that the MPLS-VoIP project would pay for itself within three years, and that projection is currently on track. "We just presented a good business case," he says. Then he gave the rundown on features in the new Cisco voice equipment, such as how Cisco's Unity Unified Messaging would integrate voice and e-mail messages into one mailbox. The fact that the network provides a platform for future multimedia applications also was a selling point, he says. Virgin has invested about $1 million in equipment and software, including Cisco 1700 and 2600 series routers and VoIP Call Managers, plus funding for the MPLS WAN service--a cost Virgin won't disclose.

Fort says he's learned first-hand how to properly present a business case for an IT project by serving on the company's board of directors, which gives him a useful perspective on Virgin's business direction. That knowledge has paid off: When the company was building its data warehouse application, for instance, management was looking to simplify the stores' key performance indicator reports, Fort recalls.

"When we heard that, we decided that would be the first thing we would do" with the application, he says. So they developed the first real-time KPI reports. "Just by listening [to the business side], there was alignment of the stars," he says.

Fort's next pitch may be instant messaging with Microsoft's Live Communications Server. "It will immediately come across as yet another techie IT tool, but we're taking time to see where we can improve our business processes with it," he says.0