Upcoming Events

Cloud Connect
Santa Clara
Feb 13-16, 2012

Cloud Connect brings together the entire cloud eco-system to better understand the transformation we're experiencing and promises to be the defining event of the cloud computing industry. Learn about the latest cloud technologies and platforms from thought leaders in Cloud Connect’s comprehensive conference.

Register Now!

W O R K S H O P

To InfiniBand ... And Beyond

March 5, 2001
By Steve J. Chapin

Although it sounds like something out of Toy Story, the InfiniBand Architecture (IBA) is actually a new industry standard in I/O interconnections for distributed computing. InfiniBand is a combination of storage area network, communications network and I/O bus. With cluster computing becoming ubiquitous, access to shared data has become the primary bottleneck for many applications in commercial computing.

This wasn't always the case. For some time the primary concern had been communication between machines (particularly in scientific applications). This concern gave rise to specialized machine-interconnection networks, called system area networks, that operated at high speeds and with low latency. These system area networks were often proprietary and intended for message-passing applications. The introduction of high-speed commodity interconnects, such as Myrinet, VIA (Virtual Interface Architecture) and to some extent Gigabit Ethernet, meant that machines could communicate with each other faster than they could access storage. This was fine when distributed computing was a tool primarily used by scientists. But as clusters became commonplace in business computing, storage became the weak link in the throughput chain.

The current solution gave rise to storage area networks, which connect centralized storage repositories to a group of machines in a distributed system. Storage area networks are used only for moving data from storage to computational nodes; message passing is done on a separate LAN or system area network. Fibre Channel is the current state-of-the-art in storage area networks, offering switched connectivity with similarities to communications networks.

Although having one network for interprocess communication and another network for storage (as shown in "Redundant Networks," above) is flexible and lets administrators partition storage traffic from message traffic, the requirement for a separate storage network is expensive and inefficient.

This is particularly true if the storage network is significantly slower than the communications network. This has been the unfortunate case for Fibre Channel -- communications networks work at multiple gigabits per second, while storage networks operate in the hundreds of megabits per second. As you would expect, vendors have worked hard to increase the speed of Fibre Channel, but this has brought on another issue: the inadequacy of the PCI bus (see "Plain PCI," below).

A standard 33-MHz, 32-bit PCI bus has a "guaranteed not-to-exceed" throughput rate of roughly 132 MB per second (or approximately 1 Gbps). Network interface cards have been able to move data that quickly for years now, leaving no bus bandwidth for the storage network. This problem is exacerbated by Moore's Law: The capability of the bus simply has not kept up with the increasing speed of the CPU. The move to 133-MHz, 64-bit PCI-X allows 1 GB per second to move across the bus. While PCI-X alleviates the problem of a single Gigabit Ethernet card saturating the bus, power configurations have multiple Gigabit Ethernet cards, and vendors including Foundry Networks and Extreme Networks have prepared switches capable of handling 10-Gbps Ethernet cards. That puts us right back where we were. The communications network can saturate the bus, leaving no bandwidth for the storage network and other devices. In addition, both PCI and PCI-X suffer because they are shared-bus architectures. All the devices contend with each other for access to the bus. There is also a scalability paradox with PCI. As an individual bus goes faster, fewer devices can be put on a single segment. This forces system designers to add extra buses in the system to maintain expandability options, which has limited potential and complicates system design.