iSCSI Takes on Fibre Channel for the All-IP Enterprise

Pay your money and place your bets: As 10GigE makes an impact in data centers worldwide, which bandwidth-hungry application will benefit first from its massive pipes? Will it be networked storage, or will it be high-speed links between Gigabit Ethernet switches embraced at the server level mounting a challenge to Gigabit Ethernet as the new standard? Today, big and fast enterprise storage is pretty much the exclusive turf of Fibre Channel SAN networks, and the FC platform isn't about to surrender ground without a fight.

In the data center, a surprising number of companies are living the all-IP dream. Most small and midsize businesses and a healthy number of small and midsize enterprises continue to use DAS and NAS as their primary storage models, for financial reasons. But as an enterprise grows, its IT support team is eventually forced to deal with many dozens of servers and manage hundreds of terabytes of critical data on a 24x7x365 basis. At some point, storage consolidation gets serious consideration. Then the challenge is choosing the method of getting data to servers.

Gigabit Ethernet is the common standard for connecting servers to the rest of the enterprise, but much of the heavy lifting required for large-scale, critical enterprise storage is done using SANs connected over a separately managed, Fibre Channel network. Developed in the mid-1990s, storage-specific FC networks have been available in 1-, 2- and 4-gigabit versions, always staying a few steps ahead of comparable Ethernet technology. Now, with 10GigE gaining popularity, Ethernet is a serious contender in the bandwidth category. But as far as storage is concerned, it's not all about network speed.

For the enterprise user, FC represents a mature storage technology that offers high availability, massive growth potential, numerous first-tier vendors and a much larger installed base than IP storage. All this provides a great deal of comfort to storage administrators late at night, and IP-based iSCSI SANs haven't been around long enough to earn the same level of respect.

But in spite of their overwhelming popularity, FC SANs also have a well-deserved reputation for being complex and labor-intensive, requiring a support staff with specialized skills. Personnel with FC experience are far more scarce and command higher salaries than those with Ethernet chops, not to mention that FC networking hardware is substantially more expensive than its Ethernet counterparts and suffers from interoperability issues between equipment from different vendors.

Perhaps the biggest challenge for companies adopting FC lies in the level of complexity that a second network architecture adds to an already taxing data center environment. This will probably remain the cost of doing business for high-end systems needing thousands of terabytes of unified storage for a while longer; but there are a large number of environments whose networked storage needs could be handled using IP technology.

Rise To The Challenges

When it comes to IP-based storage platforms there are two choices: NAS and iSCSI SAN. NAS, perhaps the most common IP storage alternative, has been around longer than SAN technology, and NAS systems are increasingly popular for a number of primary storage applications needing file-level, shared access. NAS is common in most enterprises, implemented as stand-alone systems or through the use of dedicated NAS gateways that interface directly with an existing FC SAN to provide CIFS and NFS network shares using the SANs storage pool; see "NAS Road Map".

But as useful as NAS can be, some large-scale business applications and transactional databases still demand block-level storage for performance or to support their specific data access requirements.

In the all-IP enterprise an iSCSI SAN can offer the same capabilities as an FC SAN but instead connects to clients using conventional Ethernet hardware. Standardized in 2003, the iSCSI protocol is designed to encapsulate common SCSI commands and data transfers into Ethernet packets. More important, all that's required for a client to access iSCSI networked storage from most popular operating systems is an Ethernet adapter and a freeware iSCSI initiator to handle security and protocols.Today, iSCSI accounts for only 2 percent to 3 percent of the overall storage market, but according to IDC, the iSCSI SAN market posted 105 percent revenue growth year over year. Compare this with a total worldwide disk storage systems market that grew at a record year-over-year rate of 13.3 percent and a NAS market that grew 7.5 percent in the same timeframe.

The challenge for iSCSI SAN technology will always be to stand up to an apples-to-apples comparison to FC SANs. Many enterprise storage managers still don't consider iSCSI substantial enough for large-scale primary storage, mainly due to capacity and performance rather than capability. As a result, it's usually relegated to secondary tasks, like replication and departmental storage, even though there are many case studies proving iSCSI's worth as a primary storage platform in scientific, government, banking, financial, manufacturing and educational markets, among others. EqualLogic, an iSCSI SAN leader, lists NASA-Langley, FermiLab, Lockheed Martin and the FBI as reference customers.

Room To Grow

Like FC, there are no theoretical limits to the maximum capacity of an iSCSI SAN, but most systems are designed to scale only to the 100-to-150 TB range. This isn't a technical limit but more a response to a youthful iSCSI market that isn't making any larger demands.

Don't think that the big players in FC are ignoring the growing popularity of IP storage. Even FC giant EMC is IP-friendly; with a line of high-performance IP gateways that provide NAS and iSCSI connectivity to even their largest FC SAN products. In addition, most of EMC's midrange SAN line is available in either FC or iSCSI, and the company recently introduced two new, completely IP-based systems. The NS-350 and NS-704 Scalable Integrated Platforms combine the features of iSCSI SAN and NAS, and can be expanded to include modules of EMC's Centera CAS archiving system to provide unified, multitiered IP storage.Newer iSCSI-specific SANs companies like EqualLogic and LeftHand Networks bypass the monolithic approach to scalable SAN storage and design their iSCSI offerings around a more modular concept. In this tack, each multidisk rack enclosure is based on inexpensive SATA drives and contains its own storage controller, multiple 10GigE ports and integrated SAN software that offers single-pane-of-glass management as the SAN grows to fit your needs.

These modular SAN "building blocks" let companies purchase storage in 1 TB to 7 TB segments as needed and offer highly advanced virtualization features that simplify volume expansion, support automated provisioning and offer integrated failover, replication and snapshot capabilities. The next-generation ease of use of these systems is augmented by performance that scales in a linear fashion.

The high-level management capabilities found in modern iSCSI SANs also have reduced the need for third-party management tools common to many FC environments. This market for SAN administration products has grown in response to the complexity required to manage all of the elements in a large FC infrastructure. Fortunately, these lessons have not been lost on iSCSI vendors, which are integrating many of these high-level management tools directly into their SAN products. In addition, IP storage doesn't have the low-level port, switch and device management issues inherent in FC, and IP network management for storage is no more involved than that for any Ethernet infrastructure.

Then there's iSCSI virtualization: Unified storage pools, automated volume expansion and seamless data migration are common features on new modular iSCSI SANs. Unfortunately, many of these virtualization capabilities are vendor-centric, and as the number of iSCSI vendors grows there will be a growing need for systems to support virtualization and provide unified management across multiple vendors in a mixed iSCSI environment. Although these features are also available in FC environments, they are usually not standard features of FC SANS.

Safe at Any Speed

Most modular iSCSI SANs offer multiple load-balancing Gigabit Ethernet ports, with the number of available ports increasing as new modules are added, providing for linear bandwidth growth. Vendors claim that most iSCSI applications rarely fill the bandwidth provided by multiple Gigabit Ethernet ports, but with the introduction of 10GigE this capability will be improved to a point where the network is no longer the bottleneck. The introduction of 10GigE iSCSI systems will also mean that the single-port bandwidth of Ethernet will exceed that of FC, which will be commonly available in 4 Gb within the same timeframe.

As a protocol, FC is somewhat more efficient than TCP/IP and capable of transferring data in much larger frames; but the option of using Jumbo Frames is now supported by practically every NIC, iSCSI HBA and Ethernet switch vendor. The biggest challenge for the upcoming 8-Gb and existing 10-Gb FC standards will be to match the rapidly falling port prices of Ethernet in order to remain competitive as 10GigE becomes pervasive.

Of course, raw bandwidth isn't everything. An issue that continues to concern IT managers is the higher processor overhead required by servers handling TCP/IP, compared to FC storage. This is being addressed by a number of vendors offering TOE (TCP Off-load Engine) NICs and iSCSI-specific HBAs that work with the OS to off load the IP stack. These iSCSI-capable accelerator cards offer a reduction in processor utilization and can improve throughput as either iSCSI initiators at client level or as iSCSI targets on the host storage system.

ISCSI accelerators have gained popularity in conventional environments where small transfer sizes and/or IPsec encryption put an extra burden on a server's processing capabilities. Device drivers that support iSCSI acceleration are available for most common server OSs. For Windows environments, Microsoft is working, in cooperation with Alacritech and Broadcom, on "Scalable Networking Enhancements" for Server 2003 as well as a completely redesigned networking strategy for the upcoming Longhorn and Vista platforms. Microsoft says these enhancements--known as the "Next Generation TCP/IP Stack"--will offer numerous improvements in the Windows networking environment and provide support for TCP/IP off-loading at OS-level.Also in the product-development stage are single-chip, reprogrammable 10GigE extensions that provide IP off-loading and support for multiple protocols and take advantage of IP-specific features added to the RDMA (Remote Direct Memory Address) protocol; iWarp for TCP/IP and iSER (iSCSI Extensions for RDMA) for iSCSI. These extensions are also designed to streamline IP communications by completely off-loading IP transport processing, bypassing intermediate buffer copies and eliminating the need for application-level context switching.

For iSCSI, this means that a system using an RDMA-enabled NIC and iSER can achieve true zero-copy behavior by bypassing time-consuming buffer writes to minimize IP processing overhead, even at 10GigE speeds. Several vendors of Gigabit Ethernet iSCSI accelerators already support RDMA, but surprisingly, most have not yet made any 10 Gigabit announcements.

Using conventional Gigabit Ethernet NICs, iSCSI can consume between 10 percent and 20 percent of a single processor's capacity, depending on workload. This means that the tenfold increase to 10-Gigabit Ethernet speeds could easily use more than a whole processor of a multiprocessor system, just to handle network traffic! Depending on your application this could make TCP/IP and iSCSI off-loading a requirement, and Ethernet acceleration will become a necessity rather than an option.

From a security standpoint, there's still some confusion when it comes to iSCSI. The common perception is that it's less secure than FC because it shares the same network, and therefore the same vulnerabilities, as other Ethernet applications. In reality, iSCSI traffic is usually connected on a completely different network, or at least on a dedicated storage subnet that's easily isolated from normal network traffic. In addition, iSCSI offers CHAP (Challenge-Handshake Authentication Protocol), which requires a secure passcode at both client and server level, and provides full support for IPsec as well as other IP encryption algorithms.

FC's security is based on the fact that it exists on a completely different network fabric, but only iSCSI offers both authentication and encryption as security features. Regardless of the storage fabric you use, security at the client-server will continue to be the primary problem; and ultimately the security of your network is only as strong as the weakest point in your infrastructure.

So Where Do We Go From Here?

There's little doubt that eliminating Fibre Channel as a storage interconnect will reduce cost and complexity, but the big impetus for change--common and inexpensive 10-Gigabit Ethernet--is still a year or more away.

But these are overriding concerns for only the very largest enterprises; in an increasing number of small and midsize environments it will become harder to justify FC storage.

Until that time, organizations looking to upgrade or invest in consolidated storage need to factor in realistic growth requirements, hardware and network redundancy, maintenance costs, and perhaps above all, the cost for management and support personnel.

As your company grows and you're faced with the decision to make a major move, say from 2 to 4 Gb Fibre Channel or perhaps to a larger storage system entirely, it also might make economical sense to begin a gradual replacement of FC systems with IP as best suited to the application.If 10GigE were cheap and readily available, this would have been a completely different article. But for the next 12 to 18 months the balance is still somewhat weighted in FC's favor. There's no doubt, though, that the coming of 10GigE, advancements in network adapters and increasing iSCSI SAN options will add another entry into what used be a one-horse race.

Steven Hill is a Network Computing technology editor. He previously held positions as an IT manager, imaging systems administrator and independent consultant. Write to him at [email protected].