State of the Art: How to Build an iSCSI SAN

Talk the Talk

In iSCSI parlance, an initiator is a device or software that maps SCSI into IP: It wraps SCSI commands in an IP packet and ships them to an iSCSI target. The target machine unwraps iSCSI packets from IP and acts upon the iSCSI commands. It returns an iSCSI response or multiple responses, which are usually blocks of data.

The server is your application server, and the storage box is the machine serving up iSCSI drives. (We're using storage box to represent anything from a Linux software iSCSI target to a full-blown SAN with iSCSI support.)

You need a gigabit copper network for an iSCSI SAN. If you try running iSCSI over a 100-Mbps network, you'll be disappointed. Assuming your network connection maintains 100 percent utilization, 100 Mbps is roughly equivalent to 5 MB per second of disk transfer. Because iSCSI has a request/response for every packet transferred and network performance degrades before 100 percent saturation, the best performance you'll get is 6.25 MBps of throughput. That's a rough estimate that includes time to wrap and unwrap data packets and responses to each packet.

Bottom line: 6.25 MBps of data transfer is not good, considering that most drives run in the 40- to 320-MBps transfer range. Besides, Gigabit Ethernet is affordable: Gigabit adapters start at $60; switches, $120. And Gigabit Ethernet has the throughput for treating iSCSI as a local drive.

Don't put your iSCSI SAN on your regular IP network, either. There's plenty of traffic running over that, and iSCSI is bandwidth-intensive. Also consider whether your servers have enough CPU power to handle iSCSI. Unwrapping and reassembling iSCSI packets can take a lot of CPU time. The iSCSI standard assumes packets are received in order, while TCP does not (plus iSCSI adds load from intensive TCP/IP processing). So if your server CPU is moderately utilized, you'll need an HBA (Host Bus Adapter) or TOE (TCP Offload Engine). These devices take care of some, or all, of the iSCSI and TCP processing without burdening the CPU.

HBAs are storage-only cards that connect your machine to a target using iSCSI. TOEs are TCP-only cards that off-load TCP processing for the CPU. TOE is useful in iSCSI because of the high volume of packets transferred, while HBA processes SCSI commands--another data-intensive application. HBAs costs $600 to $1,200 each, so add them only to machines that need more CPU.

And check with your HBA vendor to ensure that its product supports regular TCP/IP communications (most don't). If it doesn't, buy a separate gigabit NIC for that machine if it will handle any management for your storage network. Ideally, the NIC should sit on a separate machine on the gigabit SAN--but not participating in storage work--for network management.

Part of iSCSI's appeal is you don't need specialized networking knowledge--like you do with Fibre Channel SANs--to set it up. It's relatively simple to build and configure.

First, set up your IP network. Install Ethernet cards or HBAs, and remember to put one in your storage server if you have a target that requires a card or blade to make it iSCSI-ready. You have several options: At the low end is the UNH-iSCSI open-source project that builds a target for Linux. You can install it on any Linux machine, customize the config file and have an iSCSI target. Fill the box with drives and use it as your storage box.

Alternatively, you can buy premade storage boxes that are iSCSI targets with plenty of room for drives. This is a good place to start if your budget is tight. You'll need to choose the number of drives, the type of drive (SCSI, ATA or Fibre Channel) and how much expandability you need in the device, as well as the amount of throughput.

Another option is to make your existing Fibre Channel SAN and NAS equipment iSCSI-compatible, with iSCSI cards for SANs and iSCSI gateways for NAS products.

Next, run cables to your gigabit switch. Remember, you're creating a separate IP network from your backbone. IP networking is much the same no matter the medium--configure the network using your OS guidelines.

Pain-Free Initiation

Configuring iSCSI differs from product to product, and it's different for the initiator and the target systems. Once you've got your storage IP network running, devices can ping each other, and setting up the iSCSI portion is easy. Just enable the iSCSI functionality.

Using the UNH Linux target software, you can modify the target configuration file to set up drives, portals and targets. Windows and dedicated storage devices have interface tools for this configuration.

Once you've defined the targets and drives they offer, set up the initiators. If a given server doesn't have an HBA acting as an initiator, you'll have to install one, like the Microsoft Initiator. This adds value even if you have an HBA installed. Some HBAs can't use the Microsoft Initiator, though, so check with your HBA vendor.

Some vendors have written their own iSCSI initiators that are delivered with their cards. There are initiators and targets for both Windows and Linux available. Windows has an "iSCSI Initiator" icon in the Control Panel for automatically discovering targets, and you can set up undiscovered targets, too. Linux has a config file you modify manually to set up iSCSI, and there's a handy how-to for this purpose. If there's an HBA on your storage machine, check your HBA vendor's documentation for help.

Once the initiators are configured, try to contact the targets. Beware: If you set up the targets after the initiators, you have to rescan the iSCSI network at each initiator to see the targets.

To check that both TCP/IP and iSCSI are working, the iSCSI protocol maps a command to do an "iSCSI ping." It's a SCSI command wrapped in an IP packet that doesn't request any operation from the target--just a response.

Next, select the drives you want each initiator to use, which should be visible as local drives. Format them and begin using them like any other new drive in the system.

Over time, you can augment or replace software initiators with HBAs or TOEs, and you can swap out your lower-end target with anything up to a full-blown SAN. You can even upgrade to 10-Gbps Ethernet for greater throughput, if necessary.

Don MacVittie is a technology editor at Network Computing. Previously, he worked at WPS Resources as an application engineer.

Sites to See

IP Storage Forum

iSCSI Spec Watch

SNIA iSCSI Group

Linux iSCSI Target

Microsoft Software Initiator

Software Initiator