By Marshall Breeding  Storage for the Network: Designing an Effective Strategy Hardware Options Magnetic Disk Drives Magnetic disks offer high-performance permanent storage for computer systems. In general, we consider magnetic disks to offer excellent performance, but at a relatively high cost. Individual disk drives have a long, but finite, life expectancy. As mechanical devices, we have to assume that they can fail at any point. Magnetic drives can be combined into arrays to increase capacity and reliability. How they work The storage of data on magnetic disks is one of the most fundamental computer technologies. They use basically the same technology as audio recording--that of changing the magnetic polarity of ferrite particles. With disk drives the ferrite particles are platted onto aluminum platters. These platters are divided into concentric circles of data areas called tracks, and each track is divided into sectors. Drives will have a magneto-resistive head that traverses over the platter as the drive spins to read and write data. Performance Considerations The two main performance factors include the sustained data transfer rate and the average response time. The data throughput of the drive largely depends on the rotational speed of the platters. The faster that the drive spins, the faster that blocks of data can be written or read. Currently, drives are available that spin at 5,400 to 10,000 RPM. The average access speed is determined by how quickly the read/write head can find the required data block. The response time of the drive is a function of the quickness of the read/write head. High-performance magnetic drives have average response times of less than 12 milliseconds. Capacity The overall capacity of the drive depends on the number of platters, the number of tracks per platter, sectors per track and the overall density of the recording surface. Magnetic drives accommodate a wide range of storage capacity needs. Single drives range from 100 MB to 23 GB. Generally the smallest drive that would be used in a server-class storage system would be 2 GB. Reliability The reliability of computer components is measured in mean time between failures (MTBF). As mechanical devices, magnetic drives cannot be expected to last forever. Look for the largest possible MTBF ratings. For a network storage system, do not accept drives with less than 300,000 hours MTBF. The MTBF ratings provide a relative benchmark for comparing the reliability of equipment. Don't expect this to be a guarantee, however. In most cases the manufacturer calculates the MTBF based on empirical failure data. Theoretical MTBF ratings make a very liberal set of assumptions. Manufactures collect data on component failures, discounting failures that occur through shipping, installation mishandling, and the like to predict the number of hours the equipment should run before it fails. Some vendors also provide an Operational MTBF that includes failures of all types, including DOA's and early-life failures. Alternatives Magnetic disks dominate as the primary technology used for permanent data storage. They offer relatively fast performance at moderate cost. Higher performance and higher-cost alternatives include solid-state "disks" that rely on static memory devices rather than magnetic recording technology. Optical technologies offer options for lower-performance, high-capacity data environments. Magnetic disks require a set of controller circuits to communicate with the host computer. The controller, as its name suggests, manages the basic operation of the disk drive and transfers data from the drive to the computer's operating system. The controller circuits may reside as a separate adapter card, may be integrated on the computer's motherboard, or can be embedded inside the disk itself. The disk drive and the controller must be of the same type. Currently, two types of disk controllers prevail: ATA/IDE and SCSI. Disk Attachment Technologies One of the greatest technical issues related to storage hardware concerns the method used to connect the drives to the host computer. As you design a storage environment, don't let this part of the system become a bottleneck. Make sure that the method that you use to attach the storage devices to the server can handle the load. Large storage systems may require faster host adapters or spreading the peripherals across multiple controllers. SCSI and its enhanced variants are the standard for network storage systems. Avoid IDE drives. Fiber Channel attachment has recently emerged as the high-end approach. ATA/IDE (AT Attachment with Integrated Device Electronics). These drives are rarely used in network storage systems. This is the simpler approach for attaching magnetic disks to a computer, used primarily for desktop computers. With IDE, most of the electronics needed to control the device are imbedded on the disk itself. No separate controller hardware is required, and many computer motherboards provide connectors for IDE drives. This approach to magnetic drive attachment includes limitations that effectively prevent their use in large-scale network storage systems. In most cases only two IDE drives can be chained together. They deliver data at only about 3 MB per second, though with the newer Extended IDE architecture, the data transfer data increases up to 11 MB per second. Another major limitation is that only one data request at a time can operate on the drive chain, which is not acceptable in a multiuser envrionment. The original IDE specification supports drives up to 504 MB, but Enhanced IDE can now handle up to 8-GB units. SCSI: The Small Computer Systems Interface SCSI offers a more flexible environment for interconnecting storage devices with reasonably high performance. Almost all network storage systems use some type of SCSI to attach peripherals to the host computer. Magnetic disks, optical drives, and tape drives all connect to their servers via SCSI host adapters. The SCSI architecture allows multiple devices to be chained together off a single controller. With SCSI, peripherals of differing types can be connected together to form a single bus, with terminating resisters installed on each end. While the original SCSI implementation allowed only seven drives per SCSI bus, newer versions allow more. Each device, counting the SCSI host adapter, on the chain is configured with a Logical Unit address from 0 to 7. SCSI uses a parallel data transmission architecture-the initial implementation deals with 8 bits at a time. Extensions of SCSI increase the width of the data path up to 16 and 32 bits. The maximum data transfer rate in the original SCSI design is about 5 MB per second, a factor of the 5-MHz bus speed. Multiple requests can operate on the SCSI bus simultaneously, allowing a group of devices to work toget her. Enhancements to the original SCSI specification have been developed to yield increased capacities and performance. SCSI-2 This initial enhancement to SCSI can deliver data up to 10 MB per second with an 8bit data path. This version includes data caching and other performance-enhancing features and extends the number of devices per bus to 15. Wide Fast SCSI-2 By both increasing the bus speed and increasing the data path to a width of 16 bits, Wide Fast SCSI-2 can yield performance of up to 20 MB per second. Ultra SCSI This most recent enhancement to the SCSI architecture increases performance to 40 MB per second. Each of these enhanced SCSI implementations maintain backward compatibility with previous versions. Fiber Channel: This peripheral connection technology offers a major improvement in data throughput beyond the previous versions of SCSI. Though Fiber Channel was initially designed for fiber optic technology, its use in connecting disk drives generally relies on non-optical connectors. While SCSI uses a bus architecture, Fiber Channel connects devices in a continuous loop. This method of connecting devices is thus called Fiber Channel Arbitrated Loop, or FC - AL, and can yield data transfers at speeds up to100 MB per second. While the connectivity model differs from SCSI, the low-level data access commands and methods are much closely related and is sometimes called SCSI-3 or SCSI -FCP (Fiber Channel Protocol). In addition to faster data transfer rates, fiber channel allows greater distances between peripherals (up to 30 meters) and can connect up to 126 devices. While fiber technology, now in its early stage, costs more than traditional SCSI attachment, its costs are expected to drop to a competitive level. FC-AL, with its major advancement in performance and reliability, enables new opportunities for storage architectures. The ability to interconnect more devices and to deliver data at dramatically faster speeds facilitates the creation of much larger storage arrays than were previously possible. Through FC-AL these storage arrays can be defined as network devices, associated with multiple servers. Network-attached storage devices can be created that allow computers to access data at extremely high-speeds, completely bypassing servers. Applications that require extremely large bandwidth, such as streaming video, would benefit from such an architecture. New models of server clustering are emerging, and the possibilities for high-performance large-large scale storage systems that can be shared on the network in very flexible ways will help shape future of computing. RAID In the era of mainframe computing, the primary model of storage involved the use of very large disk drives. These refrigerator-sized Direct Access Storage Devices (DASD) were designed to offer high reliability and availability, but were expensive to own and operate. The magnetic drives designed for PCs offered lower capacities, and lacked the reliability of mainframe DASD. Beginning in the late 1980s, methods were developed for combining a number of smaller inexpensive disks to form logical storage units that exceeded the capacities and reliability of large disk structures. These multidisk arrays could be constructed for a fraction of the cost of the mainframe disk systems. We know that basic magnetic disk drives can offer reasonable amounts of storage with high performance. The strengths of magnetic drives can be expanded through the use of RAID technologies. RAID, or Redundant Arrays of Inexpensive (or Independent) Disks, combines multiple magnetic disk devices in ways that increase capacity, reliability and performance. The initial specification for RAID was developed at the University of California at Berkeley. This proposal defined five RAID levels, and compared the reliability performance and capacity of these systems to the large-scale mainframe disks commonly used at that time. To implement RAID, additional intelligence must be introduced into the storage system. In the simple disk storage system data are written and read directly from a single disk. But with RAID, a more complex process is involved in reading and writing data. The management of the disk array can be handled either through software or through hardware in the form of specialized disk controllers. For RAID levels that involve the generation of parity information, software solutions may be less expensive, but require more system resources on the host. Hardware-based RAID implementations require expensive special-purpose disk controllers, but offer better performance and reliability. The implementation of RAID involves a trade-off in that the the use of more storage devices doesn't directly scale the net usable storage capacity. (Some amount of space is used for redundancy of data.) Consequently, the relative hardware cost of a RAID system depends on the usable storage capacity percentage. For configurations that require complete redundancy, the usable storage capacity is only 50 percent of the gross hardware capacity. Some RAID options may offer 80 percent to 95 percent efficiency of the hardware's capacity. Many RAID storage systems use a special controller that manages the distribution of data on the disks. The RAID controller can connect to the host computer as another SCSI device or it can connect directly into the bus of the host computer. Since most computer system busses operate at faster speeds than SCSI allows, expect some performance gains from controllers that connect this way. SCSI-attached RAID controllers are usually easier to configure, making the entire RAID system appear as a single SCSI device. Another concern in RAID systems concerns data caching. Earlier RAID systems had performance problems, which have been largely been overcome in more modern systems through caching. Many RAID controllers offer a large data cache that allows the RAID subsystem to operate with significantly better performance. The main disadvantage concerns f ailure. If the storage system fails--especially if the failure is in the controller-- there must be a way to save the data in the memory cache and get it written to disk. Otherwise, data lost is irretrievably lost. RAID-based storage systems almost always rely on SCSI disks. These disks are widely used in both Unix and PC servers, both in conventional storage configurations and in RAID. The use of RAID usually introduces a penalty in performance. The management of parity information requires additional processing. Some improvement in the performance of RAID systems can be gained through caching. The official classification of storage devices is maintained by the RAB (Raid Advisory Board) established in 1992. Vendors of storage devices may submit their equipment to the RAB for classification for a fee. Once classified, the vendor may display a logo corresponding to their assigned RAID or EDAP (Extended Data Availability and Protection) level. (http://www.raid-advisory.com/index.html) Below are the different options, or levels, available to implement RAID. Level 0 Simple data striping. This storage strategy involves combining multiple drives into a single logical data volume. The data are then interleaved among multiple disks. Having multiple drives service data yields an increase in performance over nonarray disks, because the data is being written in parallel. Applications that use data sets larger than a physical disk benefit from the ability to create larger disk volumes through RAID 0. This RAID strategy offers no increased benefits for data availability over conventional disk storage. Level 1 Data mirroring. This level of RAID involves a redundancy of storage devices. With RAID Level 1, data are written to two separate disk drives. If one fails, the other is available to service data requests. This method provides excellent data protection, but doubles the number of devices required for storage needs. Level 3 This method involv es data striping as does Level 0, but it also uses an additional device that is dedicated to holding parity information that can be used to reconstruct the data stored on any of the other storage devices. RAID Level 3 stripes each byte of information across multiple disks and places a parity bit on an additional disk. In the event of the failure of one of the data disks, the information on the parity disk can be used to recreate the data on the failed drive onto a replacement drive. The cost of RAID 3 involves the purchase of one additional drive for each storage system beyond that needed for data only, plus the cost of the software that manages the RAID array. The overhead of RAID 3 introduces a performance penalty that make it unsuitable for applications with very intensive I/O activity, such as online transaction processing systems. Level 5 This disk configuration also uses parity to reconstruct data on a failed drive, but distributes the parity information on the data drives. This distributed parity approach offers better performance than RAID 3, reduces the extra hardware needed and still offers redundancy. This implementation of RAID still suffers somewhat in disk writes compared to RAID 0, but is generally considered the fastest reliable RAID option. Level 6 This RAID level was not part of the original Berkeley specification. While RAID 3 and 5 can handle the failure of a single disk in the array, they still are vulnerable to multiple failures. If two drives fail simultaneously or if a second drive fails during a data rebuild, the entire data structure is lost and will have to be restored from backups. Level 6 RAID was designed to accommodate the loss of two drives simultaneously or can handle a disk failure during a data rebuild after an initial failure. This additional failure resistance is accomplished through multiple parity devices, but the additional overhead introduces additional performance loss. Other RAID Levels Some vendors have extended the R AID Levels to higher numbers, indicating a combination of disk options. For example, RAID Level 30, would represent a system that combines Level 3 and duplexing. The use of these higher numbers is not supported by the RAB. EDAP While storage systems have historically been classified by RAID Levels, RAB has begun using new terminology to describe storage reliability. RAID describes particular technical implementations of data storage that protect the storage system from a disk failure. But may other components comprise the storage system. Disk controllers and their on-board cache memory can also fail and lead to data loss and unavailability. The language that describes highly reliable storage systems must encompass the entire storage system, not just the disks. For this reason, the RAB now emphasizes Extended Data Availability and Protection (EDAP), which considers all components of the storage system. There are three general EDAP levels: The RAB classifies a device as Failure Resistant Disk System (FRDS) to systems which provide protection against the loss of data during a disk failure and maintain access to the data during the disk failure. The disk mirroring and parity RAID 3, 5 and 6 accomplish this degree of EDAP. A Failure Tolerant Disk System (FTDS) represents a greater degree of EDAP capability, and applies to systems that can protect data and maintain access to data during the failure of any component of the storage system-disks, controllers, and power supplies. To achieve FTDS capability, systems must be equipped with redundancy on a wider scope than just the disks. FTDS-compliant storage systems will have redundant power supplies, redundant disk controllers, multiple cooling fans, and advanced environmental monitoring and control. Such systems take into account the relatively high percentage of failures that occur through non-disk failures. The highest level of EDAP compatibility, Disaster Tolerant Disk System (DTDS), applies to systems that protect data in the ev ent of a complete failure of equipment in a given geographic area or "zone." Systems that meet this level of protection will have data distributed in separate physical locations. When selecting a RAID system look for some of these features. Hot-swappable components. To maintain high-availability, the ability to replace components without shutting down the server is critical. Redundant power. Power supply failures comprise a fairly high percentage of storage system failures. Look for systems that include redundant power sources. Temperature controls. Excessive heat can lead to storage system failures. Not only should you consider redundant cooling fans, but you should also look for systems that monitor operating temperature and automatically generate alerts. An overheated unit can lead to multiple simultaneous failures--the worst possible scenario for a storage system. Optical Storage Optical storage technologies can play a part in an organization's storage strategy. They offer a lower-cost alternative to magnetic disks for extremely large data sets. Magnetic disks offer significantly higher performance, and their costs per storage unit consistently decline. But for extremely large storage environments, it becomes impractical to rely solely on magnetic disks. Optical technologies, as removable media, can offer a lower cost solution for large-scale data. Optical media come in several flavors-MO, Phase Change and CD-Recordable. Each of these different variations offer differences in cost, performance, capacity and number of times the media can be rewritten. Media Management You will need to select a process for managing your optical discs. All the optical technologies involve discs of removable media. Cost efficiency comes through the use of multiple media units per drive--contrasting to magnetic drives which have fixed media. Several options prevail for handling the removable media. Manual operation. If you expect to have only a few op tical platters, then you might use one or more basic optical drives and change media manually. This option suits only environments that use magnetic media for archiving data for safekeeping. If your volume of data is small enough that it fits on one or two platters that can be left in the drive constantly, you should probably consider sticking with magnetic drives. Optical disc changers. These devices manage optical discs, storing the disks in cartridges or magazines and loading them into a drive when needed. A minichanger might handle a dozen platters or less, and jukeboxes will manage hundreds. The number of optical discs that can be mounted at a time corresponds to the number of optical drives imbedded in a jukebox or minichanger. The jukebox can house a large storehouse of discs, mounting discs on demand for users and applications. The structure of an optical disc changer includes one or more optical drives, a set of slots for storing discs and a robotic mechanism that selects and loads discs into a drive. Optical disc changers store and manage a large number of discs, but include a limited number of drives. The number of embedded drives defines the number of discs that can be read simultaneously. A large capacity optical disc changer, for example, would hold 150 optical discs and include four internal optical drives. Such a unit would allow any four of the 140 discs to be accessed by users on the network simultaneously. The optical jukebox uses a robotic mechanism to transfer discs from their standby location to the drives. The changer locates the desired disc and loads it into an available drive and returns it once the data request is satisfied. Expect the optical disc changers to find and mount an optical disc in about five seconds. Typical applications Optical technologies fit within a storage environments that include requirements for large data sets, relatively modest performance and a low cost per storage unit costs. Consider optical media for the part of your storage environment that requires "near-online" access. Most implementations of optical technologies involve hardware that requires a few seconds to mount the disc into a drive, and the data transfer rates are slower than magnetic drives. Rarely would an organization use optical technologies for their primary storage environment. Organizations that implement Hierarchical Storage Management (see below) typically use optical media to off-load data from their primary storage system. Other applications for optical media include those that involve large document or image libraries. An organization might digitize a large collection of documents, for example, and store the images on optical discs, which would be called up out of a jukebox as needed. Data archiving often involves optical technologies. Information that has reached its final form may be deposited on WORM discs to ensure that no further alterations occur. Media Options Optical storage comes in several varieties. CD-ROM/Recordable (CD-R) and Write Once Read Many (WORM), support only a single write to the media. Other optical media, such as those using magneto optical and phase change technologies allow data to be re-written multiple times. CD-ROM, a read-only media, fits into the storage strategy for data or software distribution. MO (Magneto Optical) One of the most popular optical technologies uses the magneto optical process. Magneto optical drives use both magnets and lasers to read and write data. To write a bit of data, the laser first heats a spot on the disc to its Curie point (the temperature at which a material looses its magnetic properties) and then a magnet changes its polarity. The process of writing data takes two passes. During the first pass the laser is turned on to high power and the magnet is set to change all points to magnetic north, or zeros. On the second pass the laser is set to fire on only for the points to be set as one, or magnetic south . During this second pass the magnet is set to constant south polarity. For reading data, the laser fires at lower power. Differences in polarity cause slight differences in angular deflection, called the Kerr effect, causing the points to be read as ones or zeros. Phase Change The other multiple-write process, called phase change, writes data with a single pass. Phase change drives use only lasers and write in a single pass. Both methods offer similar performance for reading data. WORM Write Once Ready Many drives allow data to be written to a disk only once. This type of technology works well for those who rely on optical media for data archiving. CD-ROM/Recordable, one of several WORM implementations, employs a purely optical process. A laser fires at high power to heat microscopic crystals imbeded in the media. Differences in reflectivity cause each spot to register as a one or a zero. The same laser fires at low power to read data. CD-ROM CD-ROM fits into a storage strategy for distribution of prepared data. Organizations may purchase data that is delivered on this media, or they may master and produce CD-ROMs internally. While the equipment for creating CD-ROMs is fairly expensive, drives for reading this media are quite inexpensive and each new generation of CD-ROM drives offers remarkable improvements in performance. You can now purchase 10X or 12X drives that offer very impressive data throughput abilities. A number of products are available that provide shared network access to CD-ROM discs. Solid State Disks (SSD) If you must store data with extremely fast performance, you might consider Solid State Disks. These devices, not really disks at all, use large arrays of DRAM to store data. Since no mechanical limitations apply, SSDs can offer incredible performance-almost instantaneous writing and reading of data. Expect response times of at least 40 times that of magnetic disks. SSD's emulate SCSI magnetic disks. They use the same ca bles and connectors and appear the host computer's operating system as a conventional SCSI device. SSD's main challenge concerns the fact that they rely on volatile memory--when the unit loses power the data is lost. To be made nonvolatile, these devices must be equipped to ensure that this can't happen. Most will have battery backup and will have embedded magnetic drives that can receive the data when the unit must be powered down. A nonvolatile SSD can be used for applications that have extremely harsh performance demands, and where a finite number of files can be identified that require this performance boost. It would be completely impractical to construct a disk storage system completely from SSD. Volatile devices without data preservation safeguards are available, but must be used only with applications that do not involve critical data. Some applications, for example, might need extremely fast temporary storage for intermediate results sets or indexes. Make sure not to use volatile SSD with data that cannot be easily reconstructed. The performance advantage of SSD comes at a price. You will only use this storage technology if you need a storage tier of extremely fast performance. Plan to pay $65 per megabyte for a large-scale system (950 MB) or $80 for a smaller system (132 MB). Few organizations have applications that justify this level of expense for high-performance storage. Hardware Scoreboard Single Magnetic Drive A single drive connects to a single controller in a network server. Pros: Inexpensive, simple to manage Cons: Limited capacity Single drive or controller failure causes system unavailability and possible data loss Best suited for: small departmental networks. Tips: Perform frequent backups, use SCSI drives for expandability, consider implementing mirrored drives or RAID. Multiple Magnetic Drives Several drives conne ct to a single controller in a network server. Pros: Inexpensive, simple to manage Cons: Moderate capacity Single drive or controller failure causes system unavailability and possible data loss Best suited for: departmental networks Tips: Perform frequent backups, use SCSI drives for expandability, consider implementing mirrored drives or RAID Mirrored Drives (or RAID Level 1) All data are written to two separate magnetic drives. Pros: High failure tolerance. In the event of a failure, the storage system relies on the remaining functional drive set. Performance is about the same as nonmirrored system Cons: Expensive-mirroring doubles the number of drives required for data storage. Vulnerable to multiple simultaneous failures. A controller failure may cause system unavailability Best suited for: Networks with relatively small data sets that require high performance Tips: Consider duplexed drives for increased reliability or RAID to decrease the number of drives Duplexed Drives All data are written to two separate magnetic drives that connect to separate controllers Pros: High failure tolerance. In the event of a failure, the storage system relies on the remaining functional drive set. Performance is about the same as nonmirrored system Cons: Expensive. Duplexing doubles the number of drives required for data storage. Vulnerable to multiple simultaneous failures. Best suited for: Networks with relatively small data sets that require high performance Tips: Consider RAID to decrease the number of drives, though this might result in a slight performance penalty. Use hot-swappable components to recover from failures without scheduled downtime RAID Level 0 Data stripped across multiple drives Pros: Increased capacity with high performance. Efficient use of disk drives--all drives used for active storage Cons: No increased resistance to failure. Single drive or controller failure can result in system unavailability or data loss. Best suited for: Large data sets that demand high performance storage Tips: Consider implementing Parity RAID or mirroring the drives RAID Level 3 Data striping with a dedicated parity drive Pros: High failure tolerance. In the event of a failure the storage system can rely on the parity drive. After the replacement of the failed drive, the data can be automatically regenerated. If the parity drive fails, the parity information can be reconstructed Cons: The calculation of the parity information introduces a performance loss. Even more performance degradation can occur during and after a failure. Best suited for: Environments that require high availability Tips: Use hot-swappable components in the storage system RAID Level 5 Data striping with distributed parity Pros: High failure tolerance. In the event of a failure the storage system can rely on the parity drive. After the replacement of the failed drive, the data can be automatically regenerated. If the parity drive fails, the parity information can be reconstructed Cons: The calculation of the parity information introduces a performance loss. Even more performance degradation can occur during and after a failure. Best suited for: Environments that require high availability Tips: Use hot-swappable components in the storage system Print This Page E-mail this URL