Storage RACs Dow's Brain

ORLANDO, Fla. -- Storage Networking World -- After struggling through installations of Oracle RAC on a variety of storage and operating system Dow Chemical Company Senior Systems Administrator Steve Remsing's most useful counsel for admins has to do with staff unity.

"The biggest piece of advice I have is: Get your storage admins and DBAs [database admins] on the same page early," Remsing told a group of storage professionals today at SNW.

Remsing says his nine-person team at Dow consisted of database and storage admins, and he attributes tight integration with getting through a series of frustration problems. But he admits both groups were territorial at the start - especially when it came to decide who would manage Oracle Automatic Storage Management (ASM). ASM isn't required for RAC, but provides a file system, logical volume manager, and software RAID for RAC.

"That was a little fight we had," he says. "The DBAs are used to managing all of Oracle including the disk, and the storage admins are used to managing all the storage. Now, they're merging together. We realized we're going to have to form a much tighter relationship as we go forward."

Storage and database admins better get used to working together. Oracle is among the crucial applications for networked storage. RAC (real application cluster) lets you cluster servers together for increased reliability and availability. You can add and remove nodes from the cluster, and if a node fails, it doesn't affect sessions running on other nodes.While installing Oracle RAC, Remsing says he learned other important lessons about the relationship between Oracle and storage. Those lessons included that Oracle RAC performance doesn't rely on any single type of storage. For instance, he says there was little difference between running it on an EMC CX300 Fibre Channel or Network Appliance FAS840 IP SAN. He installed RAC on both systems, running Windows and Linux on each.

He says when Dow puts RAC in production, it will be on an iSCSI system.

"We proved it worked with iSCSI," he says. "There was a question if it would hold up to the performance load. We didn't see any difference between iSCSI and Fibre Channel. The underlying storage doesn't really matter to Oracle."

Just in case iSCSI doesn't hold up, Dow recently purchased a NetApp FAS270 system with Fibre Channel and iSCSI connections.

For all its benefits, Remsing's team found setup simplicity isn't one of RAC's advantages. His team hit snags along the way in each configuration. "Antivirus software conflicts with Oracle RAC and ASM software and prevents proper installation," he says. "After installation, it prevents access to the OCR [Oracle Cluster Registry] disk."Other tips he picked up were that the operating system versions and patch levels must be identical across the nodes, and the firmware versions of HBAs must be identical across nodes. He says Oracle's recommended workarounds didn't always work, and his team had to find its own workarounds -– in some cases writing BAT files.

But Dow eventually got RAC working with Windows 20003 Server and Red Hat Linux Version 4 Advanced Server on both storage systems.

"In the end, we could explain all the strange behavior we saw with the storage," Remsing says.

— Dave Raffo, Senior Editor, Byte and Switch