Network Computingwww.networkcomputing.com

Unfortunately, we can't say the same of our new systems. Many are evolving as technology changes, and we've found ourselves running production applications systems about which the operations staff knows little to nothing. Everything is fine and dandy until something breaks and we haven't a clue how to fix it. Worse yet, we have to deal with hundreds of individual platforms -- each with its own personality, hardware and software.

Recently, we began a serious effort to document our systems, starting from their inception, to provide better support once the systems achieve production status. Aside from documenting the obvious nuts and bolts of a system, we include dependencies on other systems, steps involved for disaster recovery, key contact personnel, vendor phone numbers, security information and so on. The more information, the better. We're also putting it online so it will be accessible from the nearest browser with appropriate security.

Much effort has been spent providing standards for hardware and software platforms, helping to provide consistency whenever possible. Having published standards means new systems have a better chance of fitting seamlessly into our environment. In the past, systems were sometimes developed without serious regard for operational issues. This created more "systems from hell" than I care to remember. With the number of applications we support today, it's impossible for us to know every idiosyncrasy. It's not easy, but we try to keep things as standardized as possible around here.

Over time, we have created a change-control process to follow the evolution of each system throughout its life. This process is aggressively backed by management and must be followed (or else!). Even when an emergency forces us to act now and follow up later, we make sure the change-control work gets done.

A change-control process has two main functions. First, it creates a mechanism whereby the proper people are notified of changes so they can provide input if any problems arise. Second, it ensures we update system documentation. Done properly, this method guarantees that we end up with a complete history of every system. And it's all just a click away. Surely, this beats scrounging to find the last person who worked on a specific system and then picking his or her brain.

Another change-control process benefit involves your retired systems. Do you ever give much thought to the possible consequences of retiring a system? Probably not. Just erase the data and shut it down. R.I.P. A long time ago, we had a power outage that knocked out a bunch of old DECservers. When the power returned, the DECservers didn't work because they couldn't find the host VAX that provided them with their download software. Someone finally realized we had decommissioned the host VAX months earlier. If we had our change-control system back then, we would have documented this dependency when the DECservers were installed and avoided the problem.

Most of the time, dealing with a change-control process seems too much of an annoyance -- and sometimes it is. But when it's 3 in the morning and you're the one trying to get a sick server up and running again, having all this information available is anything but annoying.