>> continued from previous page
Scheduling & Notification
Scheduling a change well in advance becomes extremely important from several angles. The most obvious is that a user who finds out on Friday evening that a server will be undergoing scheduled maintenance on Sunday becomes incredibly angry: maybe she had ditched her plans for Sunday to get some work done and she doesn't have any other time that weekend to do her work. The resulting delays not only have business consequences but get blamed on your department.
Advance scheduling also lets your IT teams sync up and contemplate the change before it happens. Take a case where a server is going to be decommissioned: All the users have been moved to a new server, and it is assumed that the plug on the old one can be pulled. But what if someone in the infrastructure group way back when installed a network service (nonredundant DHCP for a small department, for example) that didn't get documented and thus didn't show up when the server's dependencies were checked.
By scheduling the change well in advance, you at least have a chance that the person in the infrastructure group will realize (with a guilty start) that the DHCP service will have to be moved to another box before this one gets decommissioned. If it were simply assumed that the server could be shut off after all the users get moved, network problems would ensue as soon as the leases expire.
Choosing your change notification window depends largely on how nimble you want to be. We find that three business days tends to work well for small changes (such as decommissioning a server), but we like to give a week or longer for significant infrastructure changes (like IP renumbering or a switch to a different directory service).
One large university IT department, for example, used to differentiate between both the anticipated length of the outage as well as the immediacy of the need. For routine maintenance, it established windows when it could do maintenance with very little advance notice. Typically, these were times when the organization was least busy -- late Saturday night or very early Tuesday morning. IT personnel publicized that people should schedule their activities based on the assumption that there might be maintenance during those windows. For bigger projects, more advance notice was given. This IT organization, like others we've visited, put together a voicemail system where they could notify key people at the departmental level of any unexpected or emergency downtime, and used this system to keep users informed about problem-resolution progress.
E-mail has its place here, but don't forget face time among your peers. Daily change-management meetings -- with the entire technical staff (for small organizations) or representatives of functional groups (for larger ones) -- work well. This type of meeting should steal only 10 minutes out of your staff's day, and can be a huge whoops-stopper. Let's face it: Not everyone reads e-mail closely, particularly when it's sent in bulk. The face time and shortness of this daily change-management meeting forces folks to listen up.
Rollout strategies vary widely among technologies and businesses, but there are some constants. One good rule of thumb is have good backups; another is to break projects down into small, testable deliverables. A good incremental rollout plan (deploying first to IT testers, then to one, five and 10 production users, then one and five departments at a time) usually means that rollback isn't ever used, but that doesn't mean it's not necessary. The one time you don't formulate a rollback strategy will be the one time you need one.
Technologists also need to get good at figuring out when to roll back when things aren't working out as planned. For example, if you have a two-hour maintenance window and you are 90 minutes in and it takes 40 minutes to back out, you have a problem. While obsessive tendencies have their place, the drive of some technologists to keep plugging away, thinking "it's gotta work this time," can only blow out the maintenance window. It's hard to know when to quit: Sometimes backing out prematurely can cause hours more of labor. You simply have to weigh that against loss of productivity and act accordingly.
Technology-specific change management guidelines are available, and some vendors are more than willing to give you a hand. For example, from the network-infrastructure standpoint, Cisco Systems has a white paper you can find online. Ask your vendor rep; you might be surprised what a good resource he or she is.
Jonathan Feldman is chief technical manager of the Chatham County Government in Savannah, Ga., and a Network Computing contributing editor Send your comments on these articles to him at jf@feldman.org.
RSS
