Its the IT departments job to keep corporate networks running, and that means we IT technicians do a lot of running ourselves, because were the ones who get the frantic calls when things go wrong.
In the interest of self-preservation, we spend a lot of money on sophisticated network management systems, on security information systems, and other products that immediately alert us when the network slows down or has other problems.
But knowing that a problem exists and fixing it are two different things. Even if you get an instant page or email when a problem appears, its the amount of time it takes to locate and fix the problem that will determine just how much damage it will do to your company. Infonetics Research Inc. estimates the cost of network downtime as ranging between $1,500 and $97,000 per hour, so even if you know right away that a problem is happening, it really pays to spend some effort on speeding up problem resolution.
Unfortunately, the typical IT shop spends relatively little on tools and procedures that could speed up problem resolution. Its time to let the pendulum swing to the resolution side for a while.
Here are a few ideas about how we can start to focus more on problem solving:
- Distribute network analysis functions. You have to be able to locally analyze a problem in order to fix it, but most companies dont have a fully distributed network analysis function. In many cases, distributed means a portable unit you can ship to a remote office or send along with a technician. But travel time means downtime, so your company should invest in tools that allow you to remotely analyze packet streams, log files, and other network operational details from the network operations center. Ideally, the remote probes/appliances should be intelligent, analyzing as much data as possible for viewing at any time, real-time, or post. This distributes the analysis functions and keeps management traffic to and from the NOC to a minimum.
- Consolidate and standardize tool sets. Packet analyzers, expert tools, "syslog" servers, and other troubleshooting tools arent standardized within departments, so cross-training is difficult, and the reports from such tools will be more difficult for management or other techs to read. By standardizing on one set of tools, IT teams can better leverage training and manpower resources.
- Empower technicians. Most large IT shops have a tiered technical support structure, where technicians help end-users, perform software updates, or handle other routine tasks, while fewer, more seasoned engineers deal with tougher problems in higher support tiers. But, the higher up the expert food chain you go, the more those experts are often tasked with non-support functions like network infrastructure design, upgrades, and expansion. By providing tools that help bridge the gap between technicians and experts, you can have them solve or perform triage on complex problems and save labor in the higher, more expensive support tiers. Youll not only give the Level 3 people more time to do their other non-support jobs, but youll allow the technicians to provide higher-tier personnel with better information to solve tough problems. Standardized tools will help in this process, especially if they include expert systems that simplify and expedite the diagnosis of common problems.