On The Wire

More Horrifying Tales From The Network Crypt

Ever hear of a duplicate IP address horror story? Check out this dilly we were e-mailed from Averill Bromfield.

"We began receiving reports that end users were having trouble logging onto a server. We poked around the server and hubs and checked out the physical connectivity, which looked good. Next we broke out the protocol analyzer and the hub management software. We were able to identify one IP address for two different MAC [Media Access Control] addresses, but had no idea where these workstations were.

"Worse, we found out that the duplicate address was that of the server in question. How could that be? We were going to trace back the MAC address through the hub ports, back to the cabling plant, but then some one mentioned that some applications people had been upgrading their old machines.

"We were using a version of TCP/IP that, when connected to a host, would display the IP address of the host in the lower right hand corner of the monitor. The applications person believed that this address was his workstation's, so he reassigned it to his new PC. We ended up with a server's IP duplicated at a workstation!"

Our next story from Tom Clark of Rockwell International is definitely a "switch in time."

"A new Sun [Microsystems] application server was experiencing NFS [Network File System] time-outs and poor network performance. The Sun was on a switched Ethernet subnet and was a duplicate of another server on the same subnet that was not experiencing any problems.

"We noted that pings from the new server to any other node were taking between 15 and 200 milliseconds, while pings from the old server to the same nodes were taking about one millisecond. Analyzer traces showed no anomalies in the packets and confirmed that the time delay was network-related rather than host-related. Moving the server to other switched segments had no effect on performance.

"The only element common to all subnets was a Kalpana EtherSwitch. Since the old stations on the subnet were working fine, we felt that the switch was doing its job normally. But a quick examination of the switch statistics via the switch console gave us a hint as to what was wrong.

"The older Kalpana EtherSwitches have a limit of 1,700 nodes per port. When this limit was reached, no new nodes could be added to the address table and the switch didn't 'age out' old addresses that were no longer active. This switch had more than 3,800 nodes that had been seen on one of its ports. When the port-node limit is exceeded, all packets from new nodes are dealt with by the system CPU rather than the high-speed switch fabric, causing traffic to bottleneck and slow down.

"Clearing the port-address table allowed the new server to be added to the switch table and netwo rk performance improved immediately. Meanwhile, we're investing in newer hardware so this problem won't reoccur."

Our previous story about the night watchman's flashlight triggered this response from Jeff Bachovchin from St. Joseph's University.

"We had a Macintosh server that would constantly go down sometime during the night in one of our buildings. There were no errors or signs of any problems, but every morning the server would be down. After days of running around the building looking for any clues, a secretary stopped us and said how funny it was that this started happening at the same time she noticed that every morning when she came into the building, all the power was turned off. She said she was wondering if maybe that had anything to do with it. It turned out that there was a new maintenance man on the overnight shift who didn't like the lights on, so he turned off the main breaker in the building. Problem solved."

Think your network is hot? Check out this story from reader Mark Deal.

"For the past eight months my firm has been blowing through 3Com Corp.'s Etherlink III EISA adapters in one of our four servers. We are, unfortunately, still running 10BASE-2 coaxial segments.

"Recently, one of the segments went down and my assistant dutifully responded to the problem area and began terminating the segment in different areas searching for a break or other problem. He suddenly jumped back from one PC's T-connectors, claiming that he could actually feel current.

"We pulled out a volt meter and found 40 volts running across the connection. We immediately disconnected the segment from the server and the current remained. We disconnected the segment piece by piece looking for the source and traced it all the way to the last PC on the segment. We disconnected the PC and the segment still had current on it to the tune of 118 volts, even when nothing was connected to it!

"We eventually discovered that we could discharge the current and it would not come back. We reconnected the 'faulty' PC a nd rebuilt the segment and measured no current. Two hours later and ý BOOM! The segment stopped working. We went back to the 'faulty' PC and again found 118 volts running across the segment. We discharged it and discovered that the faulty PC was bleeding power into the segment slowly, until it built up enough energy to neutralize the segment entirely.

"This problem was difficult to trace because that particular PC was used by the general partner of the firm and he rarely had it turned on for more than an hour at a time."

This last story is one of our favorites from Alan Cohn of Wishard Memorial Hospital. We were bowled over.

"Our new workstation wiring was done using Category 5 cable. The other day we happened upon a large spool of old Ethernet thick coaxial cable. Now 1,000 feet of Category 5 wire fits a fairly light and small spool. Imagine the weight and size of 1,000 feet of thick coax on a spool. We had to move this hefty spool to a storage area elsewhere on campus. As we were loading the heavy spool of cable into the back of a pickup truck, it slipped off and began rolling down the parking ramp. One of our technicians was walking up the ramp to join us when he saw this huge spool of wire hurtling toward him.His eyes turned as big as saucers and he barely managed to jump aside as the spool sped past!"

Got a good horror story to share? There's always next year to make your claim to fame. Send us your favorites at otw@pmg.com.

Bill and Scott can be reached at otw@pmg.com. Portions of trace files from selected columns are available via Pine Mountain Group's Home Page (www.pmg.com).