I try not to use this blog simply to indulge in schadenfreude, but I just couldn't resist when I found that the Tokyo Fire Department's emergency call center was down for over four hours on Jan. 5 due to a network error that a few lines in the switch config file could have prevented.
Japanese news site The Mainichi Daily News reported that the Tokyo Fire Department's network, which automatically routes calls to ambulance and fire companies by location and type of call, was off-line for four hours, during which 506 emergency calls had to be handled manually. A manual fire watch was established using helicopters and firefighters with binoculars on the Tokyo Tower.
No officials committed Seppuku or resigned over the incident, and more importantly, there was no reported loss of life, but that helicopter fire watch must have cost a bundle. Kazuo Matsuura, a high-ranking TFD official, said at a news conference on Jan. 7, "I deeply apologize for causing so much trouble and giving anxiety to metropolitan citizens," while bowing and looking quite contrite.
So what was the cause of the problem? An Ethernet cable someone plugged into two switch ports, creating a loop. At some point in 2010, the PC it used to be connected to was removed, and on Jan. 5 someone put the loose end of the cable into an available jack. Manichi Daily reports that TFD is planning to install caps on unused cable ends and ports to prevent a recurrence.
I've seen similar problems when students tried to get more network bandwidth by connecting all the jacks in a dorm room to a cheap Ethernet switch. Since the cheap switch didn't support spanning tree it created a loop between access ports on the campus net. The solution was to fully configure spanning tree on the network. Just adding "spanning-tree bpduguard enable" to the config for all the access ports solved our dorm network problems and would probably do the same for the Tokyo Fire Department. Just goes to show that even the simplest errors can have wide ranging effects.
