The Banality of IT Failure: Overlooking Mundane Insider Threats

A few days ago, someone on a security mailing list I subscribe to mentioned that he couldn’t access two of his banking sites very early on a Saturday morning. He and others on the list wondered if it was another in a wave of denial-of-service (DoS) attacks leveraged against financial institutions. It had to be, because large banking and ecommerce sites didn’t have outages or need scheduled maintenance windows anymore, right?

Wrong. In most cases an organization’s worst enemies aren’t on the outside, but I’m not referring to an Edward Snowden type of malicious insider threat. The biggest risks to an enterprise are generally the most mundane: The absence of documentation, poor planning for upgrades or changes, lack of communication between teams, shortfalls in oversight, badly designed applications, and non-existent or inadequate testing. Most organizations don’t need to be attacked by hacktivists or a foreign power. They can implode very nicely from the inside and from the most banal causes.

I recall working in an organization with a sub-standard data center. Actually, it was much worse than that. Besides the bad wiring, improperly installed fire suppressants and constant temperature issues, it was positioned directly under a public restroom. The inevitable occurred one day while I was offsite in a training class, so I managed to escape the worst. Let’s just say the term “s***storm” took on a whole new meaning after that event. Senior management was aware of the risks, knew we needed to get out of that space, but institutional inertia prevented any action.

And that isn’t the worst I’ve seen. There were the switches that melted in the summer heat because they were placed in rooms with inadequate cooling. A PBX failed after sitting in four inches of water, unnoticed for weeks, because the environmental sensors were placed improperly or simply didn’t work.

Then there was the time when a major provider of dark fiber for a company where I worked accidentally segmented our network during maintenance by cutting the redundant ring at both paths. Wonder if the technician had a physical diagram?

These horror stories still make the most seasoned professional shudder in terror: Tales of missing backups desperately needed after a failed upgrade; incomplete documentation that should have mentioned why ICMP was necessary in a firewall rule because it was required for a critical application to function; and the SSLVPN upgrade breaking access for an executive because he didn’t have the right version of Java on his laptop.

[Read why organizations need to focus on proper security design rather than spending more on security technology in "Security Needs To Focus On Architecture, Not Products."]

Are these breakdowns directly attributable to a technical problem or is it really about poor communication? Good business continuity and disaster recovery requires collaboration, but this seems to be in short supply in most organizations. Why worry about hackers taking down our infrastructures when they’re already rotting from the inside?

Organizational psychiatrist Bill Kahn refers to this common breakdown as "the Ostrich Effect." It’s an identifiable series of events building to institutional apathy. The source is a difficult human interaction, which leads to anxiety and an unwillingness to deal with the conflict that has arisen, so the focus is directed externally, usually to some false problem. The result is a level of passive-aggressive behavior that hijacks progress and eats away at morale.

Is it actually possible that many of our problems in information technology could be solved by simply talking to each other? This might be a hard sell for hardcore engineers in a field overflowing with talented braniacs who believe they can fix anything with a line of code or a piece of hardware.

But if good communication techniques were so easy, then we wouldn’t have the need for so many conflict resolution specialists, psychologists and professional team-builders. Many technologists forget that there’s a person on the other side of the keyboard and our infrastructures are often just external expressions of universal problems related to the human condition.