Network Computing is part of the Informa Tech Division of Informa PLC

This site is operated by a business or businesses owned by Informa PLC and all copyright resides with them. Informa PLC's registered office is 5 Howick Place, London SW1P 1WG. Registered in England and Wales. Number 8860726.

When Disaster Strikes

We have been working on moving Network Computing's Green Bay lab for more than a year, and last week finally had the opportunity to tour the new lab space as they were building it out.
The reasons for our move were many, most revolving around being in the basement of a building with a sump-pump in it. I have feared a backup of that sump pump since I first came on staff.
Little did I know, that the sump-pump was the least of our worries.
Last night we received alarms from our NetBotz that said humidity was approaching 100% and the temperature was 245 degrees. We hopped on our Axis camera and took a look at the lab. All was dark (as it should be when no one is there), but the lights of the machines were glowing green in the darkness and there were no signs of fire - something you'd see in the dark if the temperature was 245 degrees in the lab.

We made some jokes and decided it was a non-issue.

We should have listened to our trusty little NetBotz. It seems that one of the tenants above us had a hot water heater, and that hot water heater broke. The rising humidity was the water pooling in our drop ceiling and evaporating. The temperature spike was when the ceiling gave way and dumped many gallons of hot water on our running infrastructure.

I have no idea how we escaped without a fire. Dozens of servers were dowsed in gallons of water.

We spent our day trying to dig out of the mess. Steve and I missed all of our calls for the day, and Lori had to send her vendor home - there just wasn't any way to test.

  • 1