Phone companies: Extending the Disaster

Rebuilding a datacenter at a new location is... painful.The fact that we lost it all in a flood is bad, the process to restore it all is worse.Insurance works at whatever speed they work at, and we're trying to work on publishing deadlines. Tough balancing act.But my story today is about phone companies. Installing phone and DSL service in the new location should be trivial in comparison to the other things we need to do. You'd think.

Twice we sent a staff member out to the new lab to sit in the emptiness for four hours and wait for the telephone installers. On the third try someone actually showed up to do the install. That's twelve hours of our time wasted.

But that's not the worst of it. When the installer arrived, I happened to be the one on-site, and within minutes I knew it would be a long day.

I don't believe in keeping people around who don't perform. It's not in the best interests of the organization, and increases the workload on everyone else. But I don't control the phone company, and this gentleman spouted a never-ending stream of vitriol about his employer and the amount of work they expect of employees.

So I settled in for a long day in an empty lab with a disgruntled employee. I was not disappointed. Our installation was all that the gentleman did for the day.Finally, at nearly 3:00 (I had predicted he would be done at 2:30, late enough to not take another call), he was packed up and gone.

I tested the phones, they worked. So you would think that would be enough, right?

Nope. While I was at the new lab, Lori was on the phone with the DSL vendor. DSL had been installed, but was not configured correctly, she was trying to sort that out.

She had to leave when I got back, so I took over for her, awaiting a return call from the vendor.

They called, and I quickly realized why Lori was still working on it.Here are some tidbits from my conversation with their Tech Support -

Me: "I can't traceroute to the router, tracerouting to it goes off to New York and then dies." (we're going from Green Bay to Green Bay)

TS: "What IP are you tracerouting to?"

Me: "Our router's external address..." I read the IP off to her.

TS: "What's the protection address?"Me: "HuH? What is a protection address?"

TS: "Your external address."

Me: "The address that I just gave you."

TS: "No, your external address, what's your public address?"

Me: I read it to her again. It's public, it's assigned by them.TS: "I'm going to reboot the router."

Me: "Why? It's working fine, we're able to get out from the lab, it's getting to it from outside that's the problem. The routes on your side are messed up."

TS: "Why do you need all those addresses anyway?" (we have a class C)

Me: (thinking "None of your business, we pay for it" and "what's that got to do with anything?") "We're a testing facility."

TS: "I'll have the tech call Lori back."Me: (frustrated after my day of dealing with phones...) "Good idea."

The connection went dead. No niceties or anything, just dropped.

That was last week. We still aren't routed correctly. Lori's still talking to them almost daily. And I have a reveiw that starts on Friday and must have DSL service. The lab will be almost empty, but testing must go on.

When Lori made arrangements to move Internet service, she looked at our options, and chose to stay with our existing provider, mostly because no one else (including the local cable provider) would commit to correctly routing a class C network.

Now I think fondly of local Time-Warner ads where an associate of ours - Jesse Schuchart, IT Administrator for the local Radisson Inn and Conference Center - is walking down a hall and the voiceover is saying "With Time Warner Cable, Jesse Schuchart gets the job done..."Perhaps working with someone smart enough to admit that they don't know how to correctly route a class C would have been a better choice. At least they know what they don't know. But our current provider was already routing it for us, so we thought maybe they could get it right. We were wrong, apparently.

There's more to frustrate a guy in this whole recovery process, but I'll be particularly happy when the DSL is up - the phones work just fine, so I guess we're technically half way there, right? Right?

Don.