Phone companies: Extending the Disaster
Posted By Don MacVittie at 11:08 AM

Rebuilding a datacenter at a new location is... painful.

The fact that we lost it all in a flood is bad, the process to restore it all is worse.

Insurance works at whatever speed they work at, and we're trying to work on publishing deadlines. Tough balancing act.

But my story today is about phone companies.

Continue reading "Phone companies: Extending the Disaster"


Comment on this blog entry

Images from Green Bay
Posted By Lori MacVittie at 10:00 AM

No, they aren't images of the Packers or cheese. I thought I'd share some of the images of the carnage that was the lab in Green Bay last Friday because as they say, a picture is worth a thousand words. I think the only word that really describes the carnage early Friday morning is "horror". There's just something very wrong about wading through 2.5" of water to get to your equipment. Kudos to our Cisco Catalyst 6500, which was still running and routing traffic despite the conditions.

Continue reading "Images from Green Bay"


Comment on this blog entry

When Disaster Strikes
Posted By Don MacVittie at 06:16 PM

We have been working on moving Network Computing's Green Bay lab for more than a year, and last week finally had the opportunity to tour the new lab space as they were building it out.

The reasons for our move were many, most revolving around being in the basement of a building with a sump-pump in it. I have feared a backup of that sump pump since I first came on staff.

Little did I know, that the sump-pump was the least of our worries.

Continue reading "When Disaster Strikes"


Comment on this blog entry

Practical Jokes?
Posted By Lori MacVittie at 02:20 PM

So I'm in the lab trying to get NWC Inc. ready for our upcoming ESB product review. I'm installing OpenJMS and setting up messages queues and reinstalling servers in preparation for product testing.

I flip the KVM to TEST8, put the CD in the drive and restart the machine. Or at least I thought I put the CD in the right machine. The machine boots up as if the CD doesn't exist.

That's odd, so I start looking at the machines in the rack. Nope, not that one. Nope, not that one. Hmmm..try again. Same behavior.

Turns out that someone put the bevel - where the machines are labeled by name and IP - for the mail server on TEST8 and left TEST8 without any identifying marks. I didn't pull the bevel off the mail server when I ejected the CD to try to identify the machine, because it's the mail server, right? It couldn't possibly be the right one.

I'm guessing someone either didn't know any better, or thought it'd be a good joke to make me spend half an hour figuring out what server I was trying to access.

Fess up, boys. Which one of ya did it?

Never mind...I'll just figure out a way to get back at you.

Bwahahahahaha...

Comment on this blog entry

Connectivity is Good!
Posted By Lori MacVittie at 09:59 AM

Woo hoo! After three weeks of messing around with ChoiceOne, NWC Inc. is back on the map.

Turns out our ISP replaced their border routers about ... well, 3 weeks ago. (Funny that we lost our routes about the same time, isn't it?)

After slogging through Tier 1 support for 3 weeks, someone finally realized that just because the support guys could "ping" our Class C from the ChoiceOne network didn't mean it was accessible from outside their network.

Ya think?

After they finally figured that out (despite being told as much by me for 3 weeks) some router jockey finally went in and began advertising the address block again and golly gee willikers, we're accessible again.

Comment on this blog entry

We are so cool...
Posted By Lori MacVittie at 01:07 PM

...in terms of temperature in the lab, that is.

Anyone who has had the pleasure of visiting NWC Inc. lately has noticed that it's really warm near the servers. Too warm, in fact. Last week the temp starting rising, and rising...until Friday I was getting e-mail like this:

-- Report from Network Computing Green Bay Real World Lab at http://216.43.82.235, at Oct 28, 2005 12:45:45p --

Notification 3 of 4 in the third alert level
The temperature is 90. The high limit has been exceeded

Turns out the air conditioning lines were frozen. Our friendly management company took care of the problem and this morning I was greeted by a NetBotz report that said:

Temperature: 67.1 degrees F {value normal}
Humidity: 44% {value normal}
Airflow: 120 feet/minute {threshold not enabled}

Dayam! Even the airflow is better - we've been hovering around 40 feet/minute - but that 67 degrees is music to my ears. Ahhhh... it's downright chilly in there now. And just in time for me to start gearing up for our ESB review...excellent.

Comment on this blog entry

Grrr....ISP Trouble
Posted By Lori MacVittie at 02:44 PM

For the past week our Class C has been routing off into the ether. Our ISP, ChoiceOne, has been unable to determine why our Class C suddenly disappeared off the face of the earth.

I don't know why either, but I know it shouldn't take you a WEEK to figure it out. Our connectivity has been gone for a week, the trouble ticket escalated twice, and every time I called I was told "the" technician was busy, could he call me back?

Sure, if he's really going to, because thus far "the" tech must be on permanent vacation because I've not heard boo from him.

C'mon, ChoiceOne, figure it out. Either that or hire a second technician so "the" tech can answer my damn calls.

Comment on this blog entry

Master of my own domain
Posted By Lori MacVittie at 10:43 AM

Every once in a while something weird happens and you remember that you can't stand some particular feature in a piece of software that you otherwise don't mind using.

Like the domain joining functionality of Windows.

I booted up a machine yesterday only to find for some odd reason it couldn't authenticate to our domain. Network connectivity to the AD server was just fine, and I couldn't for the life of me figure out why this server couldn't auth to AD.

What I needed to do was simply "rejoin" the domain. I realize that there's very few times when this is necessary, but when you have to do it, it's a PITA, because Microsoft doesn't support such functionality. You have to join a WORKGROUP, reboot, then join a DOMAIN, then reboot again in order to simulate a "rejoin" to a domain.

Needless to say this is what I ended up doing and after the second reboot the server was able to auth to the AD server and all was well, but damn... Why'd it have to be so hard?

There's no mechanism for deleting a domain on a server, and that's got to be painful for companies going through mergers or acquisitions where server consolidation may be occuring that requires a set of machines to be reconfiguring to authenticate to a new domain yet not disturb the applications running on the machine by wipe and replace, which is another solution to this problem.

It's just frustrating, even though it doesn't happen all that often. When it does, you waste half an hour just rebooting machines.

Would it really be all that difficult to add "rejoin domain" functionality? Or "delete domain membership" functionality?

Comments(1)

More Fun With Databases
Posted By Lori MacVittie at 08:30 PM

Today's entry is brought to you by the number 01114 and the letters O,R and A.

That's right, it's "ORA-01114: IO error writing block to file 201"

I started seeing this one late this afternoon while testing one of the products in our BI review and now, at 7:30 in the evening, I have the problem fixed. For some reason the TEMP tablspace in Oracle is maxed out at 4GB. But the real problem was that it must have gotten locked at some point and wouldn't allow any large queries to go through. A quick reboot of the server cleared the TEMP files and released the locks, so finally the BI tools are able to figure out how much $$$ in sales we've made off specific categories of widgets.

In case you're curious, here's the results from the horse's mouth:

SUM(ORDERS.TOTAL) CATEGORY
----------------- ------------
14103848.9 CAFFEINE
12289291.3 IM
13287830.4 LASER
16861087.5 LIGHT
9980358.7 MP3
11493839.8 ROBOTS
11513981.1 SMART CARD
22925430.2 STORAGE
12541190.6 VOICE
12713046.1 WATCHES
18839119 WIRELESS

Reports from the BI tools thus far are accurately reproducing the query, so that's something at least.

Expiring Jars
Posted By Lori MacVittie at 06:36 PM

So around the end of July our customer service application, developed by an outsourcer as part of this review in early 2005, died.

Bruce is conducting a BSM review using NWC Inc. applications, and noted that the app was broken. I rebooted the machine, but still no luck - something had seriously messed up the app. I seriously believed it had something to do with Bruce's review, like an altered classpath or replaced JAR file somewhere, but couldn't find anything wrong.

Today I was hanging with Jeff from Cognos, installing Cognos Series 7 and getting a tour for my BI review when he mentioned he needed a USB fob to transfer some files because the certs used to sign a jar file had expired and that it screws up the cryptography functions in the app if he doesn't upgrade them.

I noticed the files he was replacing were JCE_1.2.1 and a light bulb appeared above my head (no kidding, really - a light bulb, 150 watt). That appeared to be the problem with our app as well!

A quick search on expiration of JCE 1.2.1 turned up some well hidden advisories on the subject, and it turns out that on July 28, 2005 the digital certs used to sign JCE 1.2.1 expired. I replaced the JCE jar files with the updated JCE 1.2.2 files and surprise! The application is back up and running.

I'm a bit miffed that I was in the dark and had no explanation as to why the app suddenly stopped working, but I'm even more disgusted that the plethora of searches I did via Google did not turn up even ONE reference to the 'expiring cert' issue with JCE 1.2.1.

But thanks to some unexpected help, we're back in service.

Dirty Data
Posted By Lori MacVittie at 04:47 PM

I'm currently working on a business intelligence review and the first test began to run into issues as soon as it started trying to make sense of the data in NWC Inc.

Dates that are strings is bad and so is allowing just any old data into the database. BI tools like nice clean data to work with and my dirty data is driving tools crazy.

So I spent the morning cleaning up the data, including adding a new column to handle real dates. Of course then it was necessary to change the widget ordering code to make certain the new dates were entered into the database. And there's always that moment of "should I really hit enter on this SQL statement?" Nerve wracking, to say the least!

Then I added some indexes to some of the commonly used fields for reporting, because churning through more than half a million records can be a pain when you're pulling it across the network and trying to load it into Internet Explorer. I think it's helping... cause I"m finally able to pull data that makes some amount of sense:

YEAR	TOTAL REVENUE
1998	4,838,816.55
1999	4,861,384.30
2000	4,783,757.00
2001	4,813,386.75
2002	5,969.20
2003	47,345,416.00
2004	29,382,997.00
2005	19,190,789.25

Wow. Wonder what happened in 2002 that we only made $5,969? This year's looking good though, isn't it? Too bad it isn't real money, I could buy this, fix it up and still have money left for other things, like gaming books.

Alas, it's back to cleaning up all that dirty data. And you thought your mom wanted you to wash your hands just to keep germs away. She was just worried about dirty data and what problems it would cause later on.

So remember, always wash your hands before you enter data...your DBA will thank you for it.

What's Going On In There?
Posted By Lori MacVittie at 10:56 AM

NWC Inc. is busy busy busy lately. Right now, we're getting ready for Bruce Boardman to test a bevy of BSM (Business Service Management) suites which will track all our transactions and tell us ... something about our business.

Getting NWC Inc. ready for Bruce means some additional applications and infrastructure, so yours truly has been installation apps and writing code frantically to get everything in place for Bruce's testing.

At the same time, we're gearing up for a Business Intelligence suite review. We'll be examining NWC Inc.'s massive amounts of data to search for trends (which widgets are hot, which ones are not) and which region of the world likes to buy which widgets, and why has Mike Fratto been ordering so many widgets for only 5 cents??

We'll start our testing in July to get ready for an October issue, so watch this space for updates and juicy tidbits about NWC Inc.'s widgets as we dig into the data with Business Intelligence suites.

NWC Inc. Widgets have Competition
Posted By Lori MacVittie at 09:29 AM

And people thought NWC Inc. widgets were crazy, with combination Pez dispenser/MP3 players...

ION Technologies just introduced its Flashlite USB Data Drive. It is designed to be multi-functional, by doubling as a utility light and personal data storage device. It uses a super-bright white LED bulb, bright enough to tackle any situation. The added flashlight feature is the perfect solution for locating USB ports in the dark, configuring computer systems in low-lit environments or trying to insert a key into a car door at night. It attaches directly to your key chain so you’ll always have it with you.

Comment on this blog entry

Reboot the Data Center
Posted By Lori MacVittie at 09:18 AM

Some days you just know things aren't going to go your way...

Like when you get to the lab and the UPS is screaming at you. Every five seconds, two short, ear-piercing high-frequency beeps. Annoying as hell.

So you try to figure out what's wrong and realize you don't know a thing about UPS', but hey - you'll play with the little buttons anyway. And then ... you reboot the entire 'data center'.

It's very quiet in the lab when NWC Inc. is shut down.

You turn power back on and let things start booting up. Then you spend the next 10 minutes freaking out because you realize there is a username and password on the KVM and that you don't know what they are. You try to call the editor that set it up, but he's off the grid and you realize he's not going to get back to you any time soon.

You've got less than one hour before Oracle shows up for an on-site installation for a review and you can't get at the machines for the test.

You finally figure out the username and password, because in the lab you always try to make things easy to figure out, and then spend another 30 minutes making sure systems come up correctly; starting those pesky Exchange services that never seem to start correctly even though they're marked as "Automatic" and then wonder why your laptop can't get an IP addy from the DHCP server.

(It helps to turn it back on after you've accidentally shut it down)

Apparently the entire comm rack containing the GB lab equipment is also on that same UPS. So it powered off. 45 minutes later NWC Inc is up and running, but the UPS is still screaming every 5 seconds. All you managed to do was give yourself a heart attack by simultaneously powering down every piece of equipment in NWC Inc's four APC racks.

You finally call the Server and Storage editor and he tells you what to do. Only when you pull out battery R3 and push it back in and you hear a loud popping sound followed by the acrid smell of electronics burning.

Oh crap.

After you spend 5 minutes sniffing around the racks you determine that nothing is actually on fire, but you've just managed to fry one of the batteries - and that ear piercing screech has not stopped. So you tell the Server and Storage editor, "You better get in here...".

And when he does the end result is a fried battery that needs to be replaced and he stops the screeching by unplugging the console from the UPS. It doesn't change the fact that we're down a battery and had to turn off about 1/3 of the machines in the rack...

But at least it's quiet.

Comment on this blog entry

You're Gonna Do What?
Posted By Lori MacVittie at 02:05 PM

The primary connection for our Green Bay lab was down this morning - for no apparent reason. So after power cycling the router and proving to myself that it wasn't a problem on our end, I called our provider.

We traveled through the basic "what color is this light" game and of course all is well. Then the Tier 1 technician tells me with a straight face that he's going to "discharge the static on the line" and that it can take up to 5 minutes.

He's going to discharge the static on the line... by pushing a button on his screen. Yeah... and next he's going to blame solar flares for the outage.

Someone's been reading too much BOFH me thinks.

Finally I was given a ticket. Woo hoo! It's like winning the lottery. I have a ticket! An hour later a Tier 2 technician calls me.

So I walk through the basics with him and he wants me to read off the configuration from the router. I don't have the login/password so he gives me both of the default administrative logins for the router.

::grin::

No, I didn't write them down. They were at least not dictionary based and I'm not that evil. I did find it disconcerting that it was so easy to get them, however.

All is correct - except DNS entries - so I changed them and restarted the router.

Now I've already used our secondary connection (provided by someone else, of course) to trace traffic all the way to .. Milwaukee. The last hop between Milwaukee and Green Bay is the problem. All our equipment is functioning within normal parameters, but either some router in Milwaukee has decided to take the day off or their default router has gone bye-bye. Either way, it isn't our configuration, which hasn't change in a year or more and was working just fine yesterday. But he wants me to try to ping Google from the router.

By name.

Now I have no connection. I cannot ping the default gateway on their network and simple IP routing basics says if I can't route traffic through the gateway I sure as hell can't resolve Google's IP address.

But he insists I try. Because obviously the fact that packets are not getting from our router to the default gateway is all a DNS configuration issue. Yeah, I've seen that a million times. So I do and gee, golly, look at that - I can't resolve the address.

::sigh::

Eventually he tells me he'll do some more digging and get back to me and finally, about 1pm this afternoon, the connection is restored.

I'm still floored by the "discharging static on the line" line. I keep thinking maybe there's something here I'm missing, but I keep hearing Simon's voice in the background snickering because he's got my luser name now ....

Comment on this blog entry

What's Going On?
Posted By Lori MacVittie at 12:09 PM

This week we managed to successfully integrate WebSphere 5.0 with our AD server, which was a cool thing to do. Softerra's LDAP Browser is a wonderful thing, mainly because it let us quickly browse AD and figure out what the DN should be ;-) Finding a user to bind with, however, was another story, but we managed to do that as well and soon WebSphere was happily authenticating us against LDAP.

We're gearing up to put Web Services platforms to the test and, in the midst of all that fun, beginning to prepare ourselves to migrate from AD and Exchange 2000 to AD and Exchange 2003, because it's about time we did and because we want to consolidate the servers. Right now we're using 2 and we're going to put both AD and Exchange on the same machine to save some space and hopefully reduce some of the heat issues that still crop up from time to time.

We've also got a ton of storage stuff in the lab at the moment. We've got more TB of space available than we have a right to have. Of course most of it isn't yet networked and what is networked isn't ready for the NWC Inc. network. I suspect it'll be ready soon and that's a good thing because the Spanstor keeps yelling at us about being low on disk space. I've been deleting backup images from the first half of the year in an attempt to make it happy but apparently I'm not deleting fast enough.

I really want to just "rm -f *" but I've been warned against that and for the moment I'm doing things by the book, even though it is a painfully slow method of managing files.

Comment on this blog entry

Deleting a Log File
Posted By Lori MacVittie at 01:57 PM

You wouldn't think that deleting a log file would take so much thought, but it has.

After doing some testing that resulting in a 2GB + error file for NWC Inc.'s web site I decided to delete it.

DENIED. Because Apache was running, I couldn't.

Well, I didn't want to stop the processes just so I could delete the file - all those widget orders we'd lose would be a problem for the business folks, right?

But what else can you do?

;-) Share the logs directory, mount it under Linux and delete the file.

Where there's a will, there's a way.

Share your solution to the problem - the only requirement is that you can't stop Apache from running.

Comment on this blog entry

Innovation
Posted By Lori MacVittie at 03:40 PM

Innovation isn't just for high tech. Remember last year when our air conditioning died because of the cottonwood mess? Well, our friendly management company's handy man came up with an ingenious plan to ensure we didn't have a similar cottonwood party at 3am this year.

By placing window screens around the unit outside - screens with a very tight mesh - he's managed to keep the unit from becoming clogged with cottonwood and overloading itself.

Three cheers for ingenuity!

Comment on this blog entry

IPCop
Posted By Lori MacVittie at 11:05 AM

For the first time in 239 days we had to reboot IPCop. :-( I know it's sad, but sometime yesterday the poor little box that could stopped responding to requests and just couldn't take the traffic anymore.

But a quick reboot and all is well again. We'll see if that little workhorse can't last 240 days this time.

What's new, you ask? Well we'll be revisiting Web Services platforms again this summer and this time, we plan on keeping the winner and offering the ability to purchase our widgets via Web Services.

If you're interested in some nice clients that are pretty darn easy to use for checking out Web Services, check out:

CapeClear WebServices Tester (Java)
or
.NET WebServices Studio (.NET)

Both work with .NET and J2EE Web Services, but the former is a Java application and the latter a .NET application and it's always nice to have both to test interoperability. I've been using both as of late and am happy to say that both appear to work pretty darn well.

Comment on this blog entry

All Systems Clear
Posted By Lori MacVittie at 08:38 AM

Aside from having to come in really early this morning to deal with the power outage, all went well this morning.

Power was out for around 15 minutes, but the APC UPS kept NWC Inc. chugging along as though nothing had happened.

Of course when the power guys came in and asked, "Why aren't the lights on down here?" I almost bit one of them in the ankle. I mean, aren't they supposed to know why?

Sheesh! They fixed the problem right away and all is just cheesy now.

Comment on this blog entry

UPS Testing
Posted By Lori MacVittie at 08:46 PM

Well, it's been a long spring, but we're back.

Next Thursday we're going to test the UPS. Not because we want to, mind you, but because the local utility is going to be installing their automated meter reading solution and needs to cut power for, oh, say 10-15 minutes.

We'll see if our UPS lasts that long. Right now the load is pretty high because we've got all the test machines running for Don's mid-tier EAI review. We'll be keeping the winner of this one to add to our infrastructure, so it's a fairly exciting time.

Comment on this blog entry

Time Flies...
Posted By Lori MacVittie at 09:26 AM

It's already February and we haven't updated in a while, but we haven't been sitting idle.

NWC Inc. is running well with its new framework and is chugging along just fine. I'm working on a review of Enterprise Portal Systems right now within the infrastructure and that, too, is plugging along nicely.

Well, it has been going nicely until recently when everything seemed to fail during installation, but that's one of the things we have to do - struggle through and find the glitches so you don't have to.

Web Services Management software will be coming along soon, as well as mid-tier EAI solutions, which we'll be testing for use with NWC Inc. We'll see if we can once again get our shipping system hooked up to our order system.

I also had to restart the NWC Inc. Network camera - the logs show a lot of traffic that appears to be reminiscent of NIMDA. Not hurting the Axis, of course, since it's Linux based :-) but it is pushing a lot of traffic through the pipe.

Comment on this blog entry

Widgets for Christmas
Posted By Lori MacVittie at 10:31 AM

Yes, I know it's been a while but we've been really busy taking everything apart and putting it back together again.

Finally, we're back up. Still have a couple minor tasks to complete, but you can order those Widgets for Christmas.

I moved to JSP and eliminated the app server for the time being. We'll get back there eventually.

Other fascinating events have had the lab at 85 degrees as a piece of insulation in the air conditioner caused the unit to ice up and stop functioning. Several weeks ago we were freezing at 62 degrees because someone had bumped the thermostat. Our climate controls leave much to be desired some days.

Comment on this blog entry

Back in the saddle again
Posted By Lori MacVittie at 03:25 PM

Sean Doherty is working on a review and taking advantage of our infrastructure. Sometime yesterday we lost all connectivity between our Cisco router and our provider's CPE. Turns out the router was running at 99% CPU utilization and was dropping packets faster than a well configured firewall fending off MSBlast.

After trying a few tricks, which did not help, I pulled every connection from the switch hoping to use process of elimination to find out where the errant packets were coming from. Everything went well until I plugged in our CRM machine.

Wham! The router became unresponsive and packets starting dropping. I unplugged it and all was normal after the queues cleared out.

So the CRM machine remains unplugged until we can determine what odd behavior was causing all this strange traffic and freaking out the Cisco router.

It sat at 30% CPU utilization until after we rebooted it, then returned to normal. No odd processes were discovered running - though that isn't a certificate of a clean bill of health.

Comment on this blog entry

I hate CLI0621E
Posted By Lori MacVittie at 03:02 PM

This error is the bane of my existence. I hate it. With a passion.

I'm in the process of upgrading to WAS 5.0 and of course something had to go wrong.

The application deployed beautifully, all looks well and then ...

CLI0621E

Yes, the "unsupported JDBC configuration" error. The one that ostensibly can be fixed by running db2jstrt on the JDBC server. But since that'd already been run on the server once to fix this problem before, I'm not really sure what to do - other than rewrite the entire application using Type 4 drivers.

CLI0621E

I swear that if Edgar Allen Poe were a programmer he would have written the "Telltale Error: CLI0621E"

Comment on this blog entry

It just isn't my month...
Posted By Lori MacVittie at 01:21 PM

Well, the fixes are not fixing things.

I've been able to confirm with IBM that the specific problem that has cropped up is indemic to WebSphere 3.5 and 4.0.1.

WebSphere 5.0 does not have this problem, so we'll be migrating as soon as possible to WAS 5.0. I've been playing with WebSphere Studio and running our code base on a test 5.0 server with success, so I don't foresee any huge issues.

Stay tuned to see if I'm psychic or not.

Comment on this blog entry

The Continuing Saga ...
Posted By Lori MacVittie at 07:53 AM

Well, it's still freaky.

So now we're using WebSphere's clustering to see if we can keep consistently alive.

What a week.

Comment on this blog entry

Still acting weird
Posted By Lori MacVittie at 07:56 PM

When WebSphere freaks out, it's because of an exception caused by: com.ibm.ejs.sm.active.ActiveEJBServerProcess.

Apparently it's unable to ping the default server and decides it's time to restart it.

I've got two different machines installed with WebSphere and both exhibit the same behavior. Research on the web shows at least two other people have had this exact same issue crop out of nowhere, one on WAS 3.5 and one on WAS 4.0.2. We're running WAS 4.0.1.

Unfortunately, neither questions have answers.

I've adjusted the ping/default time out settings for both servers and will continue to watch them to see if this solves the problem. One post seemed to indicate that this had worked for them on one server, but not another. We'll see what happens.

Comment on this blog entry

I fixed it.
Posted By Lori MacVittie at 05:53 PM

Well, I think. The control database in DB2 had tablespaces in an inconsistent state. A shutdown and reboot caused it to sync up and it's happy again. For now.

Time to do some reading and figure out why DB2 is not happy. So we're up.

For now.

Comment on this blog entry

I broke it.
Posted By Lori MacVittie at 12:16 PM

Or maybe Bruce's testing broke it. Or maybe a network gremlin came along and decided to be mean and break it cause they thought we needed even more excitement.

Regardless of who's to blame, NWC Inc. is broken. Not everything, just the app server. It's been acting weird for a while and we haven't been able to diagnose the problem. I started a fresh install on a nice shiny clean OS and guess what - same error. Apparently I need to tell WebSphere to pretend that the local DB2 database used as a repository is a remote instance, otherwise no more than 10 connections to the database can be made.

That sounds funky, but that's what the error represents, according to IBM's web site.

Unfortunately, following the directions to fix it isn't working either.

So while I'm trying to figure it out, NWC Inc. transactions aren't working. The web services are working, but nothing that involves WebSphere is working right.

Comment on this blog entry

So if you use the right port ...
Posted By Lori MacVittie at 06:13 PM

I have forever had problems getting a TCP/IP connection to DB2 to work correctly. It just never works so I've been using named pipes for remote connectivity.

Today I was configuring remote access to DB2 for some product testing and happened to notice that there was an extra 0 in the TCP/IP port.

Instead of 5000, it was running on 50,000. Gee, when you use the right port, things work better.

If you've seen visited our Axis Network Camera lately you may notice that we've cleaned up a bit. We had visitors to NWC Inc. to show off what we've been doing and, of course, had to make things look nice for our auspicious guests.

Steven's finally finished with his last big review and we've had our lab rat shipping out products for a week. Now that Steve has some time, he's been working on our NAS and getting our automated backup systems in place.

The scary part - testing those backups - is still to come....

Comment on this blog entry

Long time no update...
Posted By Lori MacVittie at 02:03 PM

Apologies for the down time - there hasn't been a lot going on with NWC Inc. specifically, except for some testing until today.

Bruce Boardman is currently in the throes of a review of Web SLA managers and it beating hard upon the NWC Inc. infrastructure. I am working on another review of products within the infrastructure and, of course, that means that things have gone wrong.

I spent a good portion of the morning rebooting the application server and removing code from the PHP based front end to stop the calls to our .NET application that handles shipping updates. The machine must have been rebooted at some point, because the ASP.NET State Service was not started and when I tried to start it, it errored out on "bad login".

But it's never been touched! It was installed and configured by Microsoft's installation program - why did it fail now? I fixed the login information and started the service but alas and alack - the .NET apps still aren't running and, quite frankly, the level of debug information available is about as helpful as a hot cup of tea at noon on a mid-summer's day in Texas.

But the rest of NWC Inc. is up and running happily again, without interacting with the .NET applications. Since Steven and I are off to lovely Syracuse, NY to meet with our fellow Network Computing staffers this week, the .NET app will have to wait until we return.

If you're bored while we're gone, take a trip over to Network Computing's site and follow the links to Bruce Boardman's review - you can help us out and possibly win a cool toy from 3Com.

What are you waiting for? Go go go!

Comment on this blog entry

Happy Independance Day!
Posted By Tom LaSusa at 08:48 AM

We will be taking a few days off to enjoy the holiday, Lori is already on her way to Wisconsin's Northwoods for some camping and fishing relaxation. I will be spending some time with my wife and my parents this weekend and relaxing in the pool.

On behalf of Lori, myself, and the entire Network Computing Staff, we wish you a great Independance Day weekend! Stay safe and have fun! We'll be back next week.

Comment on this blog entry

Bad Searches
Posted By Lori MacVittie at 06:17 PM

Thanks to Gary Pultz for pointing out that a blank search in NWC's "Catalog-o-widgets" returned a terrible faux pas page, complete with a Java stack trace.

The fix was minor, so I ran on out and fixed it up.

In other news, our heating/cooling seems to have leveled off. The termperature is constantly hovering around 74 degrees, which makes our hardware (and Steve) happy.

Comment on this blog entry

Evening from Hell
Posted By Tom LaSusa at 11:22 AM

The Netbotz environmental monitoring in the lab began to send me email yesterday afternoon, telling me that the lab was up to about 95 degrees. Oh, boy. I was busy, so I called Lori who when down there and shut most of the lab down. I should have had her check the air conditioning condenser, because I started getting emails again around eight oclock. I ran down to the lab at about 11:30 at night. The air conditioner was dead as a doornail. I got our HVAC vendor, J&D down to take a look at it, but the unit had blown a breaker. Turns out the breaker was in a locked room in our building I don't have access too. After about an hour of calling, Joyce from KOS Realty came by and unlocked it for us.

Turns out the cottonwood had completely clogged the air conditioner, causing it to overload and blow the breaker. Joyce had to run to one of their apartment complexes and get us a 100ft hose to hook up to the nearest spigot. Jim from J&D then cleaned and serviced the unit. By morning the lab was back to about 72 degrees and still dropping. It was three in the morning before I got home. Gah.

I will update you guys if there is anymore facilities fun to be had. Take care.

Comment on this blog entry

New look
Posted By Lori MacVittie at 02:29 PM

Yesterday I was afflicted with some strange malady affecting my back, so I am forced (forced, I tell you!) to lay comfortably on the couch while everyone else is out fishing or at the park enjoying the sunny afternoon.

So I redesigned nwc inc.'s site. While doing so, I changed a few things that some of you mentioned you didn't like about the site, and hopefully it's a bit more interesting and not so dull.

Comment on this blog entry

Everything is vulnerable
Posted By Lori MacVittie at 09:17 AM

It seems everything electronic these days is vulnerable to exploitation. And even though I'm on vacation, such vulnerabilities can't be ignored.

Thanks to Tim Heagarty CISSP, MCSE for pointing out that our new Axis Network Camera was vulnerable in a nasty way. Seems that you could get around the HTTP Basic authentication and do things like change the administrative password on us. But thanks to Tim, I patched the Axis last night and all seems well.

A new firmware upgrade was called for, and I have to say that Axis makes this a simple process. Simply ftp the new firmware to the device and, if you give it the right name, it automatically upgrades itself. Sweet.

Comment on this blog entry

Back on track
Posted By Lori MacVittie at 03:32 PM

Well, reinstalling fixed my authentication problem with the CRM app. I was able to turn IIS authentication back on and viola! Instant sign-on. Phew!

There is a key in the registry that appears to turn this on and off, but fiddling with it didn't appear to affect the application, which seems odd. Of course, I didn't reboot in between so you never know.

Back to coding now - have to get some more Web services running for a coming-quicker-than-you-think review.

Comment on this blog entry

Java and blank passwords...
Posted By Lori MacVittie at 04:54 PM

is a bad thing.

For the second time in a one week period I have been reminded that Java does not like blank passwords passed via the command line. Two EAI products have now choked on this and I have been forced to give the user a password.

Yes yes, I know we should have had a password on the account in the first place, but it's a lab and we didn't. Now we do. Because Java can't handle blank ones.

I also accidently locked myself out of our CRM application. I told it to use IIS authentication and WHAM! I can't get into it. At all. I perused the database to try and find the setting and change it back, but it doesn't appear to be there. Next week it's time to look in the registry - wheeeeee!

Comment on this blog entry

OOPS
Posted By Lori MacVittie at 07:00 PM

No, I'm not talking about the Octopus, Oyster, Prawn and Scallop stew I had at the Circo Restaurant at N+I. I deployed the CustomerLookup web service before I left for Interop, but it was an older version and it turns out that it wasn't working for the last couple weeks. I've now redeployed and all is well again. Version control is something that was always a pain, but definitely worth it when things like that happen.

Comment on this blog entry

Exchange. Again.
Posted By Lori MacVittie at 01:38 PM

Yes, you guessed it. Our exchange server puked while Steve and I were in Las Vegas for Networld+Interop so you know what I did all morning.

Fix exchange. It was evident that I needed to move the MTADATA directory to another - much larger - drive. So I googled the issue and found the following KB Article at Microsoft: XCON: How to Change the Location of the MTA Database and Run Directory. Fair warning, it does involve registry editing but the good news is that it's only 2 entries and it's fairly painless.

The pain begins later. I scrupulously followed the instructions and then tried to mount the primary message store. Nope. Not gonna happen. Apparently I needed even MORE disk space. So I moved the log files ( MDBDATA ) to the bigger drive and then crossed my fingers. Wham! Message stores mounted and Exchange appears to be happily running without complaint. We'll see if this fixes the problem in, oh, about 2 days.

I also spent time installing W2K and SQL Server 2000 on one of our Dell 2650 servers in preparation to install the winner of our CRM review. Yes, I know who it is, but I'm not telling. Yet.

Comment on this blog entry

Just when you thought it couldn't get any more real...
Posted By Lori MacVittie at 11:00 AM

It does.

Our Axis Network Camera is rolling live and you can visit any time you like. We've had some visitors already. How do we know? Because the Axis let's you pan and zoom around the lab and when it's moving it catches our attention. Check it out next week and have some fun!

Awful nice of me, isn't it? Especially since both Steve and I will be at Networld+Interop and the lab should be dark while we're gone. :-) But don't worry, I'll be back after N+I and working on setting up our permanent CRM server. I hope that by the time the review prints we'll have access available for all you Widget loving readers. But the lights will be back on and you'll have more luck playing with the camera then. We'll also be having a lot of action with Don MacVittie's upcoming EAI review. He'll be in the lab testing some solutions and determining which one will be a fit for NWC Inc.

See ya after Interop!

Comment on this blog entry

The Good, The Bad and the way too real
Posted By Lori MacVittie at 06:27 AM

Exchange stopped working again. I thought perhaps it was connectivity issues with our ADS server, so I did what any good MS network admin does: I rebooted the ADS server, waited five minutes, and then rebooted the mail server.

It didn't help. With the traditional fix failing, I started digging a bit more. The MTA stack service was failing because .... there was less than 10MB disk space on the drive where the MTADATA directory resides. Ah ha! So let's make some room. Start by deleting the 3GB Exchange server install log file. Clean up the temp directories, empty the recycle bin. Still not enough room. After twenty minutes of removing everything that wasn't absolutely necessary (who needs Media Player on a server??) and moving the IIS root directories to a larger drive, the MTA stack service started and Exchange went "Wow! I can run." Note to self: reinstall Exchange on a disk with more space.

Our Axis network video camera came in late yesterday afternoon. All I can say is, "Wow." Talk about clarity and control. Easy only begins to describe how quick the installation was. Being completely web controlled helps, of course. We set it up (you can't have access just yet) and I tested it out remotely. Over GBE the streaming video was unbelievably high quality, but would it be as good over my 768kbps DSL? Not quite, but it was certainly head and shoulders above other network video I've seen. Control was less smooth, of course, but panning and zooming worked fine and I was able to zoom in and read some notes I'd left on the table in the lab. I'll get it configured shortly and get a link up to it so you can play around with it. It's definitely one amazing little camera.

Comment on this blog entry

In the midst of a review...
Posted By Lori MacVittie at 07:13 PM

We're well under way of our review of CRM products for NWC Inc. We're feeling the pain of integration and scrutinizing each product as we know you'll be interacting with it eventually. The biggest problem seems to be that the only remote connection our DB2 database will take is either via JDBC through code or an ODBC connection - using named pipes. TCP/IP communications are failing. I'm not sure why, it looks like it's configured correctly. Something else to dig into after this review is finished.

In other news, we're about to receive a new network video camera from Axis Communications. We'll see if we can strongarm Brad into putting up a link to it so you can see more of the lab.

Comment on this blog entry

Confucius say ...
Posted By Lori MacVittie at 07:47 AM

if you want DNS query results to be returned you must allow them to pass through the router. Trying to keep down the number of extended ACLs on our Cisco router is becoming difficult. But I believe this was the last one we needed to make everything work right.

All the disks are imaged for the CRM test, and we've set up one of our new Dell 2650's to run Oracle on RH Linux for our upcoming review of EAI solutions, in which we're going to improve upon our simulation infrastructure by integrating more feeds and applications together.

Comment on this blog entry

Interference
Posted By Lori MacVittie at 04:04 PM

If you've been following along you'll know that we recently moved our infrastructure over to our Cisco switches/routers.

One of the things we've been doing is implementing our management network - hooking into the APC UPS and getting all the management software to remotely administer the machines if necessary. So that means each machine has 3 network connections: a primary, a secondary for failover and the management network. Well, the management network was set up but not completely configured. The interfaces are trying to DHCP and we don't have a DHCP server on that management network. So the interfaces - a PPP connection - ends up with the default "I can't get an address" address.

This was causing headaches. For some reason, our DNS server decided that this "no address" address was its primary connection and was returning it in response to queries for it IP address. Since this is also our ADS server, you can imagine the problems on the network just from that. Then to add to the problem our mail server did the same thing, putting both of them on the same address and wreaking complete havoc throughout our network.

I shut down the management interfaces and rebooted (cause rebooting Windows is the best fix for problems) and voila! All is well again.

Comment on this blog entry

Next up: CRM
Posted By Lori MacVittie at 08:01 PM

Having finished up testing Web Services platforms now we're getting ready to test CRM solutions for NWC Inc.
This is a cool review, because we'll be keeping the winner of this review as our CRM solution.

We've been imaging our disks to get ready for the review as well as unboxing our six new servers that will serve as additional infrastructure for the lab.

Our DNS server was having problems resolving addresses, and the issue turned out to be an issue with an access list on our Cisco router. After a frustrating half hour of checking and rechecking configurations, we finally had connectivity back and were able to resolve addresses again.

Then, because we hadn't had enough challenges in the last few days, we had to find out why our NetBotz was spewing e-mail to a nonexistent user @nwc.com. After finally remembering what the uname/password was for the device, we were able to ascertain that a single field was misconfigured with a bad user name. We changed that, but found that the device was still spewing forth an unacceptable number of alerts - all of them notifying us that a set of external sensors were not responding. Probably because they don't exist. :-) So Steve changed those settings and finally our NetBotz is back online and acting normally again.

Comment on this blog entry

Hot Time in the Old Town
Posted By Tom LaSusa at 06:36 AM

Well, we had a heck of a day yesterday.

Temperatures in the NWC Inc. lab got all the way up to a high of 86 degrees when I opened the door in the morning. I shut down most of the equipment in the Green Bay Real World Lab section of our facility, and got on the phone with our realty management company. A call to the air conditioning repair guy yielded a friendly service tech in about an hour.

It appears that our air conditioning unit was low on freon, and had iced up. Iced up to the tune of three or four inches of ice. Our service tech, Tom, used the patened Smoke Wrench (a propane torch) to melt most of the ice. Then he recharged the freon, and we were good to go. No more Data Center Sauna for us.

This morning things seems to be working fine. I will keep you posted if anything changes.

Comment on this blog entry

Syslogging away
Posted By Lori MacVittie at 02:12 PM

We decided to set up a dedicated syslog server to aggregate messages from the routers and switches, so after a load of RedHat onto a litlte 1U server and the editing of a few configuration files our syslog server is up and running and doing its assigned duties.

Our addition of SQL server went well - of course, all we did was install the thing so far. We're holding off on actually replicating data for a month or so - we've got an upcoming review that it just so happens will take care of that chore in real time for us. ;-)

Comment on this blog entry

Back up and running
Posted By Lori MacVittie at 09:02 AM

We are back up and ready to take your orders again!

Comment on this blog entry

Woo hoo!
Posted By Lori MacVittie at 06:50 PM

I just happened to ping one of our class C's the other day out of frustration and guess what? Something responded. Hmmm...

Instead of adding a route, dudes at our ISP just assigned one to the DSL router. Without mentioning it to us. Nice.

After an hour spent reconfiguring our router, voila! We are live. And after two days of ripping out cables and putting in new ones we are almost ready to go. We're paying careful attention to cable management this time. We've even gone so far as to color code them - blue for primary network connections, another color for secondary, green for management and another for our IP based KVM.

We'll also be adding some databases to our infrastructure. While we're running as close to a real world production environment as possible, there are some things we require in order to support our editorial goals. DB2 is working fine, but we need to be able to test on multiple databases, so we'll be adding SQL Server shortly and, hopefully, one more RDBMS in the next few weeks. We'll replicate data between them to synchronize our corporate data and this should offer a bit more flexibility on the editorial side as well as better representing the real enterprise, which often runs at least 2, if not 3 or more, databases.

So cross your fingers - if all goes well tomorrow we'll be live again by the end of the day.

Comment on this blog entry

Frustrations
Posted By Lori MacVittie at 12:38 PM

The routers are racked.

The switches are racked.

The configurations are ready.

The Class C is routed. I think.

I'm sure you're familiar with CPE that is managed solely by the provider. It's a commonplace requirement for many broadband offerings. The ISP that supplies our 1.5Mbps SDSL does not allow us to manage
the router that sits in our lab. Now, that wouldn't be so bad if we weren't trying to figure out why our Class C's stop at their router. I'm guessing that there is no route to our router, but I can't verify that because I'm not allowed to touch the thing. Can't get in, can't even view the configuration in a read only mode. So we wait. And wait. And wait, until someone has time to check it out.

Of course life would be much easier if we just used the measly /29 that we got with our service, but we want our IPs. And we want them now. For two days I've sat trying to figure out what's going on, without being able to do much about it.

It's frustrating, to say the least. What really bothers me is that my business class DSL @ home, from the local Baby Bell, is mine. I can manage it, change it, screw it up - whatever I want. And that's at home - the place where the provider probably should be exercising a bit more control over their equipment and the connection, if for no other reason than to reduce the technical support costs. But because it's business class it's assumed that I can manage it myself. Thanks for that trust - it's a warm fuzzy that I don't get from the ISP that supplies our lab.

Unfortunately, we're stuck until dudes figure it out. Which I hope will be soon.

Comment on this blog entry

Register Away
Posted By Lori MacVittie at 03:20 PM

I just couldn't let it sit.

You can now register as a customer of NWC Inc.

I'm going to tell you now - do NOT enter real credit card information. Make it up. We don't want real info in those fields.

You can enter as much other "real" info as you like. If you'd like to be able to interact with NWC Inc. as we move forward, you might want to consider using an e-mail address that you periodically check out, cause we'll be doing some neat things soon that will hopefully get you involved. But remember - you don't have to enter anything real and we won't sell or use the info in any other way if that's your choice. We're assuming if you put a real email in there that you want to be involved. So if you don't want to be involved, don't use a real e-mail address.

Thanks - and as always, your feedback and comments are welcome and appreciated!

Comment on this blog entry

Place your orders, please
Posted By Lori MacVittie at 02:07 PM

Well finally. I managed to get back into the swing of things and get the order process complete. At least from an order entry standpoint.

You also get to choose a delivery method:

NWC Standard (guy on a bike) $5.95
NWC Second day (guy in a car) $15.95
NWC Next day (guy on a plane) $25.95

Because if we're selling 'em, someone has to deliver them.

On a development note, when I create a new Servlet via WebSphere Studio, I'd expect that it extend HttpServlet by default. It doesn't. It extends whatever class I had selected. This caused no end of trouble. I kept seeing a "Class Cast Exception". Finally it occurred to me to actually look at the class definition. WebSphere AS was right - it wasn't a servlet class. Now it is and it's working just fine.

Comment on this blog entry

Simulations
Posted By Lori MacVittie at 04:31 PM

Our base simulations are now running. We're receiving orders and getting new inventory on a daily basis.

An average of 720 transactions per day will be occuring. Right now they aren't scheduled around any specific time of day. A cron job is running a
php script that calls a Java servlet that randomly chooses a customer and then creates an order for that customer - randomly, of course.

Our mail simulation is not up and running yet, as we had to replace some hardware in the simulation servers recently and haven't downloaded and
installed LoadSim once again - but soon!

Comment on this blog entry

Switches!
Posted By Tom LaSusa at 01:28 PM

We now have our new Cisco stuff and things are very cool now. Lori should be getting around to configuring that stuff soon and we will get the cabling done soon. We also got 12 more p4 workstations for the lab and we will be setting them up soon. The