Best IT Blunders I’ve Witnessed.

After 10+ years performing system and network administration, I’ve seen a lot of things. But just when you think you’ve seen it all, something else happens that leaves you shaking your head. Here’s a short list of those goofy things:

Episode 1, The Self-Perpetuated Hi Virus

My old employer, CIENA, caught the W32.Goner.A@mm visual basic virus in December 2002 and it ran rampant. The virus is distinguishable by the subject line of simply “Hi” and some text in the body. One of the senior Exchange administrators in Maryland decides he is going to perform damage control by running the exmerge utility to delete all items with the keyword of “Hi” to prevent people from activating the virus.

In concept, a good idea, but it’s not uncommon to receive email with that subject line; even my wife has used that one. The real stinger is that he ran it as a wild card, which deleted every item in the mailbox with the two-letter combination of “Hi”. Yep, every email, contact, task, and calendar item in every folder of every mailbox with that two-letter combination was deleted. “machine, Phil, thing, Chiang, Hilton” etc., were all permanently deleted, some right while people were looking at them. Needless to say, they had to restore everything, including the virus, and halt the virus from spreading again.

Episode 2, The Other Virus Incident

About one year before the previous blunder, a virus broke out CIENA. By the time I was notified, someone in Maryland decided the best plan of action to stop the virus from spreading was to delete the mailbox. No kidding, they deleted the mailbox while we were in the middle of controlling the virus. One of the user support guys was able to restore from his .ost file, otherwise we would have been digging into the tapes.

Episode 3, Oops, I Mirrored the Wrong Drive

I went to a client late one night to upgrade his clustered NetWare server from v3.11 to v4.10. We first did a backup and inventoried all the hardware to make detailed notes of all the device names and locations. The upgrade process (if I remember correctly) is to upgrade one drive on one server, then mirror the drives, and then mirror the servers. Still not sure exactly how it happened, but within 3 seconds of hitting the final Return key to start the final mirroring process, I realized the drives were mirroring the wrong direction. In other words, the blank updated drives were overwriting the updated drives with the data, and everything got wiped.

It was one of my most embarrassing moments, although the only big mistake I’ve had to date (knock on wood). We finished the upgrade and then luckily restored from backup.

Episode 4, Command.com 

Nope, not a dot bomb, but an episode that occurred with a CNE in training back in the early 90’s. Getting a CNE was in vogue and everyone was doing it, but this one guy shouldn’t have bothered. We hired a CNE candidate from Masters Institute for a few days to do some basic work in the shop figuring we’d hire him permanently if he worked out. After a few days of so-so work, we assigned him the task of building a NetWare v3.11 server, something that I could do in less than 90 minutes. About 4 hours later, he calls me over to ask me some questions and the dialog goes something like this:

Me: What’s up?
Him: Well, I’ve made some progress, but I still can’t get the server to run.
Me: What does it do?
Him: It starts to load, but then abends. I think a file is corrupted.
Me: Really?!? Try a different version.
Him: I have, but it didn’t make a difference. Here, let me show you.
[he types a few keys, brings up a file in Edit]
Him: Check it out. This file is all gibberish, looks like it was completely munched.
Me: Um, you probably want to Exit and don’t save….

The file he showed me? It was command.com, the binary boot file for DOS and Windows. One class away from his CNE and he didn’t know what that file was!