The Case of the Backup Lemon

Lemon We’ve all had it happen. Start a new sysadmin or IT job and you inherit some really horrible decisions made in the past. So was the case with me, a new Security & System Admin for a small marketing company. One of the first things my new boss tasked me with was managing the backup system. Assuming it would be relatively simple, I delved into learning about it. The unit was a $14k hardware/software package built by a vendor who shall remain nameless. Called the Data Protection Unit, it shipped with over 1.5 TB of space. So far so good, right?

I soon found to my horror that the software that came with the hardware was the worst backup app I’d ever seen. It ran on Red Hat 2.x.x, ancient considering RHLE 5 was recently released. Not only that, it used Winsock of all things to connect to Windows clients, which seemed archaic and I wondered how old their developers were. I pictured wizened old coders sitting in front of puke green terminal screens jockeying for time on the mainframe and alpha testing on Novell.

Still, it was actively backing up the then Windows 2000 office network and some of the Linux production network just fine. To make a long story short, this box never produced the same problem consistently. It was always something new; restores would be problematic, 30 GBs of a single disk took THREE hours to do a master backup, not to mention their support department was a joke. This system was also agent-based, so I had to throw a cludgy, slow, similarly-coded program on all machines. I marveled at the brainpower behind spending $14k for it. Throw on top of that a very non-intuitive interface and the whole thing made me want to recreate an Office Space scene complete with the rap music and baseball bat. The closest thing to a secluded field downtown was the baseball stadium across the street though, so I decided against that. The worst part was, we had nothing in the budget for a new system from a different vendor, so I was stuck with what I had for the time being.

For months I wrestled with getting the thing working reliably enough to backup everything I needed, mentally counting the days I didn’t have backups for. For months I went in circles with their support engineers, letting them remote in and fix whatever new problem arose. In the end it always seemed like a band-aid was put on there until the next thing went wrong. I would repeatedly ask them to email me as I was often not at my desk, and they would call me anyway creating a vicious cycle of phone tag that could eat up an entire business week. I began to complain to both the local rep that sold us the unit, and to supervisors higher up in the support department. After quite some time, I delivered them an ultimatum outlining what we would accept as solutions to this ongoing nightmare which included a full return of the unit, a swap, or a refund. They wouldn’t even provide me with any copy of their return policies no matter how often I asked. I finally received a call from one of their sales executives, touting how they had other clients backing up TBs of space with no problems using the same unit. He finally decided he was bringing himself and one of their best engineers to us to check everything out themselves.

The day went well, and everything was amicable. We showed them our small infrastructure, outlined the network a little, and then I sat down with the engineer to go over some things on the system. At the end of the day was a short meeting in my bosses’ office with all the players, including the local sales rep. In her defense, she was an independent, and not affiliated with the backup system vendors and pretty much had more to lose herself. The VP and my boss did most of the talking, which was quite a lot. Sort of expected out of the sales pukes, no? Anyway, my boss brought up the fact that the unit could very well be a “lemon” and the idea of a new unit was bounced around. The sales exec wasn’t opposed to the idea. Everyone smiled, shook hands, and that was the end of the field trip. The only downside to the whole meeting? Our return options were “limited” we were told, since the purchase was more than a year ago. Now comes the shocker.

The next morning, I sit at my desk and discover the backups all failed.

At that point, another round began with their support department. Their verdict; make sure the swappable drives were properly seated. If they were fine a technician would come on site to make sure nothing was loose internally. This was a system that hadn’t moved in at least 6 months or more, that ran faithfully as the hardware never crashed. To it’s credit the system itself never experienced any kinds of shutdown failure. The drives themselves hadn’t been moved in months until I reseated them as they requested.

At that point we decided that was enough and didn’t renew the support contract, especially when our request to swap for a different unit was completely ignored. We ceased communication with them, and to this day their support has never followed up with us.

I think Shakespeare once said “Oh ye, so sad and comical.” but maybe not. But now what the hell was I going to do for a backup system? I certainly didn’t want to use that unit. On the Windows side I did the best I could with the native NT Backup utility by writing them daily to a separate file server for a while. I eventually came across what has saved us. It allows me to backup all of the machines in my mixed environment of Macs, Linux, and Windows and won’t cost me anything. And guess what boys and girls…it’s open source!

To be continued in A Girl Named Amanda…

Posted in nix
Leave a Reply

Leave a Reply

Your email address will not be published. Required fields are marked *