Systems Engineer trapped on Earth...

A Girl Called Amanda

Finally, my second piece of the Case of the Backup Lemon. This one is about a piece of open source software that makes a handy little backup utility on the right equipment. As mentioned in Part 1 of this story,  I inherited a decent machine and a horrible backup application at my new job as a System Admin. Faced with a vendor that had pretty much chucked us to the wind, I did some research and found Amanda by Zmanda. I had a tight budget and the need for critical data backup as well as a viable disaster recovery plan in a reasonable amount of time, it was exactly what I was looking for – It was free and ran on Linux. I had a nice machine with a 1.5 TB RAID array to run it on too w00t! (yes, I’m dropping the word of the year here…it’s all about the page rankings muhahahah.)

My first step was prepping the system, a solid yet older machine with an Intel Celeron 2.5ghz chip, 512MB of RAM, and a 1.5TB RAID 5 SCSI array. I installed Fedora Core 7 and configured it, which is like, way beyond the scope of this document. When it was ready, Zmanda has an excellent tutorial called “The 15 Min Backup Solution” you can check out here http://www.zmanda.com/quick-backup-setup.html .

Following this simple guide, I had Amanda up and running- although it took a lot more than 15 mins. There were a few hurdles, including initial problems with contacting servers in other subnets, adjusting the firewall for the ports Amanda tended to use and a couple of other things. All in all, things one would expect to see when introduced to your own network environment, but the configuration tends to need tweaking when faced with problems. The user forums were a help too when connections between the server and clients kept dropping, which turned out to be a configuration issue.

The initial server configuration is somewhat simple although the config file is fairly large with a ton of options. This can be a bit overwhelming to the novice user, but Nix dogs should have no problems. You can set it to backup to tape or a holding disk, which can be any piece of storage the system can see. In my case I of course used the 1.5TB array. With a small amount of it being used by the Linux OS, I had plenty of room on it.

Configuration of the client was a simple package install, and then setting up configuration parameters. The server and clients both need to be configured with a special amandahosts file and a few regular host files, as well as a few other system and config settings. Following the guide is the best bet to success though. You also setup a disklist on the server, which is a master list of all the servers/directories you want to backup. Backing up other Linux machines works well since Amanda will use the native client installed on the target machine.

When faced with backing up Windows machines you have two options. One is just to share the drive or directory on the Winblowz box and then give a backup service account admin rights over it. This is limiting a bit because it won’t backup open or system files. You could get around that by backing up a Shadow Copy volume though, another thing I’ve been meaning to implement. The other Windows option is a bit more elaborate involving a client and Cygwin. I decided not to bother with that part since a large part of my Windows data was static and I didn’t want to run Cygwin on every Windows system I wanted to backup.

Amanda can be configured to email reports of your backup jobs, so I have it set to send me the daily reports as soon as they’re done in the wee hours, that way when I get to my desk in the morning the report is waiting with all the shininess of a new email message. All that’s needed to run the backups is a cron job on the server that kicks off the amdump program. This allowed me to get some reliable backups on a zero buget, which is what this article has been all about. You can learn a whole lot more over at the Zmanda site, however I’ll soon be sunsetting Amanda possibly due to a new backup system and tape drive in our 2008 budget. It’s one kick ass free backup solution though.

The Case of the Backup Lemon

Lemon We’ve all had it happen. Start a new sysadmin or IT job and you inherit some really horrible decisions made in the past. So was the case with me, a new Security & System Admin for a small marketing company. One of the first things my new boss tasked me with was managing the backup system. Assuming it would be relatively simple, I delved into learning about it. The unit was a $14k hardware/software package built by a vendor who shall remain nameless. Called the Data Protection Unit, it shipped with over 1.5 TB of space. So far so good, right?

I soon found to my horror that the software that came with the hardware was the worst backup app I’d ever seen. It ran on Red Hat 2.x.x, ancient considering RHLE 5 was recently released. Not only that, it used Winsock of all things to connect to Windows clients, which seemed archaic and I wondered how old their developers were. I pictured wizened old coders sitting in front of puke green terminal screens jockeying for time on the mainframe and alpha testing on Novell.

Still, it was actively backing up the then Windows 2000 office network and some of the Linux production network just fine. To make a long story short, this box never produced the same problem consistently. It was always something new; restores would be problematic, 30 GBs of a single disk took THREE hours to do a master backup, not to mention their support department was a joke. This system was also agent-based, so I had to throw a cludgy, slow, similarly-coded program on all machines. I marveled at the brainpower behind spending $14k for it. Throw on top of that a very non-intuitive interface and the whole thing made me want to recreate an Office Space scene complete with the rap music and baseball bat. The closest thing to a secluded field downtown was the baseball stadium across the street though, so I decided against that. The worst part was, we had nothing in the budget for a new system from a different vendor, so I was stuck with what I had for the time being.

For months I wrestled with getting the thing working reliably enough to backup everything I needed, mentally counting the days I didn’t have backups for. For months I went in circles with their support engineers, letting them remote in and fix whatever new problem arose. In the end it always seemed like a band-aid was put on there until the next thing went wrong. I would repeatedly ask them to email me as I was often not at my desk, and they would call me anyway creating a vicious cycle of phone tag that could eat up an entire business week. I began to complain to both the local rep that sold us the unit, and to supervisors higher up in the support department. After quite some time, I delivered them an ultimatum outlining what we would accept as solutions to this ongoing nightmare which included a full return of the unit, a swap, or a refund. They wouldn’t even provide me with any copy of their return policies no matter how often I asked. I finally received a call from one of their sales executives, touting how they had other clients backing up TBs of space with no problems using the same unit. He finally decided he was bringing himself and one of their best engineers to us to check everything out themselves.

The day went well, and everything was amicable. We showed them our small infrastructure, outlined the network a little, and then I sat down with the engineer to go over some things on the system. At the end of the day was a short meeting in my bosses’ office with all the players, including the local sales rep. In her defense, she was an independent, and not affiliated with the backup system vendors and pretty much had more to lose herself. The VP and my boss did most of the talking, which was quite a lot. Sort of expected out of the sales pukes, no? Anyway, my boss brought up the fact that the unit could very well be a “lemon” and the idea of a new unit was bounced around. The sales exec wasn’t opposed to the idea. Everyone smiled, shook hands, and that was the end of the field trip. The only downside to the whole meeting? Our return options were “limited” we were told, since the purchase was more than a year ago. Now comes the shocker.

The next morning, I sit at my desk and discover the backups all failed.

At that point, another round began with their support department. Their verdict; make sure the swappable drives were properly seated. If they were fine a technician would come on site to make sure nothing was loose internally. This was a system that hadn’t moved in at least 6 months or more, that ran faithfully as the hardware never crashed. To it’s credit the system itself never experienced any kinds of shutdown failure. The drives themselves hadn’t been moved in months until I reseated them as they requested.

At that point we decided that was enough and didn’t renew the support contract, especially when our request to swap for a different unit was completely ignored. We ceased communication with them, and to this day their support has never followed up with us.

I think Shakespeare once said “Oh ye, so sad and comical.” but maybe not. But now what the hell was I going to do for a backup system? I certainly didn’t want to use that unit. On the Windows side I did the best I could with the native NT Backup utility by writing them daily to a separate file server for a while. I eventually came across what has saved us. It allows me to backup all of the machines in my mixed environment of Macs, Linux, and Windows and won’t cost me anything. And guess what boys and girls…it’s open source!

To be continued in A Girl Named Amanda…

pure-ftpd

Pure FTP is a nice alternative to the standard VSFTP daemon that comes with many ”Nix flavors. It has the ability to authenticate against mysql, ldap, pam, and the passwd file. You can even chain the authentication methods together to check all or some of them if another fails, this is very nifty if you want a little redundancy in your logins. LDAP down? No problem! It”ll just use the next method you have enabled.

By default all configuration options are run at the command line when starting the server, but you can enable it to use a standard configuration file instead. I found it way easier dealing with the server this way. Setting up authentication with mysql was a snap, but of course required separate configuration of the db server to create the database, table, and users. If you want a quick and easy way of setting up users the regular old pam way will work by just creating them on the system.

Pure FTP is basically like vstfpd on steroids to me. The authentication options are nice and I”ll be giving it a try on my AD server at work to see how it goes. Centralizing the FTP logins by using Active Directory is an interesting idea. All in all, it”s a good alternative to my oft used vsftpd. You can find it at pureftpd.org