Some time last week Ma.gnolia, a social bookmarking site, was hit by the corrupt file system demon. The problem is not only did their database server’s file system go belly up, but their backup server was corrupted by the same demon. Trust me, I do feel sorry for the dudes. It is a sickening feeling having lost all your data including your backups. But…
WTF where you thinking?! Really. The thing about backups is everyone knows they are critically important. It is like your lungs. You don’t sit around and think about your lungs everyday, all day. Yet, if you loose them, it is pretty much game over for you, I was told. Everyone needs them (backups) at some point in time. It has been preached millions of times, yet there are always someone that gets burnt. Why? For the love of the Flying Spaghetti Monster, just make sure your backups are solid!
The big mistake Ma.gnolia made was that it backed up everything on a separate server on the same network with no offsite copies. So if the server room burnt down they would have been fucked anyway! :P Not a solid backup plan.
Here are a couple of things one should keep in mind.
- The number one rule and together with rule #2 the most important: DO NOT TAKE A CHANCE! It is plain and simply looking for trouble if you take a chance doing something hoping nothing goes wrong while not having efficient backups. It is just plain irresponsible. Ask Ma.gnolia.
- Together with rule #2 this is equally important: DO NOT ASSUME ANYTHING! Ma.gnolia probably assumed their backups were okay, since it was stored on a different server. In the Stephen Segal movie Under Siege 2, there was this brilliant quote: “Assumption is the mother of all fuck-ups” How true.
- Have more than one copy of a backup. Backup mediums gets damaged and corrupted sometimes. You won’t necessarily know. Refer to rule #1 and #2.
- ALWAYS and I mean ALWAYS have an offsite copy stored safely away. We at Afrigator store daily backups on 3 different sources. Paranoid? Maybe. We like to think of it as being safe.
- Make sure you backup everything you need to backup. This sound silly, right? Well, what if you have a very specific Apache setup with some complex rewrite rules and you loose it because you didn’t back up your Apache conf files? Oh, right, your a rockstar and can duplicate the setup to a tee, no problem. :P
- Make sure your backups are automated. Relying on people to remember to do your daily backups are going to get you burnt. See, there you go and assume shit again. :P
- Make sure your backups are successful and working. Dumb as it sounds, a backup that isn’t working is a pretty useless backup. Trust me on that one.
This is by no means a complete list of rules. Only the basics. Want free advice? Have a good backup strategy. Please. FFS. If you don’t, you’ll sorely regret it. Just ask Ma.gnolia. :P
Welcome back! You should subscribe to my RSS feed here.
You should follow me on Twitter here
You should follow me on Gatorpeeps here.
Software developer at Afrigator.com
Love Python, do PHP.
David Perel 10:41 am on February 9, 2009 Permalink |
What is good Auto-Backup software Stii?
Richard 11:04 am on February 9, 2009 Permalink |
I agree with David, could you give us insight of how you backup your files and databases and how you’re automated it. Have you tried to take one of those backups online? If so how long does it take?
I’ve heard horrible stories about companies who’ve been backing up only to one day realize their backups weren’t complete or usable :(
Stii 11:09 am on February 9, 2009 Permalink |
Thanx Richard and David. I will write another post with some examples of how it could be done.
Reconstruction | Barry Price 5:58 pm on May 18, 2009 Permalink |
[...] The thing about backups is everyone knows they are critically important. It is like your lungs. You don’t sit around and think about your lungs everyday, all day. Yet, if you loose them, it is pretty much game over # [...]