For a few years (which was a long job gig for me) I developed software for a small publisher whose name was also the acronym for a bio molecule. I always had a warm spot for organic chemistry. Anyway, one day at this company that sounds like an organic compound, someone, not me, mistakenly deleted some files. Our IT guy (there was only one) said not to worry, he could retrieve the files because they made a backup tape each night and a second backup tape was kept off site.

You probably have guessed where this is going. The IT guy found the backup tape empty. Turns out the backup process, or rather the recovery process, had never been tested and he just assumed it worked. Ahh, he'd not tried a retrieval. Oops. Cheap lesson, though, as in this case the files were not company critical.

My web host wasn't so fortunate. Last Friday an admin issued a command that brought the company down, which in itself is impressive, who knew it was so easy? One command? Compounding the error, their local backups weren't any good. Recovery was s.l.o.w. Four days later the company is up and running again though at quite a hit to their reputation. An expensive lesson.