I Can’t Read The Back Up Tapes

“Our receptionist changes the back up tapes every night – we do daily and weekly back-ups.  But our main server crashed and now I can’t read the back up tapes…”.  We hear variations on this story quite often.

In the most recent example, a wee server was tucked into the corner of a successful business that organises events at its own prestigious venues.  Maintenance contracts for the various computers and networks had changed hands a couple of times and while new servers were backed up to modern standards, the wee server had been quietly continuing to work well and cause no bother.  It ran a booking application running over an SQL database and was in daily use.  A receptionist had inherited the task of swapping the back-up tapes at 4pm each day and checking the message on the computer screen to say the back-up had finished.

Of course, the inevitable happened, the server crashed and would not restart.  Mild annoyance quickly turned to pandemonium as it was realised that:

1. the server would not reboot after repeated attempts,
2. the main hard drive could not be accessed using an external USB caddy,
3. the main hard drive caused a PC to power-fail when it was used as a slave drive,
4. there was a second drive present (“please let it be a RAID drive”) – it appeared to be unformatted and blank,
5. the most recent daily back-up tape could not be restored,
6. each of the previous four daily back-up tapes could not be restored,
7. the weekly back-up tape could not be restored…

AAAAGH!

That’s when we received the phone call asking for assistance.

Turns out that the tape back-up procedures were followed by the receptionist to the letter – but the letters were wrong.  There was no procedure in place to verify the tape back-ups – ever.  The unfortunate receptionist had done exactly as she’d been asked, but the computer’s confirmation message was misleading – the back up log file showed it had been years since a tape back-up had properly completed. So the tapes were all useless.

The secondary hard drive was a Microsoft Windows software mirrored RAID, with dynamic disk that appeared unformatted on casual examination, but did indeed contain a full mirror – however, no files had a datestamp later than 2006.  The hard dive was less than useless – it had spent the last four years gently cooking the rest of the inside of the computer.

The main hard drive had physical problems, but we were able to work around these and recover the critical data. A final twist?  A back-up script intended to back up the booking database was in fact backing up the database for the back-up program – which had been attempting and failing to write back-ups to tape daily for years.

I count that as eight forms of back up failing unnoticed before the main system failed.

Not as unusual as you might think.