Diary of an ASP.NET bodger

Powervault 220s, Databases & BACKUPS

Ok. Don't panic but BACK YER FREAKING DATA UP NOW!!!

We have a Powervault 220s which we use half for the mailboxes and half for the main database server. Its on RAID 5, has just had new drives installed and the power supply scrubbed. We also have a well thought out (so we thought) backup solution using tapes and Backup Exec. Anyway, at 3.51pm on Saturday, it all went tits up. The powervault was just dead. Powered off and on again and it came back up but the drives were offline apart from the hotspare. Nightmare. That's like 90% of the business critical stuff. DELL were helpful, but not really much help - they suggested swapping power leads which wasn't it but I'm not getting at Geoff the support dude, I know how hard support can be.

We finally found a solution in the form of this link here
http://forums.us.dell.com/supportforums/board/message?board.id=pv_raid&message.id=214&view=by_date_ascending&page=1 

It would appear that there is an issue with the 220s. Problem is - the powervault doesn't look like a computer and there's no floppy or keyboard.. so the existance of firmware just didn't occur to me. I realise this is poor logic but *it just didn't* ok?? The loing and short of it is that the databases and mailboxes are corrupted. We have backups but it takes forever to restore from tape with Backup exec (6hrs for 16gig of mail... that can't be right can it??) and we lost two tables.

Upshot is I'm revisitng our backup strategy. As well as the backup exec, I think we also need to be doing differentials as well, and full backups to another server entirely so that we can restore to another machine and at least get back to work. And we need to do drills and simulated crashes. Disaster can strike anytime so BACK YER FREAKING DATA UP NOW!!!!

 

Posted: Oct 21 2003, 10:49 AM by Mr Pike | with 4 comment(s)

Comments

Brian Desmond said:

You're not doing anything but full tapes? I have about 80GB of data spaced across SQL, Exchange, File Servers, and some user machines. I only run full jobs to the drive on Weds and Sat because of backup time, primarily.
# October 22, 2003 12:20 AM

Damian said:

I know, I know.
It's one of those domino style catalog of errors - I was doing differential back ups to another machine but the drive filled up and then a few things stopped working and then... yada yada... so by the time we got round to it - death had struck.

You're right with the tape only slap - but you thing - hey- Powervault by Dell, Backup Exec by VEritas - all best of breed. But when the disaster strikes you're shocked by how badly thought out your recovery plan actually is.

Fer instance, my differential backup of the main database - I had that. 1 weeks worth on a rolling file. But it wouldn't do it because of the replication issues - couldn't find a way round it at that time of night and business needs were pressing so we dumped the loss of half a days data and got back online.

Brian - I'd be interested to know what your real-life strategy is, you sound like you run similiar amounts of data on the same platforms.
# October 22, 2003 4:49 AM

Walter said:

I'm sorry, but not much by Dell is best of breed. They are a discount house plain and simple.
# December 18, 2003 12:44 AM

TrackBack said:

^_^,Pretty Good!
# April 9, 2005 10:12 PM
Leave a Comment

(required) 

(required) 

(optional)

(required)