Preparing for the worst.
May 30, 2008
Now this is what I call a disaster recovery team! (follow the comments for some interesting side discussions as well).
One of the things us security guys do is think about the worst thing that could possibly happen. It’s nice playing Dr. Evil with these “what if” scenarios with clients to see how prepared they are with certain things, and helping them plan to put into place ways of protecting or mitigating the damage done.
It’s really easy to discount the risk of particular events happening (meteor taking out the server room), and also to balk about the costs involved in defense against different scenarios (especially if the risk is “small”, but the impact would be large) (please don’t make me think of all the BCP formulae in the CISSP I’m still studying for but haven’t had time to actually take). However, in several critical infrastructure organizations these are exactly the kind of things you have to have plans for – recovering from a smoking hole in the ground or a big natural disaster. I’ll never forget living in Florida in 2004, after being hit by 3 cat 3-4 hurricanes in 6 weeks, and repair crews coming in from all over the country to help put the power back on – everywhere these guys went they were met with just as much hospitality (free BBQ and drinks wherever they turned up) as they showed us by leaving their homes to help out a different state with no contracts or “agreements” required.
Telcos need similar emergency recovery teams, but in some scenarios instead of putting power lines back up (a manpower intensive task), they have to take over the processing of data to keep the backbone, Internet, telephone lines and cell phones all working.
I’m always interested in the whole “what it takes” side of the equation – perhaps its the old roadie in me trying to figure out how to “put on the road” a self-contained operation – and this is it. Some 30-odd trucks to take over the processing of one central office. And this is just some of their $500 million stash of equipment!
Sometimes, planning for the worst takes spending lots of money that you hope never has to actually get used.
Posted in 
