EF5: Too Close for Comfort

By Josh Griffin, IT Manager

In March, I gave a Webinar on disaster preparedness. We discussed the importance of having a plan, and the fact that this plan will always require fine-tuning. If you have seen the news lately, you will recall the May 25th tornado that claimed eight lives and destroyed 200 homes in Parkersburg, IA.

Parkersburg is just 20 miles from Cedar Falls, IA where our production co-location data center is located. The very same storm touched down just seven miles from our data center, which is built to withstand an EF4 category storm. At the time of the storm, I wasn’t worried that a tornado was in Cedar Falls. The likelihood of a tornado in Iowa being stronger than an EF4 is rare - the last EF5 that came through Iowa was in 1976. So my data center is safe right?

Boy was I wrong. On Tuesday May 27th, two days after the storm, the National Weather Service classified that the tornado as an EF5. Learning this really made me stop and think. I immediately went into review mode. It has been three months since I looked at my disaster recovery plan. I take our backups offsite weekly, so I have backups. But, it takes more than just offsite backups to make a good disaster recovery plan.

There are some of the questions that we need to continually revisit to keep our plan up-to-date. For example:

1. What information do we need to recover each production server (i.e., files, third party programs, etc.)?
2. Do we have a detailed load plan for each server that annotates everything installed? Is it up to date?
3. Are our current backups getting absolutely every piece of data we need?
4. If the data center is taken out where would I go?
5. What emergency funds would we require and does the insurance cover everything?
6. What hardware and software would it take to recover everything?
7. Do we have a prioritized list of steps to complete such an endeavor?
8. Do we have all the contact information for all clients to notify them of the disaster?
9. Do we have a current call list internally that we can use if we have to communicate such a disaster?
10. Who will be responsible for what communications?

I have a good plan, but it can always be improved.

So, I sat down with our IT department and we reviewed every server in production, annotating what was installed, how it was setup and ensuring we had all the data on our backups. If we pull tapes, we have everything in one location. We then made sure our hardware inventory was up-to-date and worked with our finance department to ensure our insurance covers our investments.

There are many facets to a good disaster recovery plan, and this tornado was a good reminder that it’s important to plan for the unexpected. And, if the worst happens, all you can do is execute the plan.