Preparing Our Data Centers For Hurricane Sandy

Filed in Cloud Industry Insights by Jacques Greyling | October 29, 2012 2:15 pm

Hurricane Sandy is poised to pound the mid-Atlantic and Northeast today through Wednesday, with intense winds and torrential rains that could cause widespread power outages and floods, including in areas where Rackspace operates data centers. The teams at our Virginia facilities, and also at our headquarters in San Antonio, have been working round the clock to ensure that our customers’ data remains safe and that they continue to receive the Fanatical Support that they’ve come to expect, even during what many forecasters predict will be a “perfect storm.”

We realize that there are always things that can go wrong (in good weather and bad). But we’re working hard behind the scenes – and drawing on our 14 years of experience — to take preventative action to avoid service disruptions. While we don’t expect any issues, when things do go wrong, we strive to be as responsive and transparent as possible.

Currently, our two Northern Virginia data centers – IAD1 in Herndon and IAD2 in Ashburn – are the most likely to feel the effects of Hurricane Sandy. Northern Virginia is inside the projected forecast cone for the storm, but is not expected to get hit directly.

As of Monday, the National Weather Service said Northern Virginia stands a 90 percent to 100 percent chance of experiencing tropical force surface winds, which are between 39 mph and 73 mph. Northern Virginia also has a 10 percent to 20 percent chance of hurricane force winds. Because of this, a “High Wind Warning” is in effect now through 8 p.m. EDT tomorrow with winds predicted to increase significantly from 1 p.m. today through Tuesday morning. Sustained winds could reach 35 mph to 45 mph, while gusts could hit between 60 mph and 70 mph.

Hurricane Sandy is also expected to produce between five and 10 inches of rain, with up to 12 inches in localized areas. IAD1 and IAD2 are built on land rated by the Federal Emergency Management Agency (FEMA) as being above the 500-year flood marks for the area. Both are built on high ground and not in close proximity to areas that may experience water rise due to a storm surge or flash flooding.

Power outages, too, remain a concern in the mid-Atlantic and northeast. Currently, the generator fuel tanks in both IAD1 and IAD2 are full and the generators can run for roughly 60 hours – that’s 2 ½ days — without refueling. If needed, we can refuel the generators while they are running. Our emergency fuel provider is on notice that we may require additional fuel, which will be delivered within 24 hours of a service call. Additionally, our facility partner stands at the ready and maintains its own emergency fuel delivery contracts. We have also contacted our generator maintenance and repair vendor, which will place a diesel engine specialist on site at IAD1 if required.

We have also ensured proper staffing throughout the event and will continue to offer round the clock customer support services. For our on-site staff we have prepared emergency overnight kits that include inflatable mattresses, blankets, towels and a hygiene kit. The data centers are stocked with food and drink, including military Meals Ready to Eat (MREs) and plenty of bottled water. We have ensured that critical staff is on-site and that the rest of our data center staff is safe and accounted for before, during and after the storm.

We have coordinated ticket queue coverage with the other global data center teams and IAD will remain in constant contact with those teams throughout Hurricane Sandy. We have also blocked and closed the IAD1 and IAD2 scheduled maintenance calendars for the duration of the storm, while all maintenance scheduled before the calendars were blocked will be completed as scheduled.

This isn’t our first brush with a weather event of this magnitude. In August 2011, Hurricane Irene moved through Northern Virginia and caused widespread damage throughout the mid-Atlantic and Northeast. During Irene, neither our Herndon nor Ashburn facilities required transfer to generator power as the utility grid remained unaffected. Also, in late June 2012, a derecho, also known as a “land hurricane,” moved through Northern Virginia causing damage and power outages, some of which lasted up to three weeks. Neither facility required generator power.

Preparations will be ongoing as the storm moves closer. We feel that these preparations and our track record for withstanding major storms will offer our customers continual service without disruption as we brace for Hurricane Sandy.

To our customers and everyone else in Hurricane Sandy’s path, please stay safe.

Source URL: http://www.rackspace.com/blog/preparing-our-data-centers-for-hurricane-sandy/