Preparing for IT Recovery from a Hurricane
By Wayne Rash

Preparing to protect IT assets against an imminent disaster-like the recent rash of hurricanes thrashing Florida and nearby-is no easy task, especially when employees are evacuating.   Here is an analysis of one company’s ordeal with Hurricane Ivan.

Jan Rideout knew it would be tough to get through the next few days.  

Hurricane Ivan had drawn a bead on her part of the Gulf Coast, and she knew it would not miss.

Rideout is CIO of Northrop Grumman’s Ship Systems Sector in Pascagoula, Mississippi.   Her data center’s 350 Unix, Windows 200, and Windows NT servers and 5.8 terabytes of data are the hub of a collaborative effort with the US Navy and other contractors to build the next generation of warships for the US fleet.

The Hurricane potentially could have wiped away hundreds of man years of engineering work and brought operations at Pascagoula to a hard stop.

Fortunately, Rideout has a plan.   In fact, she had a plan long standing which is important when your company periodically finds itself in the bull’s eye of disaster.   And because the organization has two locations, one just outside of New Orleans, the chance of being on the spot was doubled.  

“There’s a big emphasis on business continuity planning at Northrop Grumman,” Rideout explained.   “We have procedures and we follow them.”   The company has developed its plans out of long experience, but they also practice the plan and audit it.

But Rideout also noted that sometimes, not everything follows the plan, and then they have to do the best they can.

“Out basic procedure call for us to shut down,” Rideout explained.   She noted that they were paying very close attention to a private weather forecasting service, Impact Weather, Inc. (www.impactweather.com) for the latest word on exactly where Ivan would come ashore, and just how bad the results would be.   “They’re never wrong,” she said.  

After watching Ivan’s progress for days, on September 14, Rideout knew the time had come to take action.   “We made the decision Tuesday morning,” she explained.   Her Avondale facility, just outside of New Orleans had to be shut down because the area was being evacuated.   She stared backups that had to be completed by noon.  

The next step was to back up the massive store of data in Pascagoula-everything from engineering drawings of the ships the company builds parts lists and material for collaboration.   It all had to be saved.   “Fortunately, we had just done weekly backups Monday,” Rideout explained, so all that was required was incremental backup.

“We started backups at noon,” she said, arranging for the backups to be carried by the courier to a secure location.

Once the plans were underway, the IT staff had to decide which, if any, services they decided to keep through the hurricane.   Eventually they decided to keep their e-mail and Blackberry servers running.

Then they had to decide what to do with one of their most important projects, the US Navy’s DDX project, a new type of destroyer already being designed.

Northrop Grumman uses TeamCenter Enterprise a product lifecycle management (PLM) system for UGS, as a collaboration tool to link the efforts of other companies around the country who are working on the design with them.

“We weighed the risks and decided to make the safest decision,” Rideout said, which was to shut down the DDX Environment, as the company calls the collaboration effort, even though it was clear that lost time could be expensive.

As Northrop Grumman’s 350 servers were shut down, each was wrapped in plastic to protect the equipment from any water that might penetrate the data center from leaks or wind damage.

Then, the next challenge: how to provide electrical power for the data center so that the servers that would be kept running would have means to do so.

While Northrop Grumman’s shipyard has its own power plant as is the case with many shipyards; he data center received its power from the local company, Mississippi Power.   But Rideout knew they would lose the that when the storm hit.  

“The shipyard leased a generator for us,” Rideout said.   One problem: the data center needed a power converter before the generator could be used.   A last minute call, and an evening FedEx delivery on the eve of the storm solved that problem.

But there was a new problem.

Shipyard employees couldn’t stay.   Many were being evacuated, many more had to secure their own homes and families.   All the employees were gone, but someone had to connect the data center to the leased generator, and bring up the servers on generator power.   “Some of our employees volunteered to come in Wednesday morning,” Rideout said.   The employees worked quickly to bring down the e-mail and Blackberry servers, attach the generator, and bring the sever back up.

Finally the IT staff was ready for Ivan.   Ivan struck Pascagoula at 1:00 AM.

By then, Rideout had evacuated to Dallas, Texas to wait out the storm.   Dallas is the site of one of Northrop Grumman’s major data centers.

She knew that if it was necessary, e-mail and Blackberry services would be moved there, as could the other functions normally done at the shipyard.

Fortunately, it wasn’t necessary.

Ivan struck land east of the shipyard, resulting in less wind than there might have been.   Rideout knew this might be the case because of information she received ahead of time from Impact/Weather.   But she couldn’t take a chance.

The storm had passed by midday Thursday.

Rideout headed back from Dallas knowing that commercial power wouldn’t be available for several days.   Because of this, the IT staff connected the entire data center to the emergency generator, and one by one, brought the servers online.   The data center was alive, even though it might take some shipyard employees a while to get back to work, at least they’d have IT services available when they did.

Then, at 9:00 PM, the generator failed.   Working late into the night, shipyard electricians and IT staffers switched the data center over the shipyard’s power plant.

“It’s not as clean as Mississippi Power,” Rideout noted, but it was better than no power at all.

Sunday, Mississippi Power restored its service to the shipyard, and everything was back to normal, or as normal as it can be for an area that demonstrates what getting hit by a massive storm is like.

Rideout said that the shipyard actually had very little damage, except for the occasional broken window and subsequent rain and wind damage in isolated areas.

“We absolutely prepared for the worst,” Rideout said, reflecting on the preparations, “We were very lucky,”

Clearly, Rideout, the IT staff, and Northrop Grumman had a big role in making their own luck. Printable copy . , .

iCopyright Clearance License (3.54.2870257-0781)