Why Did the Superdome Lose Power?

5267732232_8985635c01

Because the cause has not yet been reported and, quite often, the underlying root cause is never found. But, it’s worth asking, is it possible to avoid long game outages and what would it cost?  As when looking at any system faults, the tools we have to mitigate the impact are: 1) avoid the fault entirely, 2) protect against the fault with redundancy, 3) minimize the impact of the fault through small fault zones, and 4) minimize the impact through fast recovery.

 

Fault avoidance: Avoidance starts with using good quality equipment, configuring it properly, maintaining it well, and testing it frequently. Given the Superdome just went through $336 million renovation, the switch gear may have been relatively new and, even if it wasn’t, it likely was almost certainly recently maintained and inspected.

 

Where issues often arise are in configuration. Modern switch gear have an amazingly large number of parameters many of which interact with each other and, in total, can be difficult to fully understand. And, given the switch gear manufactures know little about the intended end-use application of each switchgear sold, they ship conservative default settings. Generally, the risk and potential negative impact of a false positive (breaker opens when it shouldn’t) is far less than a breaker that fails to open. Consequently conservative settings are common.

 

Another common cause of problems is lack of testing. The best way to verify that equipment works is to test at full production load in a full production environment in a non-mission critical setting. Then test it just short of overload to ensure that it can still reliably support the full load even though the production design will never run it that close to the limit, and finally, test it into overload to ensure that the equipment opens up on real faults.

 

Here’s an analysis by James Hamilton, an Amazon Distinguished Engineer, on why the Superdome lost power during the Superbowl on Sunday. As usual, click through for the full read.

Photo by Flickr user Cordey

Advertisements

Tags:

About Matthew Peterworth

I currently work as a Technology Specialist at Henderson Engineers in Kansas City – a Mechanical-Electrical-Plumbing (MEP) design firm. Prior to that I was a Project Manager for Information Technology Services (ITS), the central IT department at the University of Texas at Austin. As of July 2008 I am a Registered Communications Distribution Designer (RCDD) through BICSI. And as of June 2011 I am a licensed Professional Engineer (PE) in the great state of Texas. Please follow me on Twitter @mpeterworth. https://twitter.com/mpeterworth

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s