Lessons From Recent Power Outages

[Andre Kesteloot]

On 7/18/2012 17:13 PM, Terry Fox wrote:
> Energy Wise - 07/18/2012
> Um, isn’t blaming the grid kind of burying your head in the sand?  If 
> they are supposed to have such a fantastic and reliable Cloud, 
> shouldn’t their cloud support include on-site backup power 
> facilities?  And, there should not be an “secret” or “unknown” closets 
> of fiber that nobody knows about until Virginia Power/Pepco goes bye-bye.
> This reminds me of the story about the American University (or was it 
> Howard?) facility that installed a new generator.  When Pepco died, 
> their generator promptly came on, ran for a few hours, then died.  
> Somehow the pump that fills the day tank from the main tank was wired 
> to Pepco instead of the generator, and the small day tank ran dry. Oops.
> I guess the Elastic got stretched too thin, and things did not 
> computer anymore.  All this because the virtual cloud got bumped 
> offline by a few very real clouds.
> In the old days, this would be a true “teaching moment”, bordering on 
> a firing offense.  Being offline for an hour or two could be 
> excusable, but for days?
> Terry, WB4JFI


Terry,
I do not know whether you recall an earlier item I posted  --> Poor 
engineering indeed:
I like their use of the passive form after "unfortunately"...

> ********************************
> In April 2012, some of Amazon's servers went down. This was their
> explanation for what happened:
>
> /"At approximately 8:44PM PDT, there was a cable fault in the high
> voltage Utility power distribution system. Two Utility substations
> that feed the impacted Availability Zone went offline, causing the
> entire Availability Zone to fail over to generator power. All EC2
> instances and EBS volumes successfully transferred to back-up
> generator power._At 8:53PM PDT, one of the generators overheated and
> powered off because of a defective cooling fan._  At this point, the EC2
> instances and EBS volumes supported by this generator failed over to
> their secondary back-up power (which is provided by a completely
> separate power distribution circuit complete with additional generator
> capacity)._Unfortunately, one of the breakers on this particular
> back-up power distribution circuit was incorrectly configured to open
> at too low a power threshold and opened when the load transferred to
> this circuit._  After this circuit breaker opened at 8:57PM PDT, the
> affected instances and volumes were left without primary, back-up, or
> secondary back-up power."/

73
André N4ICK
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://amrad.org/pipermail/tacos/attachments/20120718/54ea7605/attachment.html>