True EPO story; maintenance crew carrying new drywall into the data center backed into the EPO that didn't have a cover on it. One of the most eerie sounds in networking...a completely silent data center.
-chris On Fri, Sep 10, 2021 at 2:48 PM Christopher Morrow <[email protected]> wrote: > > > On Fri, Sep 10, 2021 at 1:49 PM Matthew Huff <[email protected]> wrote: > >> Reminds me of something that happened about 25 years ago when an >> elementary school visited our data center of the insurance company where I >> worked. One of our operators strategically positioned himself between the >> kids and the mainframe, leaned back and hit it's EPO button. >> >> > Or when your building engineering team cuts themselves a new key for the > 'main breaker' for the facility... and tests it at 2pm on a tuesday. > Or when that same team cuts a second key (gotta have 2 keys!) and tests > that key on the same 'main breaker' ... at 2pm on the following tuesday. > > <quadruple face palm> > > not fakenews, a real story from a large building full of gov't employees > and computers and all manner of 'critical infrastructure' for the agency > occupying said building. > > Matthew Huff | Director of Technical Operations | OTA Management LLC >> >> Office: 914-460-4039 >> [email protected] | www.ox.com >> >> ........................................................................................................................................... >> >> -----Original Message----- >> From: NANOG <[email protected]> On Behalf Of Sean >> Donelan >> Sent: Friday, September 10, 2021 12:38 PM >> To: [email protected] >> Subject: Never push the Big Red Button (New York City subway failure) >> >> NEW YORK CITY TRANSIT RAIL CONTROL CENTER POWER >> OUTAGE ISSUE ON AUGUST 29, 2021 >> Key Findings >> September 8, 2021 >> >> >> >> https://www.governor.ny.gov/sites/default/files/2021-09/WSP_Key_Findings_Summary-for_release.pdf >> >> Key Findings >> [...] >> >> 3. Based on the electrical equipment log readings and the manufacturer’s >> official assessment, it was determined that the most likely cause of RCC >> shutdown was the “Emergency Power Off” button being manually activated. >> >> Secondary Findings >> >> 1. The “Emergency Power Off” button did not have a protective cover at >> the >> time of the shutdown or the following WSP investigation. >> >> [...] >> Mitigation Steps >> >> 1. Set up the electrical equipment Control and Communication systems >> properly to stay active so that personnel can monitor RCC electrical >> system operations. >> >> [...] >> > -- Chris Kane
