Patrick, Sounds scary enough. Fortunately when we took a power bump at a previous employer - before we talked them into a generator - we had a nasty one too. Fortunately our UPS was big enough to keep the boxes up long enough for me to do an emergency shutdown of everything before they ran out of juice.
We had the standard 3-phase power coming into the building and in the middle of the day the lights in my office went black. My PC kept working, though. I thought somebody was messing with my mind until I went into the hall and saw that the bank of lights was out. I walked down the hall and discovered another set of offices that had lights but their PCs all died. Quick run into the machine room to find both UPSes screaming about power issues. One phase of the 3-phase power was out. Think 3380 disks that (according to rumors propagated by my CE at the time) have to be wired correctly so they don't spin backwards. I started an emergency power down of the mainframe while telling operations to hit the power down buttons on peripherals like tape drives and printers that didn't need to be up for the power down. Got everything down with about 5 minutes left on the UPSes. Once the utility restored power, no problems bringing everything back up. Turned out the breaker on the power pole outside the building decided to trip a single phase of the power coming in. I think we had a generator on order the next day. Have a great weekend. Rex -----Original Message----- From: IBM Mainframe Discussion List [mailto:[email protected]] On Behalf Of Patrick Lyon Sent: Friday, December 03, 2010 2:32 PM To: [email protected] Subject: Re: EPO's (Emergency Power Off) On Fri, 3 Dec 2010 11:45:12 -0600, Darth Keller <[email protected]> wrote: <snip> You're now on battery power and have 23 minutes to power everything off as gracefully as possible. </snip> I, unfortunately, have lived it. Minus the 23 minutes. We had a normal power bump to our building. Happens once in awhile, right? Lights go off for a second and back on, PC reboots, you've recovered from that sudden shock it gives you. So you decide to go into the computer room, "just to check", like you always do... You enter to a dark room with only emergency lights on. Dead silence... Well, almost dead, but dead compared to what the room usually sounds like. You walk through the computer room, through the spotty darkness. Wait - you hear something! Something is running! You hear fans! You walk towards the sound... It's your disk drives thank goodness! You remember they have their own internal battery backup! And within minutes you hear them slowly spin down to silence. Now what? What do I do? The Scenario: The building took a power bump. The UPS's had not been maintained by building services appropriately. They note the power hit, switch from local to battery. Only, the batteries fail. Building services manually switches the UPS's over to local, you check all the breakers. You have management standing over your shoulder asking when everything will be back up again. You don't know - you don't know if there was any data corruption. You don't know if there will be a hardware failure. You just. Don't. Know. If anyone else knows that feeling, you know what I'm talking about, half wondering if you should do "something" or go blind, I hear ya. You wonder if you are going to work for a week straight trying to fix things. You wonder where that copy of your resume is. Luckily we did not suffer any issues. Apparently the disk drives were able to de-stage all the data from cache to disk before it's battery ran out. Everything else, 3480's, 3590's, 3745, 9672, all dead. Luckily (again) we are a somewhat small community, and you know your IBM CE personally. You hunt and fish with him, know his family, invited to their kids weddings. You call him up directly and tell him to get his ___ over here pronto. He shows up and starts powering up everything. The disk drives are already spinning and alive. Everything initializes. The 3480 auto mounts click in succession. You have finally POR'ed the CPU. You IPL. DB2 and IMS do their backout. You restart batch jobs that were running. We may have expirenced a christmas miracle, I don't know. Just know I thanked the big man upstairs that night when I went to bed. But to answer your question Darth, no, we have no real "T minus what the heck to do" sort of document created. Absolutely should. Hope you have enjoyed Friday story time! The information contained in this e-mail may contain confidential and/or privileged information and is intended for the sole use of the intended recipient. If you are not the intended recipient, you are hereby notified that any unauthorized use, disclosure, distribution or copying of this communication is strictly prohibited. If you received this e-mail in error, please reply to sender and destroy or delete the message and any attachments. Thank you. ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: GET IBM-MAIN INFO Search the archives at http://bama.ua.edu/archives/ibm-main.html
