Apparently, their heads are not in the clouds... To develop and actually test a DR plan, great. And finally be successful, but it becomes such a hassle to retest, and then justification it should just work because everything is "active-active", great. So what you built for failover sits... time passes, staff changes, a manager is short on $$$ so they stop paying for something, nothing is actively checking the end to end health for failover... you have a different fast disk device, that if abruptly powered off has uncommitted data stuck in its cache, someone put some rules into the network to block access, or only allow accesses to specific IP's, when you need failover, it's broken, and no one or nothing knows why.
And maybe no one has checked what happens when you have "active-suddenly inactive". -----Original Message----- From: IBM Mainframe Discussion List <[email protected]> On Behalf Of Jesse 1 Robinson Sent: Friday, February 08, 2019 1:05 PM To: [email protected] Subject: Re: Wells Fargo? Well f*&%#d at the moment: Data center up in smoke, bank website, app down . The Register Having a tested DR failover plan in place and actually pulling the trigger on it are separate issues. We once endured a day-long outage after a failed CEC rather than fail over to a mirrored data center. There were extenuating circumstances. Three senior sysprogs were in POK for z/OS ESP training. IBM was 'on it' from the get-go, but the hanging question was how long it would take to get the CEC running again. Our DR plan was always predicated on the notion of systemic catastrophe. This was not that. There was no damage other than the z/10 book. There's a huge cost to data center failover. At the end of that day, we were up and running, crisis over. If we had failed over, we would have set about to fail back, a prodigious weeks-long effort that could not have been tested in advance. I'm no apologist for Wells--such is karma--but I understand any reticence they may have experienced. . . J.O.Skip Robinson Southern California Edison Company Electric Dragon Team Paddler SHARE MVS Program Co-Manager 323-715-0595 Mobile 626-543-6132 Office ⇐=== NEW [email protected] -----Original Message----- From: IBM Mainframe Discussion List [mailto:[email protected]] On Behalf Of Seymour J Metz Sent: Friday, February 08, 2019 10:25 AM To: [email protected] Subject: (External):Re: Wells Fargo? Well f*&%#d at the moment: Data center up in smoke, bank website, app down . The Register They put all of their eggs in one basket? No hot backup for data center? How 20th Century. -- Shmuel (Seymour J.) Metz http://mason.gmu.edu/~smetz3 ________________________________________ From: IBM Mainframe Discussion List <[email protected]> on behalf of Mark Regan <[email protected]> Sent: Thursday, February 7, 2019 5:13 PM To: [email protected] Subject: Wells Fargo? Well f*&%#d at the moment: Data center up in smoke, bank website, app down . The Register Does Wells Fargo use mainframes? https://secure-web.cisco.com/1XbdCKsnGLKgEwr85APN1jW3fG2wRa_DYqn-xDXZ9jO3awyW8jIquQnJyOyfcrmqMVniUBdat5oPC-eU7rzhVSHXMni4cgoSm21sJuWtR2jKxY5IIlBv37owgp1W83lt6KLn50ISiJGUGPbjjT2LPs_SyIIO-Th44keJWo7_qcXoCwAeaTbSzIvW903QlCMNUAmJo5FA9bCXluc_sVVbGXSeOuwpgRKunm8NlyC8zO9jltbdIKG0XPsNGkYuwixHeBWPlxN1D3qx7lhiWZbk_QD12M7CAfRZjNCUqpA9FM7vw1DSu5G1SH-c989siK6WH-_zhuKM5DLhW-8KWoHEgOXdD3WMrpO_NTiVus7SPMrf5XVZrjU8GCv0HVNgnVh9CL2EUwK17hGyu0RKEW39tgTqlb0r9iXxweNrK-44Dph_UYQic1zTj2mtmrRH8z4Fb/https%3A%2F%2Fwww.theregister.co.uk%2F2019%2F02%2F07%2Fwells_fargo_outage%2F Regards, Mark T. Regan, K8MTR CTO1 USNR-Retired, 1969-1991 ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN DISCLAIMER: This email and any attachments may contain confidential information that is intended solely for use by the intended recipient(s). If you are not the intended recipient, you are strictly prohibited from disclosing, copying, distributing or using any of the information contained in the communication. If you received this email in error, please contact the sender by reply email and immediately delete the communication. ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to [email protected] with the message: INFO IBM-MAIN
