On Tue, Jan 23, 2024 at 09:04:53AM -0600, Chris Adams wrote: > Once upon a time, Richard W.M. Jones <rjo...@redhat.com> said: > > The authentication issue being this one: > > > > https://pagure.io/fedora-infrastructure/issue/11733 > > I'd be interested in an after report on this one... as someone who has > managed FreeIPA, I'd like to know how this happened (so I can file away > how to NOT do the same thing in my own setups).
It seemed to be a number of things at once sadly, as often such things are. We took a cluster member down and reinstalled rhel9 on it (to start upgrading the cluster), but then the replication agreements for all nodes were accidentally removed. That might have been easily recoverable, but then we also hit that in our case the cluster was installed a long time ago and didn't have SID's, which became manditory to fix a CVE in the most recent version. And then we also hit some old kruft leftover from when our cluster was in another datacenter long ago. ;( Many kudos to everyone who worked on this. Especially the IPA folks. They have been calm and understanding and really helped us track things down and get back working. > Certainly not bothering anybody while there's still an outage (or while > they're recovering from dealing with it), but when things like this > happen, it's good for everybody to document how it happened - NOT to > cast blame or anything like that (sooner or later, we all do something > that breaks in wildly unexpected ways), but so we can all learn from the > mistake. Absolutely. Things are not fully normal now, but everything should be up from the user perspective. We will be working to get the cluster back to a normal state in the next few days, then we can look at retrospective. kevin
signature.asc
Description: PGP signature
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
