Some of you have probably noticed that Koji is a bit overloaded at the moment. Well, that's my fault. Sorry about that. I'm writing up a post-mortem on how it happened so we can all learn from my mistakes.
This time around, we coordinated with releng to update the Rawhide tag we trigger on as part of the branching event. That's good! We're not getting forgotten anymore! The problem is that when all of the Rawhide builds were re-tagged for the new trigger (f42), we suddenly enqueued all of those builds again, resulting in a mass rebuild. In the past, this was sort of quietly avoided because two things were different: 1. The trigger tag wasn't updated until after the mass-tag had been completed, mainly because we weren't coordinating properly. 2. We implemented a new "pause" feature in ELNBuildSync (aka EBS) that allows us to leave messages on the message queue instead of processing them immediately. In the past, this meant that we would have read and `DROP`ped the messages for the Rawhide re-tag. So with the combination of those two events, what happened this time is that we unpaused EBS and it immediately read through the list of re-tagging messages and promptly enqueued every ELN package for rebuild. There's a long discussion[1] in the ELN Matrix channel about what we can do to avoid this problem in the future, but right now I think that the simplest answer will be "just don't update the tag we listen for until after the mass-retagging". We're looking into ways we can automate this to avoid it being a manual process, but that's the basic idea. [1] https://matrix.to/#/#eln:fedoraproject.org -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
