On Sat, 2025-02-15 at 14:54 +0000, Zbigniew Jędrzejewski-Szmek wrote: > On Fri, Feb 14, 2025 at 02:40:29PM -0800, Adam Williamson wrote: > > On Fri, 2025-02-14 at 16:31 -0500, Dusty Mabe wrote: > > > IMO the bar would only need to be that high if the user had no way to > > > ignore the test results. > > > All gating does here (IIUC) is require them to do an extra step before it > > > automatically flows > > > into the next rawhide compose. > > > > again, technically, yes, but *please* let's not train people to have a > > pavlovian reaction to waive failures, that is not the way. > > IMO, the bar for *gating* tests needs to be high. I think 95% true > positives would be a reasonable threshold.
Do you mean 95% of failures must be 'real' (i.e. up to 5% can be 'false')? Is this after automatic retries and manual intervention by the test system maintainers, or before? Off the top of my head, 95% seems low. I'm pretty sure we do better than that with openQA and people would complain if that was all we managed. We usually maintain a 0% false failure rate after auto-retries and <24h manual intervention - https://openqa.fedoraproject.org/group_overview/2?limit_builds=100&limit_builds=400 has 0 false failures ATM. -- Adam Williamson (he/him/his) Fedora QA Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org https://www.happyassassin.net -- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
