On Sat, 15 Feb 2025 at 20:51, Adam Williamson <adamw...@fedoraproject.org> wrote:
> On Sat, 2025-02-15 at 14:54 +0000, Zbigniew Jędrzejewski-Szmek wrote: > > On Fri, Feb 14, 2025 at 02:40:29PM -0800, Adam Williamson wrote: > > > On Fri, 2025-02-14 at 16:31 -0500, Dusty Mabe wrote: > > > > IMO the bar would only need to be that high if the user had no way > to ignore the test results. > > > > All gating does here (IIUC) is require them to do an extra step > before it automatically flows > > > > into the next rawhide compose. > > > > > > again, technically, yes, but *please* let's not train people to have a > > > pavlovian reaction to waive failures, that is not the way. > > > > IMO, the bar for *gating* tests needs to be high. I think 95% true > > positives would be a reasonable threshold. > > Do you mean 95% of failures must be 'real' (i.e. up to 5% can be > 'false')? Is this after automatic retries and manual intervention by > the test system maintainers, or before? > > Off the top of my head, 95% seems low. I'm pretty sure we do better > than that with openQA and people would complain if that was all we > managed. We usually maintain a 0% false failure rate after auto-retries > and <24h manual intervention - > > https://openqa.fedoraproject.org/group_overview/2?limit_builds=100&limit_builds=400 > has 0 false failures ATM. > Thanks, that's interesting. What do you call <24h manual intervention? One example that comes to mind would be to disable or snooze a test that started to trigger false failures in under 24h. If that's the case, I think that sounds achievable. > -- > Adam Williamson (he/him/his) > Fedora QA > Fedora Chat: @adamwill:fedora.im | Mastodon: @ad...@fosstodon.org > https://www.happyassassin.net > > > > > -- > _______________________________________________ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > Do not reply to spam, report it: > https://pagure.io/fedora-infrastructure/new_issue >
-- _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
