On 2/15/25 9:54 AM, Zbigniew Jędrzejewski-Szmek wrote:
> On Fri, Feb 14, 2025 at 02:40:29PM -0800, Adam Williamson wrote:
>> On Fri, 2025-02-14 at 16:31 -0500, Dusty Mabe wrote:
>>> IMO the bar would only need to be that high if the user had no way to
>>> ignore the test results.
>>> All gating does here (IIUC) is require them to do an extra step before it
>>> automatically flows
>>> into the next rawhide compose.
>>
>> again, technically, yes, but *please* let's not train people to have a
>> pavlovian reaction to waive failures, that is not the way.
>
> IMO, the bar for *gating* tests needs to be high. I think 95% true
> positives would be a reasonable threshold.
I can't promise 95% true positive rate. These aren't unit tests. They are
system wide
tests that try to test real world scenarios as much as possible. That does mean
pulling
things from github/quay/s3/Fedora infra/etc.. and thus flakes happen. Now, in
our
tests we do collect failures and Retry them. If a retry succeeds we take it as
success
and never report the failure at all. However there are parts of our pipeline
that might
not be so good at retrying.
All I'm trying to say is that when you don't control everything it's hard to
say with
confidence something will be 95%.
As I promised before, maybe just work with us on it. These tests have been
enabled for
a while and I've only seen a handful of package maintainers look at the
failures (you,
Zbyszek, being one of them; thank you!).
We do want them to be useful tests and I promise when a failure happens because
of our
infra or tests themselves being flakey we try to get it fixed.
TANGENT: Maybe there's a bodhi feature request in here somewhere where there is
a concept
of soft gating. i.e. the update gets held for one cycle if the test
fails, but
if no other action is taken (i.e. negative karma or something) then it
will go
into the next cycle update. Could be a good middle ground.
>
> I know this is hard to achieve, but gating failures are very disruptive
> to packagers. And false positives are even more disruptime: a person
> not intimately familiar with the CI needs to spend a considerable
> amount of time jumping through not-very-easy-to-read pages.
> *If* I can be almost certain that the report is real and with enough
> digging I'll figure out a bug, I'm ready to spend the time. But if
> there's a considerable chance of a false positive _and_ the effort to
> figure out if the report is real is also quite high, I'll just learn
> to waive any failures.
>
> --
>
> I cannot find it right now, but I hope it's still there somewhere…
> Bodhi updates have (had?) a message that said someting like "for failing
> update.* tests, contact <this channel>, for failing fedora-ci.* tests contact
> <that channel>". Is something like that planned for the coreos.* tests?
>
> Zbyszek
--
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it:
https://pagure.io/fedora-infrastructure/new_issue
