Lines annotated with "known bug" are always skipped. What do you think about the following: 1. Introduce a new "random failure" tag and use that to annotate doctests that are known to fail sometimes (either with segfaults, timeouts, or differing output) 2. Normally on CI, skip the execution of such tests. 3. In one special run after a release (or perhaps also for PRs?), only execute the "random failure" tests. Since there will not be that many such annotated tests, we could even run it a couple of times and only report an error if none of the runs was successful.
On Friday, August 29, 2025 at 3:50:30 AM UTC+8 Vincent Macri wrote: > On 2025-08-27 6:42 p.m., 'tobia...@gmx.de' via sage-devel wrote: > > I don't think tests need to be disabled, but rather the CI should not > > report a PR as failing if the same failure occurs on develop. So still > run the failing tests, but don't report the workflow as failing if the > same test fails on develop. I think we already have something like this > for the fixed seed tests, but not for the random seeds. > > > That would work for tests where only the output differs sometimes. But the > most problems we have at the moment are from random segfaults, and for > those it's probably easier to not run them at all instead of trying to > filter them out afterwards. > > That is a trickier situation. I do think random segfaults should be > considered a bug, but also shouldn't cause CI to mark a PR as failing > (unless that PR introduced a new random segfault, although that's probably > easier to say than detect). As long as we have some CI workflow that still > runs and reports those tests (even if only on develop) I think it would be > okay to skip them on PRs. I'm not sure what the exact behaviour of #known > bug is (does it always skip the test or just in some workflows?), but > either that or something like it should work. > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To view this discussion visit https://groups.google.com/d/msgid/sage-devel/d316f849-371d-4955-ad77-3211cb3881a8n%40googlegroups.com.
