Hi Timo and Simon, On Sat, 4 Jul 2026 at 22:25, Timo Röhling <[email protected]> wrote: .. > I find it really frustrating to see my CI turn red only to realize that > it's just a random pipeline failure such as [1]. IMHO, these false > alarms attribute to much more "alert fatigue" than your implied > unwillingness of maintainers to keep their packages in good shape; > especially if said packages build just fine on the buildds and pass the > autopkgtests on debci. > > I know not all causes of these intermittent CI failures are easily > fixable (runners are a scarce resource, for example), and the last thing > I want to do is lay blame at the volunteers who put in a lot of work to > improve and further develop the CI pipeline. I do expect the same > courtesy the other way around, though.
Note that the Salsa CI pipeline definition at https://salsa.debian.org/salsa-ci-team/pipeline is maintained by one (unofficial) team, while the salsa.debian.org instance itself is maintained by the Salsa administrators. I have noticed those 503 and 502 errors lately myself too, and since you also seem to be running into them I filed now these: https://salsa.debian.org/salsa/support/-/work_items/597 https://salsa.debian.org/salsa/support/-/work_items/598 On Sun, 5 Jul 2026 at 00:15, Simon McVittie <[email protected]> wrote: ... > This doesn't just apply to CI-runner infra failures like that one: it's > relatively frequent for Salsa-CI jobs to fail for reasons that have very > little to do with the package under test, like a regression in a > dependency or in the test tool (piuparts, Lintian, or similar), or a > limitation in the pipeline (for example jobs that can't succeed when > targeting experimental or backports, as a result of apt's default > pinning). The better the Salsa-CI pipeline coverage becomes, the more > often we'll have at least one job failing for a reason that isn't a bug > in the package under test. Most new features added to Salsa CI in recent years are opt-in. The default pipeline is fairly conservative and when it flags build, lintian or autopkgtest issues they are most of the time real findings. But it is true that Debian occasionally has bugs in core basic build/test tools such as dpkg, apt or lintian. In those cases both local builds and Salsa CI break, causing disruption until a fix is uploaded. This could be mitigated by running all of Salsa CI based on software versions from Debian testing instead of unstable (discussed in https://salsa.debian.org/salsa-ci-team/pipeline/-/work_items/551). There are also cases where a change in Salsa CI itself caused a regression, like the one with piuparts a few weeks ago. This is however quite rare, as Salsa CI has extensive CI itself, and all changes require a separate person to approve and merge them, making sloppy/hasty changes unlikely. > Pushing commits to Salsa to disable the affected jobs, and then > re-enable them when the regression has been addressed, could easily > result in generating more CI runner load than leaving them failing (as > well as taking up more contributor time). I don't see this in practice as a big roblem. You can re-run individual jobs without having to run the full pipeline if the failure was external and not in the commit itself. Keeping a lot of failiing pipelines seems like a way larger issue. But in general of course false positives erode respect for the CI, of course. We need to guard that the pipeline definition and the infra running stays as solid as possible. For individual packages using the CI I suggest that if the CI is too flaky, it would be better to turn it off than to keep running and failing over and over.
