Source: automake-1.16 Version: 1:1.16.5-1.3 Severity: important User: [email protected] Usertags: flaky X-Debbugs-Cc: [email protected] Control: affects -1 + src:glib2.0 src:vala src:python3-defaults
The automake test suite routinely takes more than 2h30 on slower CPU architectures (armel and armhf), which is uncomfortably close to autopkgtest's arbitrary limit of 10k seconds (a bit more than 2h45). If it runs a bit slower for whatever reason - perhaps extra load on the testbed machine - then the test fails with a timeout. This means that when packages that are depended on by the test suite, such as glib2.0, python3-defaults and vala, are trying to migrate to testing, the test will randomly pass or fail, causing those packages to be detected as having caused a regression when in fact they have not. For example, in the recent history of <https://ci.debian.net/packages/a/automake-1.16/testing/armel/>, we can see these: * testing only (migration reference): pass, 2h50 * testing only (migration reference): fail, 2h51 * testing + src:glib2.0 -3 from unstable: fail, 2h52 * testing + src:glib2.0 -3 from unstable: fail, 2h51 * testing + src:glib2.0 -2 from unstable: pass, 2h37 * testing + src:glib2.0 -2 from unstable: fail, 2h52 * testing + src:vala from unstable: pass, 2h27 * testing + src:vala from unstable: fail, 2h53 and this is getting in the way of glib2.0's migration. I am quite confident that there is no runtime regression between glib2.0/2.76.4-2 and glib2.0/2.76.4-3: they are the same codebase, with only build-time changes to work around a Meson bug. The times shown on ci.debian.net are a bit longer than the test timeout of 10k seconds, which is normal: they include activities like installing dependencies and reporting results, which are not included in the test timeout. automake maintainers: can this test suite be made a bit faster without sacrificing too much coverage, perhaps by skipping some repetitive or slow tests, or by breaking it up into batches that each take 1 hour or less? CI infrastructure maintainers: can we mitigate this by having debci run autopkgtest with --timeout-factor=2 on the slower architectures, or by only testing automake-1.16 on the faster architectures? I see riscv64 is already using --timeout-factor=2. smcv
