Hi Alexis,
> On 19 Oct 2023, at 09:36, Alexis Lothoré via lists.yoctoproject.org
> <alexis.lothore=bootlin....@lists.yoctoproject.org> wrote:
> The regression report looks worryingly empty. It has been generated with
> yocto-4.2 as comparison base. When taking a look at yocto-testresults content
> for it, I only find a few test results (11, while it should be something
> around
> 300). I am not sure yet if we did some mistakes while re-uploading manually
> missing tests results during the 4.2 cycle, or if tests results are indeed
> suffering some issues for release builds (this is repeatable on different
> 4.2.x
> releases).
>
> So in the mean time, here is a manual regression report between 4.3_M3 and
> yocto-4.3.rc1:
>
> https://pastebin.com/6QbfGstR
>
> The report has a "rate limit" applied for noisy regression (patch incoming)
Thanks for that, much appreciated.
I just skimmed your report and have some feedback to hopefully make it easier
to read in the future.
I’d suggest sorting the output in order of importance. For example, this is a
section that I really don’t care about:
Match: sdk_core-image-sato_x86_64_fvp-base_20230910083055
sdk_core-image-sato_x86_64_fvp-base_20231017222150
Put those at the bottom, or even better collate them into a single section
where there have been no changes.
Similarly:
Match: runtime_core-image-sato_qemux86_20230911011430
runtime_core-image-sato_qemux86_20231017223736
Additionally, 1 new test(s) is/are present
I guess marginally more important than identical results, but “there are new
tests that passed” isn’t very interesting.
Now on to the regressions.
Regression: oeselftest_ubuntu-22.04_qemux86-64_20230911011940
oeselftest_almalinux-9.2_qemux86-64_20231017221342
Should they have matched? The host distro doesn’t match and this matters for
some of the tests, as some distros don’t support some of the selftests. In this
case specifically, there are seven regressions and six of them are specific to
the host changing, which has the side-effect of hiding the one actual
regression.
The report then lists the first however many regressions before announcing the
summary:
(In total, 7134 regressions/status changes detected)
Additionally, 7 previously failing test(s) is/are now passing
Additionally, 4622 new test(s) is/are present
The headline figure of 7134 regressions should be first, as that’s the most
important data point in a skim of the report. List the summary first, and then
the breakdown.
Grouping the results would be interesting, because the list got truncated I
can’t see easily if all 7134 regressions were in ptestresult.gcc-g++-user.* or
if that was just the first 100 and the rest were other components. Breaking
the ptest results up by the second level component would be interesting, if it
said something like this then we’d be able to get a feel for what components
have broken from the report.
7134 regressions detected.
ptestresult.gcc-libstdc++-v3-user.30_threads/thread/native_handle/cancel.cc
execution test: PASS -> FAIL
[ say 10 results per component ]
And 6123 more in ptestresult.gcc-libstdc++v3
ptestresult.gcc-libgomp.libgomp.c++/ctor-10.C: UNSUPPORTED -> UNRESOLVED
This one also caught my eye:
Regression: runtime_core-image-sato_qemuppc_20230910082140
runtime_core-image-sato_qemuppc_20231017222112
systemd.SystemdJournalTests.test_systemd_boot_time: PASSED -> SKIPPED
Additionally, 1 new test(s) is/are present
Is that comparing a systemd test run with a sysvinit test run?
Thanks for the work on the tool so far, this is a lot easier to read than the
full reports!
Ross
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#61411): https://lists.yoctoproject.org/g/yocto/message/61411
Mute This Topic: https://lists.yoctoproject.org/mt/102034502/21656
Group Owner: yocto+ow...@lists.yoctoproject.org
Unsubscribe:
https://lists.yoctoproject.org/g/yocto/leave/6691583/21656/737036229/xyzzy
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
