On 16.05.25 14:29, Mark Brown wrote:
On Fri, May 16, 2025 at 10:02:16AM +0200, David Hildenbrand wrote:
On 15.05.25 10:57, Mark Brown wrote:
The kselftest framework uses the string logged when a test result is
reported as the unique identifier for a test, using it to track test
results between runs. The gup_longterm test completely fails to follow
this pattern, it runs a single test function repeatedly with various
parameters but each result report is a string logging an error message
which is fixed between runs.
As the person who wrote that test (that you apparently didn't CC for some
I just CCed whoever get_maintainers told me to CC for the patch.
For the future, it's a good idea to look for the author of the
problematic bits.
reason), what exactly is the problem with that?
We run tests. If all pass, we're happy, if one fails, we investigate.
None of the tooling is able to either distinguish between the multiple
tests that are being run in gup_longterm, nor compare the results of
multiple runs effectively. If all the tests run they report themselves
as being duplicates of the same test name, if one of them starts failing
the effect is that one of the duplicates disappears and we get an
entirely new test that's never passed reported. If multiple tests fail
it's even worse. This means that UIs displaying test results end up
reporting things unclearly (Was there a regression or was a new test
that never worked added? What was the test?). Since it's difficult
to track the tests between runs tooling that does reporting of things
like "This last worked in X" in the UI doesn't work properly, and tool
driven bisection of issues similarly struggles since it can't tell
what's going on with any of the tests between runs.
Okay, so this is purely to make tooling happy. Humans are smart enough
to figure it out.
What mechanism do we have in place to reliably prevent that from
happening? And is this at least documented somewhere ("unique identifier
for a test")>
I guess when using kselftest_harness, we get a single identifier per
tests (and much less output) just automatically.
> > Basically, the output is garbled and vastly less useful for people
running this as a matter of routine or as part of a broader kselftest
run. For example with my own automation I probably won't notice that a
previously working test failed unless every single test fails, and newly
added tests that never worked are a much lower priority to the point
where I may never look at them depending on where they are.
If a selftest is reporting multiple tests it should report them with
names that are stable and unique.
I'm afraid we have other such tests that report duplicate conditions.
cow.c is likely another candidate (written by me ;) ).
Probably, the affected tests should be converted to use
kselftest_harness, where we just report the result for a single tests,
and not the individual assertions.
That would reduce the output of these tests drastically as well.
So that is likely the way to clean this up properly and make tooling happy?
--
Cheers,
David / dhildenb
