On Fri, 2025-08-29 at 10:40 +0100, Richard Earnshaw via Gcc wrote: > On 29/08/2025 10:36, Iain Sandoe wrote: > > > > > > > On 29 Aug 2025, at 10:32, Richard Earnshaw > > > <richard.earns...@arm.com> wrote: > > > > > > On 29/08/2025 04:08, Jacob Bachmeyer wrote: > > > > On 8/28/25 10:10, Jeff Law wrote: > > > > > On 8/28/25 8:09 AM, Richard Earnshaw (lists) wrote: > > > > > > On 28/08/2025 15:01, Iain Sandoe wrote: > > > > > > > [...] > > > > > > > > > > > > Well really, the compare-tests script should report > > > > > > duplicate results as a problem as well, since > > > > > > > > > > > > PASS: abcd > > > > > > ... > > > > > > PASS: abcd > > > > > > > > > > > > is just a dup pass/fail waiting to happen. > > > > > Yup. A duplicate testname should be reported. These cause > > > > > major headaches if one passes, but the other fails -- it > > > > > looks like a regression to the comparison scripting we have. > > > > The problem with detecting duplicate names in the DejaGnu > > > > framework is that it would add memory overhead that scales with > > > > the number of tests and DejaGnu tries to avoid that kind of > > > > unbounded space requirement. (OK, it *is* bounded for any > > > > finite testsuite, but the idea of a steadily growing memory > > > > footprint during a test run still bothers me.) > > > > I suggest that the comparison script GCC uses is probably the > > > > best place to check for duplicate test names, since that seems > > > > to also be the script that can be confused by them. > > > > > > That's exactly what I was suggesting. Trying to do it in dejagnu > > > would be a nightmare given that we run multiple instances of it > > > to get parallel testing. > > > > For the record, I’ve now proposed a BoF for the cauldron on > > ‘improving the raw output’ from the testsuite, since that seems at > > least one place we can make progress - by making the input to the > > post-processing tools more machine-friendly. Hopefully the > > concerns and ideas from this thread can contribute there. > > I don't know how much maintenance DejaGNU is getting these days, but > certainly something like a --json option to runtest that caused the > .sum > files to be written as structured data would make some post- > processing > significantly easier.
I wonder if that json format could be SARIF? (which might give us some nice IDE integration and other capabilities "for free") Perhaps SARIF is a good fit (or perhaps it isn't). In any case, it sounds like I should try and attend the BoF on improving the testsuite output - thanks for proposing it. Dave
