Hi!
On 2025-04-13T17:15:05+0200, I wrote:
> On 2025-03-18T14:54:23+0100, Jakub Jelinek <ja...@redhat.com> wrote:
>> The following patch offers [...] dg-output-file
>> directive where one can supply a text file with expected output
>
>> --- gcc/doc/sourcebuild.texi.jj 2025-03-11 09:18:21.750133577 +0100
>> +++ gcc/doc/sourcebuild.texi 2025-03-18 14:41:59.253345259 +0100
>> @@ -1315,6 +1315,10 @@ Prune messages matching @var{regexp} fro
>> @item @{ dg-output @var{regexp} [@{ target/xfail @var{selector} @}] @}
>> This DejaGnu directive compares @var{regexp} to the combined output
>> that the test executable writes to @file{stdout} and @file{stderr}.
>> +
>> +@item @{ dg-output-file @var{file} [@{ target/xfail @var{selector} @}] @}
>> +Compares the content of @var{file} against the combined output that the
>> +test executable writes to @file{stdout} and @file{stderr}.
>> @end table
>
> Do we have to clarify (in documentation and/or code) what happens if both
> 'dg-output' and 'dg-output-file' are used?
>> --- gcc/testsuite/lib/gcc-dg.exp.jj 2025-03-13 14:05:09.707017142 +0100
>> +++ gcc/testsuite/lib/gcc-dg.exp 2025-03-18 14:37:32.476088575 +0100
>
>> @@ -497,6 +498,75 @@ if { [info procs ${tool}_load] != [list]
>
>> + set name [file tail [lindex ${output-file} 1]]
>
>> + fail "$name output file test"
>> + send_log "Unexpected character $c on line [expr
>> $linenum - 1] where new-line expected\n"
>> + verbose "Failed test for output line [expr $linenum -
>> 1]" 3
>
>> + fail "$name output file test"
>> + send_log "Output line $linenum was:\n$output_line\nShould
>> match (from [lindex ${output-file} 1]):\n$line\n"
>> + verbose "Failed test for output line $linenum $line" 3
>
>> + fail "$name output file test"
>> + send_log "Unexpected character $c on line $linenum where
>> and of output expected\n"
>> + verbose "Failed test for output line $linenum" 3
>
>> + pass "$name output file test"
>> + verbose "Passed test for output file [lindex ${output-file}
>> 1]" 3
>
> OK to push the attached "Polish 'dg-output-file' test logs"?
> --- a/gcc/testsuite/lib/gcc-dg.exp
> +++ b/gcc/testsuite/lib/gcc-dg.exp
> @@ -507,8 +507,7 @@ if { [info procs ${tool}_load] != [list] \
> set linenum 1
> set outfile [open [lindex ${output-file} 1]]
> set do_fail 0
> - set name [file tail [lindex ${output-file} 1]]
> - verbose "output-file args is $args program is $program" 1
> + upvar name name
OK, that would've been too simple... It did work when testing, for
example, 'RUNTESTFLAGS=dg.exp=dg-output-file-1.c' in isolation, but
breaks when doing a full test run:
[...]
spawn [open ...]
This is a test output for ilp32 target
to verify
dg-output-file directive
ERROR: tcl error sourcing [...]/source-gcc/gcc/testsuite/gcc.dg/dg.exp.
ERROR: can't read "name": no such variable
while executing
"pass "$name output file test""
(procedure "saved_asan_gcc_load" line 90)
invoked from within
"saved_asan_gcc_load ./dg-output-file-1.exe"
("eval" body line 1)
invoked from within
"eval [list saved_asan_${tool}_load $program] $args"
(procedure "gcc_load" line 5)
invoked from within
"${tool}_load $output_file"
(procedure "saved-dg-test" line 218)
invoked from within
"saved-dg-test [...]/source-gcc/gcc/testsuite/gcc.dg/dg-output-file-1.c {}
{ -ansi -pedantic-erro..."
("eval" body line 1)
invoked from within
"eval saved-dg-test $args "
(procedure "dg-test" line 1)
invoked from within
"dg-test $testcase $options ${default-extra-options}"
(procedure "dg-runtest" line 10)
[...]
Spot in there our dear friend 'saved_asan_${tool}_load',
<https://inbox.sourceware.org/87bjsy7trd....@euler.schwinge.ddns.net>
"Re: Asan/Tsan Unit/Regression testing". But even with that fixed,
there'd be legitimate uses of wrapping '${tool}_load', so we have to
support that. Fortunately, there's 'testname-for-summary' available,
which is already used in a lot of similar cases, and takes care to
'upvar' through the levels, until it finds 'name' of 'dg-test'. OK to
push the attached v2 "Polish 'dg-output-file' test logs"?
Grüße
Thomas
>From 86f294b84d18facf980181d9c4caa9eee6f309e0 Mon Sep 17 00:00:00 2001
From: Thomas Schwinge <tschwi...@baylibre.com>
Date: Sun, 13 Apr 2025 16:50:37 +0200
Subject: [PATCH] Polish 'dg-output-file' test logs
Per commit r15-8260-g563e6d926d9826d76895086d0c40a29dc90d66e5
"testsuite: Add support for dg-output-file directive", this currently produces
test logs as follows:
PASS: gcc.dg/dg-output-file-1.c (test for excess errors)
PASS: dg-output-file-1-lp64.txt output file test
PASS: gcc.dg/dg-output-file-1.c execution test
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O0 (test for excess errors)
PASS: COMP-6_arithmetic.out output file test
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O0 execution test
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O1 (test for excess errors)
PASS: COMP-6_arithmetic.out output file test
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O1 execution test
[Etc.]
Notice that the 'PASS: [...] output file test' lines easily produce duplicate
test names, or might even produce PASS plus FAIL for the same test names.
Make the "output file test" use the same "descriptive name" as the other parts,
and get properly sorted with parallel-testing 'contrib/dg-extract-results.sh'
processing:
PASS: c-c++-common/zero-scratch-regs-leafy-2.c -Wc++-compat (test for excess errors)
-PASS: dg-output-file-1-lp64.txt output file test
PASS: gcc.dg/20000108-1.c (test for excess errors)
[...]
PASS: gcc.dg/devnull-dump.c (test for excess errors)
PASS: gcc.dg/dg-output-file-1.c (test for excess errors)
PASS: gcc.dg/dg-output-file-1.c execution test
+PASS: gcc.dg/dg-output-file-1.c output file test
PASS: gcc.dg/dg-test-1.c (test for excess errors)
..., and gets de-duplicated test names, for example:
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O0 (test for excess errors)
-PASS: COMP-6_arithmetic.out output file test
+PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O0 output file test
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O0 execution test
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O1 (test for excess errors)
-PASS: COMP-6_arithmetic.out output file test
+PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O1 output file test
PASS: cobol.dg/group2/COMP-6_arithmetic.cob -O1 execution test
[Etc.]
(Given that only ever a single 'dg-output-file' directive is active, don't
print the output filename.)
gcc/testsuite/
* lib/gcc-dg.exp (${tool}_load): Polish 'dg-output-file' test
logs.
---
gcc/testsuite/lib/gcc-dg.exp | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/gcc/testsuite/lib/gcc-dg.exp b/gcc/testsuite/lib/gcc-dg.exp
index eadc1cd90bc..6dd8fa3fce9 100644
--- a/gcc/testsuite/lib/gcc-dg.exp
+++ b/gcc/testsuite/lib/gcc-dg.exp
@@ -507,8 +507,7 @@ if { [info procs ${tool}_load] != [list] \
set linenum 1
set outfile [open [lindex ${output-file} 1]]
set do_fail 0
- set name [file tail [lindex ${output-file} 1]]
- verbose "output-file args is $args program is $program" 1
+ set name [testname-for-summary]
while { [gets $outfile line] >= 0 } {
if { $linenum != 1 } {
set c [string index $output $idx]
--
2.34.1
