Hi folks.
I've been having troubles comparing the results of different test runs
for quite some time, and have finally decided to whine about it.
Perhaps someone can point out to whatever I may be doing wrong.
I generally do "make check -k -j60" on two different trees and compare
the results with:
cd /dir/current-build-tree
/gcc-sources/contrib/compare_tests /dir/previous-build-tree .
However, this gives false positives virtually every time:
New tests that FAIL:
g++.dg/tree-prof/pr79259.C execution, -g
gcc.dg/tree-prof/ic-misattribution-1.c execution, -g
gcc.dg/tree-prof/merge_block.c execution, -g
gcc.dg/tree-prof/section-attr-1.c execution, -g
gcc.dg/tree-prof/section-attr-3.c execution, -g
...
...
New tests that PASS:
g++.dg/tree-prof/partition1.C compilation, -g
g++.dg/tree-prof/partition2.C compilation, -g
g++.dg/tree-prof/partition3.C compilation, -g
g++.dg/tree-prof/pr79259.C compilation, -g
...
...
Old tests that passed, that have disappeared: (Eeek!)
g++.dg/tree-prof/indir-call-prof-2.C compilation, -g
g++.dg/tree-prof/morefunc.C compilation, -g
g++.dg/tree-prof/pr63581.C compilation, -g
gcc.dg/tree-prof/20050826-2.c compilation, -g
gcc.dg/tree-prof/bb-reorg.c compilation, -g
...
...
etc
etc
etc
More often than not the tests that get mentioned are in the
gcc.dg/tree-prof directory, but I've seen failures in gcc.dg/tree-ssa,
gcc.dg/vect and others.
When I run these tests manually (make check-gcc
RUNTESTFLAGS=tree-ssa.exp=whatever.c), they succeed.
Running with smaller -jNN usually yields less false positives, but
rarely do I get a clean run.
This has been happening for as long as I can remember (which may just be
12 months ;-)).
Has anyone seen this behavior? Is it my test box? Are there known
problems with parallel checks?
FWIW, I'm running on an x86-64 Fedora 25 running a 4.9.6-200.fc25.x86_64
kernel. The machine has 56 cores, and has dejagnu 1.6 installed.
Thanks.
Aldy
p.s. Attached is the full output from contrib/compare_tests.
# Comparing directories
## Dir1=/home/cygnus/aldyh/bld/pristine/: 8 sum files
## Dir2=.: 8 sum files
# Comparing 8 common sum files
## /bin/sh /home/cygnus/aldyh/src/gcc/contrib/compare_tests
/tmp/gxx-sum1.39318 /tmp/gxx-sum2.39318
New tests that FAIL:
g++.dg/tree-prof/pr79259.C execution, -g
gcc.dg/tree-prof/ic-misattribution-1.c execution, -g
gcc.dg/tree-prof/merge_block.c execution, -g
gcc.dg/tree-prof/section-attr-1.c execution, -g
gcc.dg/tree-prof/section-attr-3.c execution, -g
gcc.dg/tree-prof/tracer-1.c execution, -g
gcc.dg/tree-prof/val-prof-8.c execution, -g
New tests that PASS:
g++.dg/tree-prof/partition1.C compilation, -g
g++.dg/tree-prof/partition2.C compilation, -g
g++.dg/tree-prof/partition3.C compilation, -g
g++.dg/tree-prof/pr79259.C compilation, -g
gcc.dg/tree-prof/comp-goto-1.c compilation, -g
gcc.dg/tree-prof/ic-misattribution-1.c compilation, -g
gcc.dg/tree-prof/merge_block.c compilation, -g
gcc.dg/tree-prof/pr47187.c compilation, -g
gcc.dg/tree-prof/pr52150.c compilation, -g
gcc.dg/tree-prof/section-attr-1.c compilation, -g
gcc.dg/tree-prof/stringop-1.c compilation, -g
gcc.dg/tree-prof/tracer-1.c compilation, -g
gcc.dg/tree-prof/tracer-1.c scan-tree-dump-times tracer "c =" 2
gcc.dg/tree-prof/update-cunroll-2.c compilation, -g
gcc.dg/tree-prof/val-prof-8.c compilation, -g
gcc.dg/tree-prof/wcoverage-mismatch.c compilation, -g
gcc.dg/tree-ssa/20030807-6.c scan-tree-dump-times vrp1 "ABS_EXPR " 0
gcc.dg/tree-ssa/pr18589-6.c scan-tree-dump-times optimized " \\* " 4
Old tests that passed, that have disappeared: (Eeek!)
g++.dg/tree-prof/indir-call-prof-2.C compilation, -g
g++.dg/tree-prof/morefunc.C compilation, -g
g++.dg/tree-prof/pr63581.C compilation, -g
gcc.dg/tree-prof/20050826-2.c compilation, -g
gcc.dg/tree-prof/bb-reorg.c compilation, -g
gcc.dg/tree-prof/cold_partition_label.c compilation, -g
gcc.dg/tree-prof/ic-misattribution-1a.c compilation, -g
gcc.dg/tree-prof/inliner-1.c compilation, -g
gcc.dg/tree-prof/peel-1.c compilation, -g
gcc.dg/tree-prof/pr49299-1.c compilation, -g
gcc.dg/tree-prof/pr49299-2.c compilation, -g
gcc.dg/tree-prof/pr52027.c compilation, -g
gcc.dg/tree-prof/pr79587.c compilation, -g
gcc.dg/tree-prof/prof-robust-1.c compilation, -g
gcc.dg/tree-prof/section-attr-2.c compilation, -g
gcc.dg/tree-prof/stringop-2.c compilation, -g
gcc.dg/tree-prof/time-profiler-1.c compilation, -g
gcc.dg/tree-prof/time-profiler-3.c compilation, -g
gcc.dg/tree-prof/unroll-1.c compilation, -g
gcc.dg/tree-prof/val-prof-6.c compilation, -g
gcc.dg/tree-prof/val-prof-7.c compilation, -g
gcc.dg/tree-prof/val-prof-9.c compilation, -g
gcc.dg/tree-ssa/20030530-2.c scan-tree-dump-times dom2 "\\+" 1
gcc.dg/tree-ssa/20030530-2.c scan-tree-dump-times dom2 "first_gp_reg_save" 1
gcc.dg/tree-ssa/20030730-1.c scan-tree-dump-times dom2 "if " 0
gfortran.dg/vect/vect-1.f90 -O scan-tree-dump-times vect "vectorized 3
loops" 1
Old tests that failed, that have disappeared: (Eeek!)
g++.dg/tree-prof/inline_mismatch_args.C execution, -g
g++.dg/tree-prof/pr63581.C execution, -g
gcc.dg/tree-prof/bb-reorg.c execution, -g
gcc.dg/tree-prof/cold_partition_label.c execution, -g
gcc.dg/tree-prof/peel-1.c execution, -g
gcc.dg/tree-prof/pr34999.c execution, -g
gcc.dg/tree-prof/pr49299-1.c execution, -g
gcc.dg/tree-prof/pr50907.c execution, -g
gcc.dg/tree-prof/pr52027.c execution, -g
gcc.dg/tree-prof/pr79587.c execution, -g
gcc.dg/tree-prof/time-profiler-1.c execution, -g
gcc.dg/tree-prof/time-profiler-3.c execution, -g
gcc.dg/tree-prof/val-prof-6.c execution, -g
gcc.dg/tree-prof/val-prof-7.c execution, -g
gcc.dg/tree-prof/val-profiler-threads-1.c execution, -g
## Differences found:
# 1 differences in 8 common sum files found
