On Thu, Sep 25, 2014 at 08:22:29AM -0400, Andrew MacLeod wrote: > So to be fair, I could use test_summary, but I think the concern is > warranted because if this inconsistent ordering can happen to PASS, I > would expect the same non-deterministic behaviour if those tests happen > to FAIL. we just have far less FAILS so we aren't seeing it with > test_summary at the moment... > > Aggregating all my .sum files, I see a sampling of about 257,000 PASSs, > whereas I see a total of 141 FAILs. FAILs only account for < 0.06% of > the output. ( I'm getting an average of about 510 mis-ordered PASSs, so > it only affects a small portion of them as well.)
0.24% here (2241 FAILs, 917715 PASSes). You're seeing about 1 in 500 misordered, so if it was independent (which of course it is not) I should see it in the FAILs already. > I would think the output of .sum needs to be consistent from one run to > the next in order for test_summary to consistently report its results as > well. Yes. There also is the problem of the summaries being messed up (which they were already before the parallelisation changes, but now the result is much worse). I'll have another look. Segher
