On 6/7/11, Diego Novillo <[email protected]> wrote:
> After getting new failures due to an unrelated fix, I think this will
> be more trouble than it's worth.
>
> First, we can't get rid of the XPASSes, so those will always be noisy.
Remove all the noise with:
sed -e '
/-fpph-map=pph.map/ ! {
/^XPASS: .*test for bogus messages/ d
/^XPASS: .*test for excess errors/ d
}
/^#/ p
/^ERROR: / p
/^XFAIL: / p
/^XPASS: / p
/^FAIL: / p
d
'
>
> Second, some XPASSes will need to be unmarked because we just fixed
> the underlying problem.
That is how we know that what progress we have made. These should
be reflected in the work list.
>
> Third, we are at such an early stage, that fixes to a test case will
> generally expose failures in other already failing tests, but these
> failures will be in a different place. So more noise.
That is now we expose remaining tasks. These should be reflected
in the work list.
>
> I really think that for now the easiest way to keep track of this is
> to have a clean build to compare against.
But the clean build isn't a decent comparison. The existing tools
only compare against whether or not tests fail. They fail to
report that a test failed for different reasons. Given that there
are only twelve tests for which there is a concern, I think we are
better off know where we are.
--
Lawrence Crowl
