Eric Blake <ebl...@redhat.com> writes:
> On 09/05/2015 07:25 AM, Eric Blake wrote:
>>>
>>> Oh, and tests/qapi-schema/test-qapi.py is annoying - even when it exits
>>> non-zero, it does NOT cause a failure in 'make check-qapi-schema'.
>>
>> Rather, it does not leave an obvious stack trace or immediate non-zero
>> status; and 'make check-qapi-schema' only fails as a side effect if
>> later output happens to differ from expectations. I would have had a
>> much easier time chasing the problem if I'd had an obvious python stack
>> trace.
>
> Oh, I see what happened. It did capture a trace in .err; but since the
> code checks for differences in .out before .err, I didn't even realize
> there was a trace to look at.
Yes, and it has annoyed me, too. Perhaps we should do something like
this (entirely untested):
diff --git a/tests/Makefile b/tests/Makefile
index 92d682f..d622356 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -519,13 +519,12 @@ check-tests/test-qapi.py: tests/test-qapi.py
$(patsubst %, check-%, $(check-qapi-schema-y)): check-%.json:
$(SRC_PATH)/%.json
$(call quiet-command, PYTHONPATH=$(SRC_PATH)/scripts \
$(PYTHON) $(SRC_PATH)/tests/qapi-schema/test-qapi.py \
- $^ >$*.test.out 2>$*.test.err; \
+ $^ >$*.test.out 2>$*.test.raw-err; \
echo $$? >$*.test.exit, \
" TEST $*.out")
- @diff -q $(SRC_PATH)/$*.out $*.test.out
@# Sanitize error messages (make them independent of build directory)
- @perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.err | diff -q
$(SRC_PATH)/$*.err -
- @diff -q $(SRC_PATH)/$*.exit $*.test.exit
+ @perl -p -e 's|\Q$(SRC_PATH)\E/||g' $*.test.raw-err >$*.test.err
+ @diff -q $(SRC_PATH)/$*.out $*.test.out && diff -q $(SRC_PATH)/$*.err
$*.test.err && diff -q $(SRC_PATH)/$*.exit $*.test.exit
# Consolidated targets
