On Mon, Aug 24, 2015 at 05:37:43PM -0700, via lldb-dev wrote:
> On Mon, Aug 24, 2015 at 03:37:52PM -0700, Todd Fiala via lldb-dev wrote:
> > On Linux on non-virtualized hardware, I currently see the failures below on
> > Ubuntu 14.04.2 using a setup like this:
> > [...]
> >
> > ninja check-lldb output:
FYI, ninja check-lldb actually calls dosep.
> > Ran 394 test suites (15 failed) (3.807107%)
> > Ran 474 test cases (17 failed) (3.586498%)
>
> I don't think you can trust the reporting of dosep.py's "Ran N test
> cases", as it fails to count about 500 test cases. The only way I've
> found to get an accurate count is to add up all the Ns from "Ran N tests
> in" as follows:
>
> ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee
> test_out.log
> export total=`grep -E "^Ran [0-9]+ tests? in" test_out.log | awk '{count+=$2}
> END {print count}'`
Of course, these commands assume you're running the tests from the lldb/test
directory.
> (See comments in http://reviews.llvm.org/rL238467.)
I've pasted (and tweaked) the relavent comments from that review here, where I
describe a narrowed case showing how dosep fails to count all the test cases
from one test suite in test/types. Note that the tests were run on OSX, so
your counts may vary.
The final count from:
Ran N test cases .*
is wrong, as I'll explain below. I've done a comparison between dosep and
dotest on a narrowed subset of tests to show how dosep can omit the test cases
from a test suite in its count.
Tested on subset of lldb/test with just the following directories/files (i.e.
all others directories/files were removed):
test/make
test/pexpect-2.4
test/plugins
test/types
test/unittest2
# The .py files kept in test/types are as follows (so
test/types/TestIntegerTypes.py* was removed):
test/types/AbstractBase.py
test/types/HideTestFailures.py
test/types/TestFloatTypes.py
test/types/TestFloatTypesExpr.py
test/types/TestIntegerTypesExpr.py
test/types/TestRecursiveTypes.py
Tests were run in the lldb/test directory using the following commands:
dotest:
./dotest.py -v
dosep:
./dosep.py -s --options "-v"
Comparing the test case totals, dotest correctly counts 46, but dosep counts
only 16:
dotest:
Ran 46 tests in 75.934s
dosep:
Testing: 23 tests, 4 threads ## note: this number changes randonmly
Ran 6 tests in 7.049s
[PASSED TestFloatTypes.py] - 1 out of 23 test suites processed
Ran 6 tests in 11.165s
[PASSED TestFloatTypesExpr.py] - 2 out of 23 test suites processed
Ran 30 tests in 54.581s ## FIXME: not counted?
[PASSED TestIntegerTypesExpr.py] - 3 out of 23 test suites processed
Ran 4 tests in 3.212s
[PASSED TestRecursiveTypes.py] - 4 out of 23 test suites processed
Ran 4 test suites (0 failed) (0.000000%)
Ran 16 test cases (0 failed) (0.000000%)
With test/types/TestIntegerTypesExpr.py* removed, both correctly count 16 test
cases:
dosep:
Testing: 16 tests, 4 threads
Ran 6 tests in 7.059s
Ran 6 tests in 11.186s
Ran 4 tests in 3.241s
Ran 3 test suites (0 failed) (0.000000%)
Ran 16 test cases (0 failed) (0.000000%)
Note: I couldn't compare the test counts on all the tests because of the
concern raised in http://reviews.llvm.org/rL237053. That is, dotest can no
longer complete the tests on OSX, as all test suites fail after test case 898:
test_disassemble_invalid_vst_1_64_raw_data get ERRORs. I don't think that issue
is related to problems in dosep.
Thanks,
-Dawn
_______________________________________________
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
