On Mon, Aug 24, 2015 at 05:37:43PM -0700, via lldb-dev wrote: > On Mon, Aug 24, 2015 at 03:37:52PM -0700, Todd Fiala via lldb-dev wrote: > > On Linux on non-virtualized hardware, I currently see the failures below on > > Ubuntu 14.04.2 using a setup like this: > > [...] > > > > ninja check-lldb output:
FYI, ninja check-lldb actually calls dosep. > > Ran 394 test suites (15 failed) (3.807107%) > > Ran 474 test cases (17 failed) (3.586498%) > > I don't think you can trust the reporting of dosep.py's "Ran N test > cases", as it fails to count about 500 test cases. The only way I've > found to get an accurate count is to add up all the Ns from "Ran N tests > in" as follows: > > ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee > test_out.log > export total=`grep -E "^Ran [0-9]+ tests? in" test_out.log | awk '{count+=$2} > END {print count}'` Of course, these commands assume you're running the tests from the lldb/test directory. > (See comments in http://reviews.llvm.org/rL238467.) I've pasted (and tweaked) the relavent comments from that review here, where I describe a narrowed case showing how dosep fails to count all the test cases from one test suite in test/types. Note that the tests were run on OSX, so your counts may vary. The final count from: Ran N test cases .* is wrong, as I'll explain below. I've done a comparison between dosep and dotest on a narrowed subset of tests to show how dosep can omit the test cases from a test suite in its count. Tested on subset of lldb/test with just the following directories/files (i.e. all others directories/files were removed): test/make test/pexpect-2.4 test/plugins test/types test/unittest2 # The .py files kept in test/types are as follows (so test/types/TestIntegerTypes.py* was removed): test/types/AbstractBase.py test/types/HideTestFailures.py test/types/TestFloatTypes.py test/types/TestFloatTypesExpr.py test/types/TestIntegerTypesExpr.py test/types/TestRecursiveTypes.py Tests were run in the lldb/test directory using the following commands: dotest: ./dotest.py -v dosep: ./dosep.py -s --options "-v" Comparing the test case totals, dotest correctly counts 46, but dosep counts only 16: dotest: Ran 46 tests in 75.934s dosep: Testing: 23 tests, 4 threads ## note: this number changes randonmly Ran 6 tests in 7.049s [PASSED TestFloatTypes.py] - 1 out of 23 test suites processed Ran 6 tests in 11.165s [PASSED TestFloatTypesExpr.py] - 2 out of 23 test suites processed Ran 30 tests in 54.581s ## FIXME: not counted? [PASSED TestIntegerTypesExpr.py] - 3 out of 23 test suites processed Ran 4 tests in 3.212s [PASSED TestRecursiveTypes.py] - 4 out of 23 test suites processed Ran 4 test suites (0 failed) (0.000000%) Ran 16 test cases (0 failed) (0.000000%) With test/types/TestIntegerTypesExpr.py* removed, both correctly count 16 test cases: dosep: Testing: 16 tests, 4 threads Ran 6 tests in 7.059s Ran 6 tests in 11.186s Ran 4 tests in 3.241s Ran 3 test suites (0 failed) (0.000000%) Ran 16 test cases (0 failed) (0.000000%) Note: I couldn't compare the test counts on all the tests because of the concern raised in http://reviews.llvm.org/rL237053. That is, dotest can no longer complete the tests on OSX, as all test suites fail after test case 898: test_disassemble_invalid_vst_1_64_raw_data get ERRORs. I don't think that issue is related to problems in dosep. Thanks, -Dawn _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev