Thanks for the details on dosep.py, Dawn.
For counting I will probably go back to my old method of parsing the output
of a serial dotest run, since IIRC I can get skip counts accurately there
as well. (Or perhaps that should be added to dosep.py, it's been a while
since I last heavily modified that script).
-Todd
On Mon, Aug 24, 2015 at 6:50 PM, <d...@burble.org> wrote:
> On Mon, Aug 24, 2015 at 05:37:43PM -0700, via lldb-dev wrote:
> > On Mon, Aug 24, 2015 at 03:37:52PM -0700, Todd Fiala via lldb-dev wrote:
> > > On Linux on non-virtualized hardware, I currently see the failures
> below on
> > > Ubuntu 14.04.2 using a setup like this:
> > > [...]
> > >
> > > ninja check-lldb output:
>
> FYI, ninja check-lldb actually calls dosep.
>
> > > Ran 394 test suites (15 failed) (3.807107%)
> > > Ran 474 test cases (17 failed) (3.586498%)
> >
> > I don't think you can trust the reporting of dosep.py's "Ran N test
> > cases", as it fails to count about 500 test cases. The only way I've
> > found to get an accurate count is to add up all the Ns from "Ran N tests
> > in" as follows:
> >
> > ./dosep.py -s --options "-v --executable $BLDDIR/bin/lldb" 2>&1 | tee
> test_out.log
> > export total=`grep -E "^Ran [0-9]+ tests? in" test_out.log | awk
> '{count+=$2} END {print count}'`
>
> Of course, these commands assume you're running the tests from the
> lldb/test directory.
>
> > (See comments in http://reviews.llvm.org/rL238467.)
>
> I've pasted (and tweaked) the relavent comments from that review here,
> where I describe a narrowed case showing how dosep fails to count all the
> test cases from one test suite in test/types. Note that the tests were run
> on OSX, so your counts may vary.
>
> The final count from:
> Ran N test cases .*
> is wrong, as I'll explain below. I've done a comparison between dosep and
> dotest on a narrowed subset of tests to show how dosep can omit the test
> cases from a test suite in its count.
>
> Tested on subset of lldb/test with just the following directories/files
> (i.e. all others directories/files were removed):
> test/make
> test/pexpect-2.4
> test/plugins
> test/types
> test/unittest2
> # The .py files kept in test/types are as follows (so
> test/types/TestIntegerTypes.py* was removed):
> test/types/AbstractBase.py
> test/types/HideTestFailures.py
> test/types/TestFloatTypes.py
> test/types/TestFloatTypesExpr.py
> test/types/TestIntegerTypesExpr.py
> test/types/TestRecursiveTypes.py
>
> Tests were run in the lldb/test directory using the following commands:
> dotest:
> ./dotest.py -v
> dosep:
> ./dosep.py -s --options "-v"
>
> Comparing the test case totals, dotest correctly counts 46, but dosep
> counts only 16:
> dotest:
> Ran 46 tests in 75.934s
> dosep:
> Testing: 23 tests, 4 threads ## note: this number changes randonmly
> Ran 6 tests in 7.049s
> [PASSED TestFloatTypes.py] - 1 out of 23 test suites processed
> Ran 6 tests in 11.165s
> [PASSED TestFloatTypesExpr.py] - 2 out of 23 test suites processed
> Ran 30 tests in 54.581s ## FIXME: not counted?
> [PASSED TestIntegerTypesExpr.py] - 3 out of 23 test suites
> processed
> Ran 4 tests in 3.212s
> [PASSED TestRecursiveTypes.py] - 4 out of 23 test suites processed
> Ran 4 test suites (0 failed) (0.000000%)
> Ran 16 test cases (0 failed) (0.000000%)
>
> With test/types/TestIntegerTypesExpr.py* removed, both correctly count 16
> test cases:
> dosep:
> Testing: 16 tests, 4 threads
> Ran 6 tests in 7.059s
> Ran 6 tests in 11.186s
> Ran 4 tests in 3.241s
> Ran 3 test suites (0 failed) (0.000000%)
> Ran 16 test cases (0 failed) (0.000000%)
>
> Note: I couldn't compare the test counts on all the tests because of the
> concern raised in http://reviews.llvm.org/rL237053. That is, dotest can
> no longer complete the tests on OSX, as all test suites fail after test
> case 898: test_disassemble_invalid_vst_1_64_raw_data get ERRORs. I don't
> think that issue is related to problems in dosep.
>
> Thanks,
> -Dawn
>
--
-Todd
_______________________________________________
lldb-dev mailing list
lldb-dev@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
