Just to make sure I'm reading these right: ========== Compiler: totclang Architecture: x86_64 ==========
UnexpectedSuccess TestMiInterpreterExec.MiInterpreterExecTestCase.test_lldbmi_settings_set_target_run_args_before (250/250 100.000000%) TestRaise.RaiseTestCase.test_restart_bug_with_dwarf (119/250 47.600000%) TestMiSyntax.MiSyntaxTestCase.test_lldbmi_process_output (250/250 100.000000%) TestInferiorAssert.AssertingInferiorTestCase.test_inferior_asserting_expr_dwarf (195/250 78.000000%) This is saying that running the tests with a top of tree clang, on x86_64, we see (for example): * test_lldbmi_settings_set_target_run_args_before() is always passing, * test_inferior_asserting_expr_dwarf() is always passing * test_restart_bug_with_dwarf() is failing more often than passing. This is incredibly useful for figuring out the true disposition of a test on different configurations. What method did you use to gather that data? On Tue, Sep 15, 2015 at 9:03 AM, Todd Fiala <todd.fi...@gmail.com> wrote: > Wow Tamas, this is perfect. Thanks for pulling that together! > > Don't worry about the bigger file. > > Thanks much. > > -Todd > > On Tue, Sep 15, 2015 at 8:56 AM, Tamas Berghammer <tbergham...@google.com> > wrote: > >> I created a new statistic what separates the data based on compiler and >> architecture and I also extended it to the last 250 builds on the Linux >> build bot. If you would like to see the build IDs for the different >> outcomes then let me know, because I have them collected out, but it is a >> quite big file. >> >> Tamas >> >> On Tue, Sep 15, 2015 at 3:37 PM Todd Fiala <todd.fi...@gmail.com> wrote: >> >>> On Tue, Sep 15, 2015 at 2:57 AM, Tamas Berghammer < >>> tbergham...@google.com> wrote: >>> >>>> Hi Todd, >>>> >>>> I attached the statistic of the last 100 test run on the Linux x86_64 >>>> builder ( >>>> http://lab.llvm.org:8011/builders/lldb-x86_64-ubuntu-14.04-cmake). The >>>> data might be a little bit noisy because of the actual test failures >>>> happening because of a temporary regression, but they should give you a >>>> general idea about what is happening. >>>> >>>> >>> Thanks, Tamas! I'll have a look. >>> >>> >>>> I will try to create a statistic where the results are displayed >>>> separately for each compiler and architecture to get a bit more detailed >>>> view, but it will take some time. If you want I can include the list of >>>> build numbers for all outcome, but it will be a very log list (currently >>>> only included for Timeout and Failure) >>>> >>>> >>> I'll know better when I have a look at what you provided. The hole I >>> see right now is we're not adequately dealing with unexpected successes for >>> different configurations. Any reporting around that is helpful. >>> >>> Thanks! >>> >>> >>>> Tamas >>>> >>>> On Mon, Sep 14, 2015 at 11:24 PM Todd Fiala via lldb-dev < >>>> lldb-dev@lists.llvm.org> wrote: >>>> >>>>> On an Ubuntu 14.04 x86_64 system, I'm seeing the following results: >>>>> >>>>> *cmake/ninja/clang-3.6:* >>>>> >>>>> Testing: 395 test suites, 24 threads >>>>> 395 out of 395 test suites processed - TestGdbRemoteKill.py >>>>> Ran 395 test suites (0 failed) (0.000000%) >>>>> Ran 478 test cases (0 failed) (0.000000%) >>>>> >>>>> Unexpected Successes (6) >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestConstVariables.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestEvents.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py >>>>> >>>>> >>>>> *cmake/ninja/gcc-4.9.2:* >>>>> >>>>> 395 out of 395 test suites processed - TestMultithreaded.py >>>>> Ran 395 test suites (1 failed) (0.253165%) >>>>> Ran 457 test cases (1 failed) (0.218818%) >>>>> Failing Tests (1) >>>>> FAIL: LLDB (suite) :: TestRegisterVariables.py >>>>> >>>>> Unexpected Successes (6) >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestDataFormatterSynth.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiBreak.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiGdbSetShow.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiInterpreterExec.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestMiSyntax.py >>>>> UNEXPECTED SUCCESS: LLDB (suite) :: TestRaise.py >>>>> >>>>> >>>>> I will look into those. I suspect some of them are compiler-version >>>>> specific, much like some of the OS X ones I dug into earlier. >>>>> -- >>>>> -Todd >>>>> _______________________________________________ >>>>> lldb-dev mailing list >>>>> lldb-dev@lists.llvm.org >>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev >>>>> >>>> >>> >>> >>> -- >>> -Todd >>> >> > > > -- > -Todd > -- -Todd
_______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
