On Tue, Feb 6, 2018 at 7:09 AM, Pavel Labath <lab...@google.com> wrote: > On 6 February 2018 at 04:11, Davide Italiano via lldb-dev > <lldb-dev@lists.llvm.org> wrote: >> Hi, >> in the last couple of months a lot of people put a lot of attentions >> and energy into lldb and we're starting to see the first results. I >> decided to sit down and write this e-mail to state where we are and >> what are some possible future directions for the projects, in terms of >> better quality/higher testability. >> >> Current state: >> >> 1) We got the testsuite on MacOS to build with zero unexpected >> successes and zero failures (modulo one change I'm going to push >> tomorrow). This is a collaborative effort and it's very important >> because it allows us to push for unexpected successes as failures on >> the bots, allowing us to discover issues quicker. Other platform I >> think are improving their state as well, mainly thanks to the work of >> Pavel and Jan. > > I don't mean to belittle this statement, as I think the situation has > definitely improved a lot lately, but I feel I have to point out > that's I've never been able to get a clean test suite run on a mac > (not even the "0 failures" kind of clean). I'm not sure what are these > caused by, but I guess that's because the tests are still very much > dependent on the environment. So, I have to ask: what kind of > environment are you running those tests in? > > My machine is not a completely off-the-shelf mac, as it has some > google-specific customizations. I don't really know what this > encompasses, but I would be surprised if these impact the result of > the test suite. If I had to bet, my money would be on your machines > having some apple-specific stuff which is not available in regular > macs. > > I tried an experiment today. I configured with: cmake ../../llvm > -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON -GNinja. First > problem I ran into was that I couldn't run check-lldb, as the clang I > have just built was unable to compile any of the test binaries due to > missing headers (this could be a manifestation of the SDKROOT issue we > ran into a couple of weeks ago). So, I tried running with the system > compiler and I got this output: > > FAIL: test_c_global_variables_dwarf > (lang/c/global_variables/TestGlobalVariables.py) > FAIL: test_c_global_variables_gmodules > (lang/c/global_variables/TestGlobalVariables.py) > FAIL: test_dsym (functionalities/ubsan/basic/TestUbsanBasic.py) > FAIL: test_dwarf (functionalities/ubsan/basic/TestUbsanBasic.py) > FAIL: test_gmodules (functionalities/ubsan/basic/TestUbsanBasic.py) > FAIL: test_with_python_api_dsym (lang/cpp/class_static/TestStaticVariables.py) > FAIL: test_with_python_api_dwarf > (lang/cpp/class_static/TestStaticVariables.py) > FAIL: test_with_python_api_gmodules > (lang/cpp/class_static/TestStaticVariables.py) > ERROR: test_debug_info_for_apple_types_dsym > (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py) > ERROR: test_debug_info_for_apple_types_dwarf > (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py) > ERROR: test_debug_info_for_apple_types_gmodules > (macosx/debug-info/apple_types/TestAppleTypesIsProduced.py) > UNEXPECTED SUCCESS: test_lldbmi_output_grammar > (tools/lldb-mi/syntax/TestMiSyntax.py) > UNEXPECTED SUCCESS: test_process_interrupt_dsym > (functionalities/thread/state/TestThreadStates.py) > UNEXPECTED SUCCESS: test_process_interrupt_gmodules > (functionalities/thread/state/TestThreadStates.py) > > So, I guess my question is: are you guys looking into making sure that > others are also able to reproduce the 0-fail+0-xpass state? I would > love to run the mac test suite locally, as I tend to touch a lot of > stuff that impacts all targets, but as it stands now, I have very > little confidence that the test I am running reflect in any way the > results you will get when you run the test on your end. > > I am ready to supply any test logs or information you need if you want > to try to tackle this. >
Yes, I'm definitely interested in making the testusuite working/reliable on any configuration. I was afraid there were a lot of latent issues, that's why I sent this mail in the first place. It's also the reason why I started thinking about `lldb-test` as a driver for testing, because I found out the testsuite being a little inconsistent/brittle depending on the environment it's run on (which, FWIW, doesn't happen when you run lit/FileCheck or even the unit tests in lldb). I'm not currently claiming switching to a different method would improve the situation, but it's worth a shot. >> 3) In the short term I plan to remove support for unmaintained >> languages (Java/Go/Swift). This allows us to bring them back again (or > I hope you meant OCaml instead of Swift. :P Oh, yes, sigh. _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
