> On Sep 19, 2016, at 3:19 PM, Zachary Turner <ztur...@google.com> wrote: > > Obviously I defer to you on whether testing via the SB API is better than > what GDB does or used to do. But these are not the only two systems in the > world. Having used both LLDB and LLVM's test suite extensively, I still > remain unconvinced that LLDB's testing situation could not be improved. Do > we improve it by doing what GDB did? Obviously not. Can we improve it > further by doing something completely different than what GDB did *and* what > LLDB currently does? I remain convinced that we can.
I think you misread my message. I said: Having a Python interface so that developers can program the debugger is very powerful. As evidence, see all the work the gdb folks did to add one (you have to know a little bit about how gdb works to see how hard this would be, but you can imagine a debugger written in C primarily to dump text to a terminal, think of how you would wrap that in Python and you'll get a sense.) Now forget about gdb, that was only an example of how much some folks thought a scripting interface was worth. Instead, focus on this good thing, the scripting interface. We should test this thing that we think is valuable. What's a good way to make sure we test that? USE IT FOR TESTING. Jim > > > On Mon, Sep 19, 2016 at 3:07 PM Jim Ingham <jing...@apple.com> wrote: > > > On Sep 19, 2016, at 2:11 PM, Zachary Turner via lldb-dev > > <lldb-dev@lists.llvm.org> wrote: > > > > > > > > On Mon, Sep 19, 2016 at 2:02 PM Greg Clayton <gclay...@apple.com> wrote: > > > > > • Separate testing tools > > > • One question that remains open is how to > > > represent the complicated needs of a debugger in lit tests. Part a) > > > above covers the trivial cases, but what about the difficult cases? In > > > https://reviews.llvm.org/D24591 a number of ideas were discussed. We > > > started getting to this idea towards the end, about a separate tool which > > > has an interface independent of the command line interface and which can > > > be used to test. lldb-mi was mentioned. While I have serious concerns > > > about lldb-mi due to its poorly written and tested codebase, I do agree > > > in principle with the methodology. In fact, this is the entire > > > philosophy behind lit as used with LLVM, clang, lld, etc. > > > > We have a public API... Why are we going to go and spend _any_ time on > > doing anything else? I just don't get it. What a waste of time. We have a > > public API. Lets use it. Not simple enough for people? Tough. Testing a > > debugger should be done using the public API except when we are actually > > trying to test the LLDB command line in pexpect like tests. Those and only > > those are fine to covert over to using LIT and use text scraping. But 95% > > of our testing should be done using the API that our IDEs use. Using MI? > > Please no. > > > > FWIW, I agree with you about mi. > > > > That said, the public api is problematic. Here you've got an API which, > > once you do something to, is set in stone forever. And yet in order to > > test something, you have to add it to the API. So now, in order to test > > something, you have to make a permanent change to a public API that can > > never be undone. > > > > It's also a public facing API, when a lot of the stuff we need to be able > > to look at and inspect is not really suitable for public consumption. > > > > If something is a public API that can never be changed once it's added, > > then there should be an extremely strict review process in order to add > > something to it. It should be VERY HARD to modify (it's currently not, > > which is a debate for another day, but anyway...). And yet that's > > fundamentally incompatible with having tests be easy to write. When it's > > hard to add the thing you need to add in order to write the test, then it's > > hard to write the test. > > > > Furthermore, with a system such as the one I proposed, you could even run > > tests with LLDB_DISABLE_PYTHON. IDK about you, but that would be *amazing* > > from my point of view. Even though I spent months making Python 3 work, I > > would be happy to burn it all with fire and go back to just Python 2 if we > > had a viable testing strategy. > > It would not be amazing. One of the primary benefits of lldb IS the Python > API. If you don't think that's wonderful in a debugger, go look at what > effort it took to make that happen in gdb once lldb started providing it and > you'll see that it was really highly motivated. So switching our testing > away from one of our very strong points would be a very bad choice IMHO. > > On the "should I add it to the SB API to test it or not question." Most of > the time, when I find I need to add an API in order write a test for > something, the thing I'm trying to get my hands on is something that other > people might be interested in as well. If we go to the mode of adding > affordances only for ourselves when writing for testing, we remove one of the > times you should be asking yourself "should I add this to the SB API so other > people can use it." I'm fine with having an lldb-testing module with API's > based on the same methodology as the SB API's. If you think something you > are adding is too esoteric to be part of the SB API put it there. But by > default put it into the SB API's. > > Jim > > > > _______________________________________________ > > lldb-dev mailing list > > lldb-dev@lists.llvm.org > > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev > _______________________________________________ lldb-dev mailing list lldb-dev@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev
