The last time this discussion came up (Sep 2016), I pointed out that a great approach here would be to use the lldb-mi driver. The MI (Machine Interface) is a JSON-like debugger UI which closely models how people use the lldb command line driver. It was written at Cygnus (long before JSON, hence the "-like") to allow for the separation of an IDE and a debugger. All of the responses from lldb-mi are structured output; they can be ingested and queried programmatically so tests are stable.
This topic has come up many times, and I know there's a real desire to say "I did these five commands & saw a problem; I just want to write a test case that does those five commands and checks for the expected output." Going from lldb command interpreter to lldb-mi command interpreter is not a big leap once you've seen how MI is structured; this may be easier for casual contributors to write. I think it would be interesting to write a test harness (based on lit?) to write tests using lldb-mi as the driver. Once people spend more time working on lldb, the SB API quickly becomes second nature so writing tests in that form is even easier. As Jim notes, we learned a very painful lesson with gdb which was based on output matching; it was fine for a while but after a couple of decades, it made it extremely difficult to change the command line tool's behavior in any way because you knew you'd be fixing ornate regular expressions that had accumulated over the years for 4x as long as it took to write the original change. It was miserable. J > On Nov 29, 2017, at 1:59 PM, Jim Ingham via lldb-commits > <lldb-commits@lists.llvm.org> wrote: > > I'm mostly basing this concern on the bad effect this had on gdb all of whose > testing was expect based command scraping. gdb is a tool that's much closer > to lldb than any of the compiler tools so that experience seems relevant. > It's been a decade or so since I worked on gdb, but back when I was working > on it, you really had to tread very carefully if you wanted to change the > output of, say, the break command, or to thread listings, etc, and a bunch of > times I just had to bag some cleanup of output I wanted to do because fixing > up all the tests was too time consuming. Because Jason and I had both had > this experience when we started working on lldb, we promised ourselves we > wouldn't go down this path again... > > Jim > > >> On Nov 29, 2017, at 1:43 PM, Davide Italiano <dccitali...@gmail.com> wrote: >> >> On Wed, Nov 29, 2017 at 1:38 PM, Jim Ingham <jing...@apple.com> wrote: >>> I'm a little confused by your response. >>> >>> My stated objection to command output dependent tests is and has always >>> been that they make the test dependent on the details of command output. >>> Over time doing so makes it hard to modify command output. This is sort of >>> related to interactivity, in the sense that since lldb is an interactive >>> tool with lots of different commands producing different reports of >>> information we can gather, the desire to improve and modify that output is >>> more present than in tools that are less output dependent or whose output >>> is meant to be processed, in which case it really is API. Command output >>> for lldb is explicitly NOT API, that's why we have a real API for people >>> who want to program lldb... So the bad effect of the tests in calcifying >>> this output is an issue for lldb where it may not be for other tools. >>> >> >> Hi Jim, >> in my experience command output changes can be automated via `sed/grep/awk`. >> I'm responsible (and many others are) for fundamental changes in LLVM >> tools output (i.e. typeless pointers changed pretty much every >> load/store/memory_op* in tree) and I found out changing the output of >> tests isn't that troublesome. I'm not yet very familiar with LLDB, so >> the story might be different here. >> I'm personally in favour of this approach because it scaled very well >> in llvm (and lld, FWIW), with many more tests than lldb has. >> >> -- >> Davide > > _______________________________________________ > lldb-commits mailing list > lldb-commits@lists.llvm.org > http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits _______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
