And of course, one such discipline would be "only use SB API's"...
Jim > On Nov 29, 2017, at 5:00 PM, Jim Ingham <jing...@apple.com> wrote: > > I thought we weren't talking about why you want this lldb-test, but rather > why tests that poke deep into the lldb_private API's are or are not > appropriate for your test dingus. > > The thing I worry about is if you start using this for very special purpose > "reach deep into the lldb_private API's" type tests, then the command-set of > lldb-test is not going to be a limited set of commands that are nicely > composible, it is going to be a whole assortment of odd little one-off > commands and trying to figure out how to use it is going to get harder and > harder over time because it will be so noisy. So if I were doing this I'd > impose some discipline on myself to keep that from happening, and use unit > tests for anything that was really esoteric. > > But again, I'm unlikely to be the one that implements this so my opinions > should have the weight of a kibitzer... > > Jim > > >> On Nov 29, 2017, at 4:33 PM, Zachary Turner <ztur...@google.com> wrote: >> >> Many reasons. >> >> 1) a test that looks like this: >> >> ``` >> RUN: yaml2obj %p/Inputs/compressed-section.yaml > %t/compressed-section.obj >> RUN: lldb-test %s | FileCheck %s >> >> create-module -f %t/compressed-section.obj >> dump-module-sections compressed-section.obj >> >> ; CHECK: Section: .hello_elf >> ; CHECK-NEXT: Size: 9 >> ; CHECK-NEXT: Data: Hello ELF >> >> ; CHECK: Section: .bogus >> ; CHECK-NEXT: Size: 0 >> ``` >> >> takes about 1 minute to write, and a couple of seconds to understand, and is >> 12 lines of code, compared to a unit test which takes significantly longer >> to write and understand, and is (at least in this case) 37 lines not >> counting the code to implement the command >> >> 2) this is a standard test format that is understood by hundreds of >> developers. >> >> 3) It does not contribute to build time. >> >> 4) Any work you do to implement the `create-module` and >> `dump-module-sections` testing command can later be used by any other test >> without any additional code. With a reasonable set of commands you would >> reach a point where you were rarely having to update the tool. >> >> 5) Parsing text output is often the most straightforward and easiest way to >> verify something. Consider for example a testcommand like >> "dump-unwind-plan" where you could easily represent an unwind plan in text >> as a sequence of rules and/or heuristics. Try doing that in code, for >> several hundred different test cases of obscure unwind scenarios and I think >> you'll quickly give up and decide it's more trouble than it's worth. >> >> On Wed, Nov 29, 2017 at 4:14 PM Jim Ingham <jing...@apple.com> wrote: >> >> >>> On Nov 29, 2017, at 3:46 PM, Zachary Turner <ztur...@google.com> wrote: >>> >>> FWIW, it can certainly use the SB API where it makes sense, but I think >>> requiring that it only use the SB API would be very limiting and a big >>> mistake. >>> >>> The entire point of a tool such as this is that it allows you to dig deep >>> into internals that would be difficult to access otherwise. >> >> I'm not sure about that. Making a test that "digs deep into internals" in >> this method is almost certainly going to require writing a custom command in >> lldb-test to poke those API's. How would this be any easier than writing a >> unit test? >> >> Jim >> >> >>> >>> On Wed, Nov 29, 2017 at 3:23 PM Jason Molenda <jmole...@apple.com> wrote: >>> >>> >>>> On Nov 29, 2017, at 2:51 PM, Zachary Turner via lldb-commits >>>> <lldb-commits@lists.llvm.org> wrote: >>>> >>>> >>>> >>>> On Wed, Nov 29, 2017 at 1:59 PM Jim Ingham <jing...@apple.com> wrote: >>>> I'm mostly basing this concern on the bad effect this had on gdb all of >>>> whose testing was expect based command scraping. gdb is a tool that's >>>> much closer to lldb than any of the compiler tools so that experience >>>> seems relevant. It's been a decade or so since I worked on gdb, but back >>>> when I was working on it, you really had to tread very carefully if you >>>> wanted to change the output of, say, the break command, or to thread >>>> listings, etc, and a bunch of times I just had to bag some cleanup of >>>> output I wanted to do because fixing up all the tests was too time >>>> consuming. Because Jason and I had both had this experience when we >>>> started working on lldb, we promised ourselves we wouldn't go down this >>>> path again... >>>> >>>> >>>> Couple of things: >>>> >>>> 1) I wouldn't dare to use this approach for anything that required >>>> interactivity. If you need to run one command, extract a value from the >>>> output, and use that value as input to another command, I think that would >>>> be a big mistake. I have no intention of ever proposing something like >>>> that. >>>> >>>> 2) FileCheck is very flexible in the way it matches output and tests can >>>> be written so that they are resilient to minor format tweaks. I have no >>>> doubt that with pure regex matching, or with pretty much any other tool, >>>> you would have a really bad time. Of course, that doesn't mean it would >>>> be hard to construct an example of a format change that would break a >>>> FileCheck test. But I think it would happen *far* less frequently than it >>>> did on GDB. That said, I still understand your concerns that it's >>>> fragile, so... >>>> >>>> 3) I would not be opposed to a tool called lldb-test, which was basically >>>> just LLDB with a different, and much more limited set of commands, and was >>>> completely non-interactive and would produce output in a format designed >>>> for being scraped, and which never had to be changed since it was never >>>> presented to the user. >>> >>> >>> 100% agree with #3. We could go back and forth about using lldb-mi, but I >>> think a specialized driver using SB API, designed for testing, would be a >>> great approach. >>> >>> >> > _______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
