Many reasons. 1) a test that looks like this:
``` RUN: yaml2obj %p/Inputs/compressed-section.yaml > %t/compressed-section.obj RUN: lldb-test %s | FileCheck %s create-module -f %t/compressed-section.obj dump-module-sections compressed-section.obj ; CHECK: Section: .hello_elf ; CHECK-NEXT: Size: 9 ; CHECK-NEXT: Data: Hello ELF ; CHECK: Section: .bogus ; CHECK-NEXT: Size: 0 ``` takes about 1 minute to write, and a couple of seconds to understand, and is 12 lines of code, compared to a unit test which takes significantly longer to write and understand, and is (at least in this case) 37 lines not counting the code to implement the command 2) this is a standard test format that is understood by hundreds of developers. 3) It does not contribute to build time. 4) Any work you do to implement the `create-module` and `dump-module-sections` testing command can later be used by any other test without any additional code. With a reasonable set of commands you would reach a point where you were rarely having to update the tool. 5) Parsing text output is often the most straightforward and easiest way to verify something. Consider for example a testcommand like "dump-unwind-plan" where you could easily represent an unwind plan in text as a sequence of rules and/or heuristics. Try doing that in code, for several hundred different test cases of obscure unwind scenarios and I think you'll quickly give up and decide it's more trouble than it's worth. On Wed, Nov 29, 2017 at 4:14 PM Jim Ingham <jing...@apple.com> wrote: > > > > On Nov 29, 2017, at 3:46 PM, Zachary Turner <ztur...@google.com> wrote: > > > > FWIW, it can certainly use the SB API where it makes sense, but I think > requiring that it only use the SB API would be very limiting and a big > mistake. > > > > The entire point of a tool such as this is that it allows you to dig > deep into internals that would be difficult to access otherwise. > > I'm not sure about that. Making a test that "digs deep into internals" in > this method is almost certainly going to require writing a custom command > in lldb-test to poke those API's. How would this be any easier than > writing a unit test? > > Jim > > > > > > On Wed, Nov 29, 2017 at 3:23 PM Jason Molenda <jmole...@apple.com> > wrote: > > > > > > > On Nov 29, 2017, at 2:51 PM, Zachary Turner via lldb-commits < > lldb-commits@lists.llvm.org> wrote: > > > > > > > > > > > > On Wed, Nov 29, 2017 at 1:59 PM Jim Ingham <jing...@apple.com> wrote: > > > I'm mostly basing this concern on the bad effect this had on gdb all > of whose testing was expect based command scraping. gdb is a tool that's > much closer to lldb than any of the compiler tools so that experience seems > relevant. It's been a decade or so since I worked on gdb, but back when I > was working on it, you really had to tread very carefully if you wanted to > change the output of, say, the break command, or to thread listings, etc, > and a bunch of times I just had to bag some cleanup of output I wanted to > do because fixing up all the tests was too time consuming. Because Jason > and I had both had this experience when we started working on lldb, we > promised ourselves we wouldn't go down this path again... > > > > > > > > > Couple of things: > > > > > > 1) I wouldn't dare to use this approach for anything that required > interactivity. If you need to run one command, extract a value from the > output, and use that value as input to another command, I think that would > be a big mistake. I have no intention of ever proposing something like > that. > > > > > > 2) FileCheck is very flexible in the way it matches output and tests > can be written so that they are resilient to minor format tweaks. I have > no doubt that with pure regex matching, or with pretty much any other tool, > you would have a really bad time. Of course, that doesn't mean it would be > hard to construct an example of a format change that would break a > FileCheck test. But I think it would happen *far* less frequently than it > did on GDB. That said, I still understand your concerns that it's fragile, > so... > > > > > > 3) I would not be opposed to a tool called lldb-test, which was > basically just LLDB with a different, and much more limited set of > commands, and was completely non-interactive and would produce output in a > format designed for being scraped, and which never had to be changed since > it was never presented to the user. > > > > > > 100% agree with #3. We could go back and forth about using lldb-mi, but > I think a specialized driver using SB API, designed for testing, would be a > great approach. > > > > > >
_______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
