zturner added a comment. In https://reviews.llvm.org/D24591#543242, @jingham wrote:
> Writing tests this way means we're going back to testing the command line > commands. That was what gdb did for the most part, and you ended up > terrified of making changes that might effect command output, not because the > change was hard but because knew you would have to go waste a day or your > life making annoying fixes to a whole bunch of tests. This happened in lldb > with the command line tests we did have, specifically with the breakpoint > reporting. So I had to go make functions for setting breakpoints and making > sure they got set correctly - and change all the tests to use them - so I > could change just one place when I added output to breakpoint reporting. > > We made a choice a while ago to favor tests using the SB API's for this > reason. Is the virtue of lit sufficient that we really want to go back on > this decision? I had been thinking about this the other day as well. An example that springs to mind that I personally encountered is when a test on Windows was trying to verify that the value of argc was 3. So it was trying to match "argc=3" in stack trace. Seems reasonable, except that the code was actually completely broken on Windows, and argc was pointing to junk memory, which was something like "argc=3239082340982", and the test was passing because this value happened to start with a 3. One idea might be to add a developer command to LLDB that is able to drill down much deeper to return specific values of interest instead of large blocks of text. For example, imagine commands like this: (lldb) print-dev stack-frame[0].params[argc] 3 (lldb) print-dev threads.count 7 (lldb) print-dev --hex threads[6].id 0x1234 Now imagine you can get every little nook and cranny of the process's state through a syntax such as this. As you know I've been a big supporter of testing the API since the beginning for the same reason you mention -- that it makes people too afraid to change the output format -- but to answer your question: I do think the virtue of lit is high enough that we find a way to solve the problem. The above idea is one possibility for reducing the fragility of command line tests, but if we brainstorm we can probably come up with others as well. It's certainly a lot of work, but I do think the benefits are worth it. https://reviews.llvm.org/D24591 _______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
