> On Jan 24, 2021, at 8:30 PM, David Blaikie <dblai...@gmail.com> wrote: > > I guess lldb doesn't have a machine readable form, like gdb's machine > interface that might make for a more robust thing to test against most of the > time (& then leaving a limited number of tests that test the user-textual > output)? Instead the python API is the machine interface?
That was the original idea, except that we also tried to make it easy to test command line output in the same tests that test the API's. Really the only part of the system the API tests aren't great at is stuff that depends on mid-command interactivity (prompts, line-editing interactions, the REPL, stuff like that). The difference between the two to my mind is pretty much that the Shell tests don't require you to know Python, or learn the SB API's to write tests. > > Another aspect of end-to-endness is being able to test a specific > component of lldb, instead of just the debugger as a whole. Here the API > tests cannot help because the "API" is the lldb public API. > > Not sure I followed here - you mean the API tests aren't more narrowly > targeted than the Shell tests, because the API is the public API, so it's > mostly/pretty close to what you can interact with from the Shell anyway - > doesn't give you lower-level access akin to unit testing? Fair enough. > Yes, this is particularly true for lldb's plugins. There isn't any really compelling reason for an lldb user to know the details of how DWARF was ingested. At that level, lldb should present the data from the generic Symbol interface, not how it came from DWARF. So if you want to test the DWARF parser specifically then UnitTests are the way to go. And for testing utility classes which don't require lots of other subsystems to construct themselves, the UnitTests are much easier to write. These also are easy to migrate as code changes because we know how to do "I added a parameter to an API, go fix the calling code" pretty well. My take is that API tests will always be more precise because each time you get a bit of data that you want to test against you get only that data, you don't get it intermixed with other text that you have to decide how to either capture (and maybe end up testing things you didn't intend to test) or run the risk that the patterns you were using to grab the data you thought you were testing actually grabbed a different part of the command text. The solution to that if you wanted to do more rigorous tests from the command line is to write special purpose commands that present output in a form that's not good for users but is better structured for testing. But that seems like wasted effort and support burden when the API's to get what you want are right there in front of you... Jim _______________________________________________ lldb-commits mailing list lldb-commits@lists.llvm.org https://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
