I have been thinking about something similar. Ildb-mi specifically I have
serious concerns about because the code itself is a bit of an abomination.
It has its own entire test suite, which also doesn't work very well.
Using a tool like lldb mi is very similar in spirit to how llvm already
uses lit
I thought about this more overnight and I'm more convinced that lit and lldb-mi
make a great pair. lldb-mi is a programmatic text format that isn't subject to
the whims of command-line UI design over the years; well tests written in terms
of MI will be resilient and stable. lldb-mi MUCH more c
One thing I wonder about. It seems like everyone is mostly on the same page
about command line output .
What about input? Would anyone have objections to a test which ran a few
commands to get the debugger into a particular state before doing something
to verify the output? Let's assume I'm waving
> On Sep 15, 2016, at 8:02 AM, Zachary Turner wrote:
>
>
> It sounds like your goal is also "tests have to use the SB API and no other
> API", which if so I think that's counterproductive. More productive, IMO,
> would be being open to any alternative that addresses the concerns you have
>
This revision was automatically updated to reflect the committed changes.
Closed by commit rL281651: [LIT] First pass of LLDB LIT support (authored by
cbieneman).
Changed prior to commit:
https://reviews.llvm.org/D24591?vs=71537&id=71546#toc
Repository:
rL LLVM
https://reviews.llvm.org/D245
beanz updated this revision to Diff 71537.
beanz added a comment.
- Removed the XFAIL for icc
- Added support for LLDB_TEST__COMPILER options
Assuming there are no objections, since the patch has been approved by two
reviewers I'll commit it this afternoon.
Thanks,
-Chris
https://reviews.llvm
lldb-commits to bcc, lldb-dev to cc.
> The biggest feature I see missing here is the ability to run tests
remotely. Remote debugging is the most important use case for our team, and
now we have a number of (experimental) bots running the remote test suite.
We want to make sure we can debug correct
On Wed, Sep 14, 2016 at 11:43 PM Jason Molenda wrote:
> If we want to add a testsuite runner which takes a source file, a place to
> put a breakpoint, the name of a variable to examine, and it runs through
> those in SB API, I'm all in favor. I don't know if that's going to add a
> lot of test c
labath added a comment.
Btw, I will be in the bay area from 3rd to 7th of October. Maybe we could sit
down and talk the design and requirements through in person? I've been hoping
to speak to some of you in person anyway, and this could be a good first topic
on the agenda...
https://reviews.l
labath added a comment.
The biggest feature I see missing here is the ability to run tests remotely.
Remote debugging is the most important use case for our team, and now we have a
number of (experimental) bots running the remote test suite. We want to make
sure we can debug correctly when the
If we want to add a testsuite runner which takes a source file, a place to put
a breakpoint, the name of a variable to examine, and it runs through those in
SB API, I'm all in favor. I don't know if that's going to add a lot of test
coverage to lldb, but I have no problem with such a thing.
My
It's great to make writing tests easier. We'd all love to have more tests.
If "writing tests easier" is "command line output scraping", that's only
hurting the project in the long term. I'm telling you this from years of
experience on a project where we did exactly this. It's great to have ex
Also, sb api and command line test are neither mutually exclusive nor the
only 2 options.
Sean mentioned the idea of a dsl for example. I mentioned the idea of a
command that is independent of the normal ui command line.
There are lots of possibilities if you look.
Also our current situation is
There's also a cost to *not* writing test cases, which is paid
proportionally to how difficult it is to write test cases.
That said, aren't the existing lldbinline tests an example of the behavior
you're objecting to? And that is a relatively recent addition which I have
never seen or heard of any
On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote:
> I don't blame you for being scared of command tests. I don't support their
> use in the current LLDB test suite either, for exactly the same reasons you
> and Jason have expressed. But I do think it's possible to come up with
> something
zturner accepted this revision.
zturner added a comment.
lgtm from my point of view after `LLDB_TEST_CXX_COMPILER` and
`LLDB_TEST_C_COMPILER` are added.
One question: Currently the tests appear to run all commands until the end of
the file, and then do 1 pass over the output through `FileCheck`
I actually like the idea of feeding command lines. Again, not when the
command you're running prints a huge block of text and you try to match it.
But for "get the debugger into the state where i can reproduce the bug",
and sometimes the step to actually repro it, I think it's hands down the
best.
I agree completely with tackling the easy stuff first.
That said, the easy stuff (probably >50% of the testsuite) doesn't even require
command-line interaction and is just of the form "stop here, run this one
expression, maybe print this variable using 'frame variab'e'"
I would argue that we can
I agree that if all the tests look like the ones in this review there will
be no way to test 100% of the things we currently test. At the same time,
I see this as just a beginning. A replacement for lldbinline but one which
is more extensible. To test things like you said like multi-threaded
ste
> On Sep 14, 2016, at 5:13 PM, Zachary Turner wrote:
>
> I'm only saying that we should have an open mind. Obviously there are
> (valid!) concerns. If we can't solve them then we can't solve them. The
> goal (my goal anyway) is always to make things better, not to use X because
> it's X.
Also, you mentioned that this is very similar to the lldbinline tests. For
that reason, I would actually propose dropping lldbinline tests in favor of
this. If they are essentially the same, then it seems better to have 2
types of tests rather than 3, and it seems better for those 2 to be lit +
p
I'm only saying that we should have an open mind. Obviously there are
(valid!) concerns. If we can't solve them then we can't solve them. The
goal (my goal anyway) is always to make things better, not to use X because
it's X. There's value in consistency, but that doesn't mean that the value
fr
> On Sep 14, 2016, at 3:51 PM, Zachary Turner wrote:
>
> zturner added a comment.
>
> Also, it occurred to me that if all tests were like this (and yes, that's a
> tall order to imagine a world where not a single test was written using the
> Python API), we could probably actually drop the Py
Also, w.r.t:
> Aside from write imperative control flow constructs, which I see as a
> positive rather than a negative.
I wrote a bunch of tests to test that stepping behavior for swift and C was
reasonable. When stepping through source code, there is not one correct way to
write the line ta
You can get thing 1, thing 2, and thing 3 in declarative ways with a
command line api too. Like the example I gave earlier:
(lldb) print-dev threads[3].id
0x1234
(lldb) print-dev threads[4].id
0x2345
You're getting one value at a time, not a space separated list. Although I
think we could take
> On Sep 14, 2016, at 3:56 PM, Chris Bieneman wrote:
>
> @jingham and @zturner, we can also take advantage of FileCheck's use of
> regular expressions to write robust matchers. In general LLVM has managed to
> change text output formats many times in radical ways, and LIT's testing has
> stil
If the internal thing is three things in a list, you get back thing one, thing
two and thing three in declarative ways rather than "I got three space
separated things, maybe they had spaces in them, maybe there were quotes, etc,
and don't ever change this now or you'll have to go fix all the tes
On Wed, Sep 14, 2016 at 4:39 PM Jim Ingham wrote:
>
> > 4. You can test a LOT more things when you are able to use an api that
> doesn't have to be stable.
>
> When I mentioned that API, I had in mind an internal Python module that
> you could use to grub into internals. I'm all for that. I'm n
> On Sep 14, 2016, at 4:00 PM, Zachary Turner wrote:
>
> On Wed, Sep 14, 2016 at 3:51 PM Sean Callanan wrote:
> How different is that really from
>
> (lldb) script lldb.frame.FindVariable("argc").GetValue()
> '1'
> (lldb) script lldb.process.GetNumThreads()
> 1
> (lldb) script lldb.process.Get
zturner added inline comments.
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
beanz wrote:
> zturner wrote:
> > beanz wrote:
> > > Disallowing setting both seems reasonable to me. I'm not
beanz added a comment.
@zturner, we can absolutely add flags into the substitutions. Other projects do
that too.
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
zturner wrote:
> beanz wr
On Wed, Sep 14, 2016 at 4:28 PM Jason Molenda wrote:
> I have to concur with Jim's point -- writing & maintaining the gdb
> testsuite for years, which was based on commands & expected output like
> these lit tests, was a huge drag on everyone's productivity as the debugger
> changed over time. T
zturner added a comment.
One more question: Is there a way in lit that we can append command line flags
to the run lines even if the user doesn't specify them? For example in the
substitution?
For example, if someone writes `# RUN: %cxx %p/Inputs/call-function.cpp -g -o
%t && %lldb -b -s %s -
I have to concur with Jim's point -- writing & maintaining the gdb testsuite
for years, which was based on commands & expected output like these lit tests,
was a huge drag on everyone's productivity as the debugger changed over time.
This style of test looks wonderfully easy to read & write & d
beanz added inline comments.
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
Disallowing setting both seems reasonable to me. I'm not entirely sure how to
connect `LLDB_TEST_COMPILER` up i
zturner added inline comments.
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
beanz wrote:
> The `LLDB_TEST_COMPILER` option doesn't signify that it is using an in-tree
> or out-of-tree c
zturner added a comment.
Todd probably knows about this since he added the gtest target to the Xcode
project. I imagine there's a python test suite target and a gtest target.
Isn't it possible to have a third target which when you build it, builds the
other two targets (which would run the te
beanz added a comment.
@granata.enrico, it seems to me that this is a problem the people maintaining
the Xcode project should address. I know it is possible to create targets in
Xcode projects that call shell commands or makefiles (CMake does this). It is
therefore possible that the Xcode proje
beanz updated this revision to Diff 71460.
beanz added a comment.
- Use lit.util.which instead of constructing paths, which adds exe suffixes and
uses proper path seperators
- Fixed spelling error in list(APPEND ...) command
https://reviews.llvm.org/D24591
Files:
lit/CMakeLists.txt
lit/Exp
On Wed, Sep 14, 2016 at 4:00 PM Zachary Turner wrote:
>
> 4. You can test a LOT more things when you are able to use an api that
> doesn't have to be stable.
>
Regarding this point specifically, I think in the past Jim has even floated
the idea of having a "private API" that doesn't have to be s
On Wed, Sep 14, 2016 at 3:51 PM Sean Callanan wrote:
> How different is that really from
>
> (lldb) script lldb.frame.FindVariable("argc").GetValue()
> '1'
> (lldb) script lldb.process.GetNumThreads()
> 1
> (lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
> 3514809
>
> ? If it's deve
granata.enrico added a comment.
In https://reviews.llvm.org/D24591#543277, @beanz wrote:
> @granata.enrico, we could migrate the existing tests into being executed by
> lit even if they aren't using lit's features, so if that direction is desired
> we could get everything in lit. That said, you
beanz added a comment.
@zturner, on many of your comments I need to do some research and get back to
you. Particularly I need to understand how lit works on Windows better. I do
have one inline response.
@granata.enrico, we could migrate the existing tests into being executed by lit
even if th
zturner added a comment.
Also, it occurred to me that if all tests were like this (and yes, that's a
tall order to imagine a world where not a single test was written using the
Python API), we could probably actually drop the Python 3.5 requirement on
Windows.
Another thing that's nice about t
How different is that really from
(lldb) script lldb.frame.FindVariable("argc").GetValue()
'1'
(lldb) script lldb.process.GetNumThreads()
1
(lldb) script lldb.process.GetThreadAtIndex(0).GetThreadID()
3514809
? If it's developer-only, then this is even fairly well-documented using e.g.
"script
zturner added a comment.
In https://reviews.llvm.org/D24591#543242, @jingham wrote:
> Writing tests this way means we're going back to testing the command line
> commands. That was what gdb did for the most part, and you ended up
> terrified of making changes that might effect command output,
jingham added a comment.
Writing tests this way means we're going back to testing the command line
commands. That was what gdb did for the most part, and you ended up terrified
of making changes that might effect command output, not because the change was
hard but because knew you would have t
granata.enrico added a subscriber: granata.enrico.
granata.enrico added a comment.
One piece of concern that I have is the multiplication of things that "testing
LLDB" means.
We have unit tests, Python tests, and now LIT tests.
That means, for each change I want to commit, if I want confidence
beanz updated this revision to Diff 71452.
beanz added a comment.
Added lldb-server as a test dependency based on feedback from @tfiala.
https://reviews.llvm.org/D24591
Files:
lit/CMakeLists.txt
lit/Expr/Inputs/anonymous-struct.cpp
lit/Expr/Inputs/call-function.cpp
lit/Expr/TestCallStdS
tfiala added a comment.
One other items we discussed that is sure to come up:
- Right now this is geared towards one compile per .test file (similar to
something Zachary brought up before). One way we could get the multiple debug
info formats handled is to have a .test for each of the formats,
zturner added a comment.
Couple questions:
1. What is the status of lit and Python 3? Running the test suite on Windows
**requires** Python 3.5+, so if we want this to work on Windows, we will need
to make sure the lit infrastructure is compatible with Python 3.
Comment at:
spyffe added a subscriber: spyffe.
spyffe added a comment.
I like this concept a lot, and I think it's great for testcases that actually
need to interact with the command line.
I'm concerned, though, that the separation of input from command files makes
tests more complex to write. One thing th
tfiala accepted this revision.
tfiala added a comment.
This revision is now accepted and ready to land.
Assuming this doesn't break anybody, this LGTM.
https://reviews.llvm.org/D24591
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://
tfiala added inline comments.
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
tfiala wrote:
> The macOS bots (and all the Swift ones) build with in-tree clang. I'm glad
> to have this opt
tfiala added a comment.
Looks like a nice start. Thanks for pulling this together, Chris!
Comment at: lit/CMakeLists.txt:14
@@ -13,1 +13,3 @@
+option(LLDB_TEST_CLANG "Use in-tree clang when testing lldb" Off)
+
The macOS bots (and all the Swift ones) build wi
zturner added a comment.
Huh... This is a surprise. Looking forward to reviewing this.
https://reviews.llvm.org/D24591
___
lldb-commits mailing list
lldb-commits@lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-commits
beanz created this revision.
beanz added reviewers: zturner, labath, jingham, tfiala.
beanz added a subscriber: lldb-commits.
Herald added subscribers: mgorny, beanz.
This patch supplies basic infrastructure for LLDB to use LIT, and ports a few
basic test cases from the LLDB test suite into LIT.
57 matches
Mail list logo
