Re: [lldb-dev] Using FileCheck in lldb inline tests

FWIW, I’m supportive of this. I do find SB API based tests to be powerful but extremely cumbersome to write. If Vedant wants to write 15 different tests for the various cases he’s covering, it’s easy to see that they would be much easier to write this way. It is very powerful to have the test s

Re: [lldb-dev] Using FileCheck in lldb inline tests

I’m fine with it. I still would like to see inline tests ported to a custom lit test format eventually, but this seems orthogonal to that and it can be done in addition to this On Thu, Aug 23, 2018 at 4:25 PM Vedant Kumar wrote: > Pinging this because I'd like this to go forward to make testing e

Re: [lldb-dev] Using FileCheck in lldb inline tests

Pinging this because I'd like this to go forward to make testing easier. I know folks have concerns about maintaining completeness of the scripting APIs and about keeping the test suite debuggable. I just don't think making FileCheck available in inline tests is counter to those goals :). I thi

Re: [lldb-dev] Using FileCheck in lldb inline tests

They add support for injecting artificial tail call frames into a StackFrameList. D50478 is the first one -- there's no SB API extension in the initial patch, although that's planned. For context, tail calling frames go missing in backtraces because the frame of

Re: [lldb-dev] Using FileCheck in lldb inline tests

What do your patches do, out of curiosity? On Wed, Aug 15, 2018 at 12:45 PM Vedant Kumar wrote: > > On Aug 15, 2018, at 12:27 PM, Zachary Turner wrote: > > Back to the original proposal, my biggest concern is that a single inline > test could generate many FileCheck invocations. This could cau

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 15, 2018, at 12:27 PM, Zachary Turner wrote: > > Back to the original proposal, my biggest concern is that a single inline > test could generate many FileCheck invocations. This could cause measurable > performance impact on the test suite. Have you considered this? That's a good

Re: [lldb-dev] Using FileCheck in lldb inline tests

Back to the original proposal, my biggest concern is that a single inline test could generate many FileCheck invocations. This could cause measurable performance impact on the test suite. Have you considered this? Another possible solution is what i mentioned earlier, basically to expose a debug

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 15, 2018, at 12:12 PM, Jason Molenda wrote: > > > >> On Aug 15, 2018, at 11:34 AM, Vedant Kumar wrote: >> >> >> >>> On Aug 14, 2018, at 6:19 PM, Jason Molenda wrote: >>> >>> It's more verbose, and it does mean test writers need to learn the public >>> API, but it's also much m

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 15, 2018, at 11:34 AM, Vedant Kumar wrote: > > > >> On Aug 14, 2018, at 6:19 PM, Jason Molenda wrote: >> >> It's more verbose, and it does mean test writers need to learn the public >> API, but it's also much more stable and debuggable in the future. > > I'm not sure about this.

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 15, 2018, at 10:15 AM, Jim Ingham via lldb-dev > wrote: > > > >> On Aug 14, 2018, at 7:48 PM, Zachary Turner via lldb-dev >> wrote: >> >> >> On Tue, Aug 14, 2018 at 6:58 PM Jason Molenda wrote: >> >> >>> On Aug 14, 2018, at 6:39 PM, Zachary Turner wrote: >>> >>> Having bugs

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 14, 2018, at 6:58 PM, Jason Molenda wrote: > > > >> On Aug 14, 2018, at 6:39 PM, Zachary Turner wrote: >> >> Having bugs also makes the debugger harder to innovate in the future because >> it’s, not having tests leads to having bugs, and sb api tests leads to not >> having tests.

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 14, 2018, at 6:19 PM, Jason Molenda wrote: > > It's more verbose, and it does mean test writers need to learn the public > API, but it's also much more stable and debuggable in the future. I'm not sure about this. Having looked at failing sb api tests for a while now, I find them ab

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 14, 2018, at 6:39 PM, Zachary Turner wrote: > > Having bugs also makes the debugger harder to innovate in the future because > it’s, not having tests leads to having bugs, and sb api tests leads to not > having tests. Yeah, this might be a bit of an under-appreciated point. If ther

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 14, 2018, at 6:07 PM, Zachary Turner wrote: > > > > On Tue, Aug 14, 2018 at 5:56 PM Vedant Kumar > wrote: > > >> On Aug 14, 2018, at 5:34 PM, Zachary Turner > > wrote: >> >> I’ve thought about this in the past but the conclusion I

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 14, 2018, at 7:48 PM, Zachary Turner via lldb-dev > wrote: > > > On Tue, Aug 14, 2018 at 6:58 PM Jason Molenda wrote: > > > > On Aug 14, 2018, at 6:39 PM, Zachary Turner wrote: > > > > Having bugs also makes the debugger harder to innovate in the future > > because it’s, not ha

Re: [lldb-dev] Using FileCheck in lldb inline tests

On Tue, Aug 14, 2018 at 6:58 PM Jason Molenda wrote: > > > > On Aug 14, 2018, at 6:39 PM, Zachary Turner wrote: > > > > Having bugs also makes the debugger harder to innovate in the future > because it’s, not having tests leads to having bugs, and sb api tests leads > to not having te > > Yes, l

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 14, 2018, at 6:39 PM, Zachary Turner wrote: > > Having bugs also makes the debugger harder to innovate in the future because > it’s, not having tests leads to having bugs, and sb api tests leads to not > having tests. At the end of the day, it doesn’t matter how stable the tests > a

Re: [lldb-dev] Using FileCheck in lldb inline tests

Having bugs also makes the debugger harder to innovate in the future because it’s, not having tests leads to having bugs, and sb api tests leads to not having tests. At the end of the day, it doesn’t matter how stable the tests are if there arent enough of them. There should be about 10x-20x as man

Re: [lldb-dev] Using FileCheck in lldb inline tests

It's more verbose, and it does mean test writers need to learn the public API, but it's also much more stable and debuggable in the future. It's a higher up front cost but we're paid back in being able to develop lldb more quickly in the future, where our published API behaviors are being teste

Re: [lldb-dev] Using FileCheck in lldb inline tests

On Tue, Aug 14, 2018 at 5:56 PM Vedant Kumar wrote: > > > On Aug 14, 2018, at 5:34 PM, Zachary Turner wrote: > > I’ve thought about this in the past but the conclusion I came to is that > lldbinline tests are actually just filecheck tests in disguise. Why do we > need both? I’d rather delete the

Re: [lldb-dev] Using FileCheck in lldb inline tests

It'd be easy to update FileCheck tests when changing the debugger (this happens all the time in clang/swift). OTOH, the verbosity of the python API means that fewer tests get written. I see a real need to make expressive tests easier to write. vedant > On Aug 14, 2018, at 5:38 PM, Jason Molend

Re: [lldb-dev] Using FileCheck in lldb inline tests

> On Aug 14, 2018, at 5:34 PM, Zachary Turner wrote: > > I’ve thought about this in the past but the conclusion I came to is that > lldbinline tests are actually just filecheck tests in disguise. Why do we > need both? I’d rather delete the lldbinline infrastructure entirely and make > a new

Re: [lldb-dev] Using FileCheck in lldb inline tests

I'd argue against this approach because it's exactly why the lit tests don't run against the lldb driver -- they're hardcoding the output of the lldb driver command into the testsuite and these will eventually make it much more difficult to change and improve the driver as we've accumulated this

Re: [lldb-dev] Using FileCheck in lldb inline tests

I’ve thought about this in the past but the conclusion I came to is that lldbinline tests are actually just filecheck tests in disguise. Why do we need both? I’d rather delete the lldbinline infrastructure entirely and make a new lit TestFormat that basically does what lldbinline already does On Tu

[lldb-dev] Using FileCheck in lldb inline tests

Hello, I'd like to make FileCheck available within lldb inline tests, in addition to existing helpers like 'runCmd' and 'expect'. My motivation is that several tests I'm working on can't be made as rigorous as they need to be without FileCheck-style checks. In particular, the 'matching', 'subs