I wasn't forgetting about this -- this is what i meant by a
"FileCheck-like utility" in the original email.
The thing I am afraid of there is the complexity hidden in the "" in your example. The thing is this will rarely be
a fixed string. For example, to set a breakpoint, you need to send
other packets to find the address of a suitable place to place the
breakpoint in. So now you need to have the ability to refer to the
previous packets (and their parts, e.g. to pluck out a field of a json
response) and do at least some basic arithmetic on them. I mean, it
can be done, I just feel it would be a significant amount of code to
basically implement a scripting language with a set of integer and
string operations to be able to construct and parse packets. I think
that's a bit of a lost effort as we already have a language that
supports all those operations, and everyone is familiar with. I am
still aiming to make the syntax of the test as readable as possible,
ideally a sequence of assert statements:
ASSERT_FOO(SendPacket());
ASSERT_FOO(RecieveOK()) and something like that.
Of course, it could be I am overestimating the difficulty of this, and
I have to admit I am biased towards gtest.
However, maybe these don't have to actually be incompatible goals.
Most of the patch in question is about writing a simple client capable
of sending and receiving messages and presenting their contents to the
test. You would need something like this even if you went for the
check-lldb-server utility, which could then use this client as a back
end, coupled with an additional library for parsing the contents of
the test scripts. So if someone ever writes that, hopefully he will
find this code useful. And we could even have the two frameworks
coexisting side-by-side -- the scripted one could be used for simple
tests, and the gtest one when you need advanced flow control or data
manipulation.
On 15 May 2017 at 16:41, Zachary Turner wrote:
> One thing about lit that most people either don't understand or forget about
> is that FileCheck has nothing to do with lit. You can have lit tests without
> FileCheck. It's more work because you would have to define an
> LLDBServerTestFormat and invent some DSL that isn't just a bunch of run
> lines and check statements. You could then write a c++ program like
> lldb-server-test, which gives you all the benefits of code reuse and packet
> parsing that you're talking about, and have your test consist of something
> like:
>
> check-lldb-server --prefix=TEST_ERROR_RESPONSE < %s
> check-lldb-server --prefix=TEST_SUCCESS_RESPONSE < %s
>
> TEST_ERROR_RESPONSE: SEND:
> TEST_ERROR_RESPONSE: RECV-ERROR: 62
>
> TEST_SUCCESS_RESPONSE: SEND:
> TEST_SUCCESS_RESPONSE: RECV-SUCCESS: eax = 7
>
> I think this would make tests both easier to write and easier to understand
> than what is being proposed here.
>
> That said, what is being proposed here can't exactly be called a lateral
> move, because I do agree it's better. So because of that, I'm willing to
> let it go in. But I'm 100% confident that a better solution can be devised
> in lit with some thought. Unfortunately, given that I don't work on
> lldb-server, all I can really do is offer some high level ideas, and it will
> be up to you guys to figure out the details.
>
>
> On Mon, May 15, 2017 at 7:34 AM Pavel Labath wrote:
>>
>> Hello all,
>>
>> In case you haven't noticed it, I'd like to draw your attention to
>> D32930, where we're proposing a new test framework for lldb-server
>> tests. The discussion has so far been about low-level implementation
>> details, so you don't have to read through it if you don't feel like
>> to (but I do encourage it)
>>
>> However, I'd like to explain some of the high-level motivations which
>> led to our proposed design and open a discussion on them here. I'll do
>> this in an FAQ form: :)
>>
>> - why new framework?:
>>
>> Lldb-server tests were never really suited for the dotest.py model
>> anyway (for example they end up creating an SBDebugger, only to be
>> completely ignoring it and opening a socket connection to lldb-server
>> directly). Perhaps for this reason, they are also harder to write than
>> usual lldb tests, and a lot more messy internally (e.g., after Chris's
>> ipv6 patch, which caused all lldb-server connections in the test to
>> fail, I've learned that the test harness will attempt the lldb-server
>> connection 400(!!!) times before conceding defeat). The test suite
>> operation is also very illogical when it comes to doing remote tests:
>> to test lldb-server on a remote target, you first have to build
>> lldb-server for the target, THEN you have to build lldb for the HOST,
>> and THEN you run dotest.py in the HOST build folder while passing
>> funny dotest.py arguments.
>>
>>
>> - why lldb-server ?:
>> We'd like this to be a first step in porting the existing test off of
>> the dotest.py test runner. Unlike the full test suite, the number of
>> lldb-server tests is not that big, so porting
