Victor Toso <victort...@redhat.com> writes:
> Hi,
>
> On Wed, Aug 31, 2022 at 02:01:54PM +0200, Markus Armbruster wrote:
>> Victor Toso <victort...@redhat.com> writes:
>>
>> > The goal of this generator is to validate QAPI examples and transform
>> > them into a format that can be used for 3rd party applications to
>> > validate their QAPI/QMP introspection.
>> >
>> > For each Example section, we parse server and client messages into a
>> > python dictionary. This step alone has found several ill formatted
>> > JSON messages in the examples.
>> >
>> > The generator outputs another JSON file with all the examples in the
>> > QAPI module that they came from. This can be used to validate the
>> > introspection between QAPI/QMP to language bindings.
>> >
>> > When used with the POC qapi-go branch, we have found bad QMP messages
>> > with wrong member names, mandatory members that were missing and
>> > optional members that were being set with null (not needed).
>> >
>> > A simple example of the output format is:
>> >
>> > { "examples": [
>> > {
>> > "id": "ksuxwzfayw",
>> > "client": [
>> > {
>> > "sequence-order": 1
>> > "message-type": "command",
>> > "message":
>> > { "arguments":
>> > { "device": "scratch", "size": 1073741824 },
>> > "execute": "block_resize"
>> > },
>> > } ],
>> > "server": [
>> > {
>> > "sequence-order": 2
>> > "message-type": "return",
>> > "message": { "return": {} },
>> > } ]
>> > }
>> > ] }
>> >
>> > If this idea seems reasonable, we can add python-qemu-qmp to validate
>> > each message at generation time already.
>> >
>> > Signed-off-by: Victor Toso <victort...@redhat.com>
>>
>> If I understand you correctly, there are two benefits:
>>
>> 1. Mechanical syntax check for examples
>>
>> Love it.
>
> Not just JSON syntax but can be extend to the introspection
> layer. Errors like wrong member names would fail while parsing
> the examples (issues such as fixed by patches 11 and 13/16 should
> not happen anymore).
It's also a mechanical check against the schema. Still love it :)
>> 2. Can extract examples for use as test cases
>>
>> Sounds good to me. Possible redundancy with existing tests.
>> Probably nothing to worry about.
>>
>> Can you explain in a bit more detail how the extracted data
>> is (to be) used?
>
> Sure.
>
> The Golang test that consumes this is 152 lines of code [0]. The
> idea is that we can use the examples to feed Golang unmarshalling
> code and then marshall it back to JSON and compare input JSON
> with output JSON and see that their content matches.
>
> [0] https://gitlab.com/victortoso/qapi-go/-/blob/wip-v3/test/examples_test.go
>
> I have generated the examples with this patch series and stored
> the output here [1]
>
> [1] https://gitlab.com/victortoso/qapi-go/-/tree/wip-v3/test/data/examples
>
> The examples are QMP messages that are either sent by Client "->"
> or sent by Server "<-". The order matters so I take the order set
> in the examples and store it as "sequence-order".
>
> In the Go test code, I follow the sequence-order. One example of
> this being useful is that we know which Return type to expect
> after a Command is issued.
>
> I've also included metadata about the type of message, which is
> one of three options: command, event or return. (Errors are
> return too).
>
> This is important because it makes the tests very easy to write.
> Different Unmarshal/Marshal code can be set in the code block of
> the specific message type.
>
> --
>
> The things that makes me quite excited with this idea are:
>
> 1. We have valid functional examples documented. If the examples
> break, we would have the software in place to know it (plug
> to ci or some other ninja check seems reasonable to me)
>
> 2. Developers should get more interested in documenting examples
> as that alone is is a valid test case, even if only useful
> for language binding's syntax.
Thanks! Would you like to work some of this into your commit message?
