Hi, everyone! I'd like to discuss some of this in the upstream KVM call.
TLDR: I would like to begin an organized effort to consolidate our CLI
parsing, moving it onto QAPI. I'd like to talk about how we should
proceed on the KVM call, prior to KVM Forum, where we should continue
these discussions.
Background
----------
You may recall late last December that Stefan started a big discussion
thread about Modernizing the QEMU API:
"Making QEMU easier for management tools and applications" [1]
There were lots of opinions and directions proposed for this, with many
competing visions for where QEMU should go, or what it ought endeavor to be.
Though many of these visions conflict in terms of the implementations
for their end goal, many shared a similar logical end-goal and shared
some concrete intermediate steps. One of those concrete intermediate
steps is the consolidation of our configuration and startup mechanisms
into a unified API.
The QEMU API, Today
------------------
At the moment, QAPI is our most formal system for declaring types,
structures and interfaces. I believe QAPI is not going anywhere, and so
I am doubling down and committing to improving and expanding the QAPI
subsystem.
I wanted to understand what QEMU's existing interface even *was*. We can
understand QEMU's interface to be four components presently:
1. The QEMU Monitor Protocol (QMP)
QMP is based explicitly on top of QAPI, which we do indeed have formal
specifications for in ./qapi. They are not standards-compliant, but they
are at least unified.
2. The GTK UI
The GTK UI offers very minimal interface, and does not offer any feature
that is not available through one of the other interfaces or standard
operating system UI. Good!
3. The Human Monitor Protocol (HMP)
HMP is increasingly based on QMP, though the conversion is not complete
and it is not clear if it will be "complete". This was a major sub-topic
of the thread last December with no clear consensus. Some work continues
to bring major HMP features over to QMP; notably Daniel Berrange has
been trying to port savevm/loadvm over to QMP [2]. For now, it seems
like HMP will be with us at least as a debugging and development
interface. There is work to be done to audit and inventory any remaining
features that must be ported to QMP, which are reundant with QMP, and
which are uniquely useful as debugging interfaces.
4. The QEMU command-line
Last, we have the QEMU CLI. This interface was grown organically over
time and features many different parser subsystems and loosely federated
components with no unified specification document that explains the
entire shape of the CLI.
Auditing the CLI interface
--------------------------
I wanted to know what our CLI really looked like. Not trusting any of
our existing documentation to be complete/authoritative, I used the QEMU
5.1 release as a basis and audited the entirety of that interface. [3]
Here's what I found:
- QEMU 5.1 has 131 command line flags
- 93 of these take an argument
- 38 of them do not.
If we want to unify the parsing into a single formal declaration, it
would be helpful to know what we're dealing with. Of those flags that
take arguments:
- 3 use QAPI to parse their argument directly
- 51 use QemuOpts in some way:
- 36 use qemu_opts_parse[_noisily] directly
- 10 desugar to `qemu_opts_parse_noisily` (-fdc, -hda, et al)
- 5 add a single option using `qemu_opt_set`.
- 1 is parsed rather directly by QOM (-tb-size)
- 14 are stored directly as (const char *)
- 3 are parsed into numerical types with atoi/strtol/etc.
- 21 are parsed by custom parsing mechanisms.
For full, gory details, please see the document referenced at [3]. It's
about 4000 lines of markdown detailing the QAPI/C structures that define
each command line argument, as well as some fairly detailed analyses of
the custom parsers and exactly which values they accept.
I'm not reading a 4,000+ line markdown document
-----------------------------------------------
Good news! I made a summary spreadsheet to summarize what I found. [4]
This spreadsheet summarizes the types of arguments we have and what
parsers they utilize and their support status. The spreadsheet follows
the order of flags as defined in qemu-options.hx, category-by-category.
I also tried to broadly assign "topics" to each flag for the purposes of
laying out a better manual in the future, but I wasn't fully confident
in many flags that affect things like boot, firmware, chipset, etc, so
this is a work in progress.
https://docs.google.com/spreadsheets/d/1OJvzDwLpo2XsGytNp9Pr-vYIrb0pWrFKBNPtOyH80ew/edit?usp=sharing
If you don't have google, I have an ODS exported version of this
spreadsheet too -- feel free to relay your feedback back to me here. [5]
Paolo Bonzini helped re-organize my initial draft and used it to
identify flags perhaps most in need of attention to bring onto a new
standard, annotated in yellow.
(Those items are: -k, -uuid, -no-hpet, -no-reboot, -no-shutdown,
-incoming-, and -enable-fips.)
Of the remainder, quite a few are either already deprecated, are
aliases, or are simple sugar for another command that could be expressed
more compactly. Quite a few are already using *at least* QemuOpts such
that porting them to QAPI should not be extremely mechanically difficult.
I would like to use the KVM call to discuss a roadmap for converting the
remaining options to QAPI, what that would take, and who will take
ownership for which subsystems/flags. I would like to bring these
discussions to KVM Forum and lend serious, dedicated effort to finishing
this task.
Related work and ongoing efforts
--------------------------------
Mentioned above, Daniel Berrange is porting HMP features to QMP [2].
I am adding python static typing to our QAPI generator in the belief
that QAPI will continue to grow in importance for us, and inviting more
developers to participate in writing QAPI generator backends by
formalizing that interface. [6].
I am prototyping a new QAPI generator backend that produces Json-Schema,
attempting to target various SDK generators that are compatible with
e.g. OpenAPI (which uses a modified version Json-Schema as a
sub-specification.)
Eduardo Habkost is working on making all QOM type definitions fully data
driven, in the hopes that we might eventually be able to integrate these
types with QAPI to eliminate stub types from the API. [7]
Marc-André is adding a Rust backend to the QAPI generator, along with a
new API frontend that can communicate with dbus. [8]
--
[1] https://lists.gnu.org/archive/html/qemu-devel/2019-12/msg04840.html
[2] https://lists.gnu.org/archive/html/qemu-devel/2020-10/msg00587.html
[3] https://gitlab.com/jsnow/qemu/-/blob/cli_audit/docs/cli_audit.md
[4]
https://docs.google.com/spreadsheets/d/1OJvzDwLpo2XsGytNp9Pr-vYIrb0pWrFKBNPtOyH80ew/edit?usp=sharing
[5] http://people.redhat.com/~jsnow/qemu-5_1-audit.ods
[6] https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg10723.html
[7] https://lists.gnu.org/archive/html/qemu-devel/2020-08/msg08304.html
(And more to come)
[8] https://lists.gnu.org/archive/html/qemu-devel/2020-09/msg03971.html
