Whoops, didn't mean to reply off-list.
On Thu, Nov 9, 2023 at 4:34 PM John Snow <js...@redhat.com> wrote:
>
> On Thu, Nov 9, 2023 at 10:59 AM Paolo Bonzini <pbonz...@redhat.com> wrote:
> >
> > Given the recent confusion around how QEMU detects the system
> > Meson installation, and/or decides to install its own, it is
> > time to fill in the "Python virtual environments and the QEMU
> > build system" section of the documentation.
> >
> > As a curiosity, a first and partial draft of the text was generated
> > by an LLM[1]. It required quite a bit of editing and probably did not
> > save much time, but some expressions do remain in the finished text.
>
> boo :p
>
> >
> > [1] https://chat.openai.com/share/42c1500d-71c1-480b-bab9-7ccc2c155365
> >
> > Signed-off-by: Paolo Bonzini <pbonz...@redhat.com>
> > ---
> > docs/devel/build-system.rst | 54 ++++++++++++++++++++++++++++++++++---
> > pythondeps.toml | 3 ++-
> > 2 files changed, 53 insertions(+), 4 deletions(-)
> >
> > diff --git a/docs/devel/build-system.rst b/docs/devel/build-system.rst
> > index 21f78da7d1d..4def2e55e73 100644
> > --- a/docs/devel/build-system.rst
> > +++ b/docs/devel/build-system.rst
> > @@ -122,10 +122,49 @@ functioning. These are performed using a few more
> > helper functions:
> > indicated by $TMPC.
> >
> >
> > -Python virtual environments and the QEMU build system
> > ------------------------------------------------------
> > +Python virtual environments and the build process
> > +-------------------------------------------------
> > +
> > +An important part of configure is to create a Python virtual environment
> > +(venv) during the configuration phase, using the Python interpreter that
> > +``configure`` identified, or that was requested via the ``--python``
> > +command line option and the ``$PYTHON`` variable from the environment.
>
> Which takes precedence, in what order? "and" makes it sound as if both
> --python and $PYTHON are considered, but I don't think that's actually
> true.
>
> > +The venv resides in the ``pyvenv`` directory in the build tree,
> > +and provides consistency in how the build process runs Python code.
> > +In particular it avoids a potential mismatch, where Meson and Sphinx
>
> I think you can drop the comma. This is so pedantic that if you left
> it in to spite me, I'd not blame you. :)
>
> > +binaries on the PATH might operate in a different Python environment
> > +than the one chosen by the user during the build process.
> > +
> > +At this stage, ``configure`` also queries the chosen Python interpreter
> > +about QEMU's build dependencies. ``configure`` does *not*
> > +pick the ``meson``, ``sphinx-build`` or ``avocado`` binaries in the PATH;
> > +likewise, there are no options such as ``--meson`` or ``sphinx-build``.
>
> should we say ``--sphinx-build``?
>
> I also might say "does not ^necessarily pick the ..." because they
> could be the same, it just isn't the criteria it uses to choose them.
>
> > +If QEMU does not find a dependency, check that it was installed in the
> > +right ``site-packages`` directory or with the right ``pip`` program.
>
> I don't actually know what this means. >_>
>
> > +
> > +If the package is available as a system package for the chosen
>
> technically, I think if the package is available at all for the chosen
> interpreter. We supported nested venvs, so it's more than just system
> packages.
>
> > +interpreter, ``configure`` prepares a small script that invokes it
> > +from the venv itself. If not, ``configure`` can also optionally
> > +install dependencies in the virtual environment with ``pip``.
>
> What may not be clear here is that this can refer to installing from
> bundled sources in the source tree. IIRC, it's used in preference to
> internet sources, isn't it? (Unless that changed.)
>
> > +Downloading is triggered only when a ``configure`` option (currently,
> > +only ``--enable-docs``) is explicitly enabled but the dependencies are
> > +not present, and can also be disabled with ``--disable-download``.[#pip]_
> > +
> > +.. [#pip_] Avocado can also be installed with ``pip`` in the virtual
> > + environment when running ``make check-avocado``. In this
> > + case, it is not currently possible to block the downloading.
> > +
>
> Similarly to my above comment: if downloads are disabled, we do have
> some vendored packages we'll attempt to use.
>
> > +The required versions of the packages are stored in a configuration file
> > +``pythondeps.toml``. The format is custom to QEMU, but it is documented
> > +at the top of the file itself and it should be easy to understand. The
> > +requirements should make it possible to use the version that is packaged
> > +that is provided by supported distros.
> > +
> > +When dependencies are downloaded, instead, ``configure`` uses a "known
> > +good" version that is also listed in ``pythondeps.toml``. In this
> > +scenario, ``pythondeps.toml`` behaves like the "lock file" used by
> > +``cargo``, ``poetry`` or other dependency management systems.
> >
> > -TBD
> >
> > Stage 2: Meson
> > ==============
> > @@ -376,6 +415,15 @@ This is needed to obey the --python= option passed to
> > the configure
> > script, which may point to something other than the first python3
> > binary on the path.
> >
> > +By the time Meson runs, Python dependencies are available in the virtual
> > +environment and should be invoked though the scripts that ``configure``
>
> through
>
> > +places under ``pyvenv``. One way to do so is as follows, using Meson's
> > +``find_program`` function::
> > +
> > + sphinx_build = find_program(
> > + fs.parent(python.full_path()) / 'sphinx-build',
> > + required: get_option('docs'))
> > +
> >
> > Stage 3: Make
> > =============
> > diff --git a/pythondeps.toml b/pythondeps.toml
> > index 0a35ebcf9f0..4beadfd96f5 100644
> > --- a/pythondeps.toml
> > +++ b/pythondeps.toml
> > @@ -10,7 +10,8 @@
> > # - accepted: accepted versions when using a system package
> > # - installed: fixed version to install in the virtual environment
> > # if a system package is not found; if not specified,
> > -# the minimum and maximum
> > +# defaults to the same as "accepted" or, if also missing,
> > +# to the newest version available on PyPI.
> > # - canary: if specified, use this program name to present more
> > # precise error diagnostics to the user. For example,
> > # 'sphinx-build' can be used as a bellwether for the
> > --
> > 2.41.0
> >
