On 18/1/22 19:04, John Snow wrote:
On Tue, Jan 18, 2022 at 5:06 AM Daniel P. Berrangé <berra...@redhat.com> wrote:
It would be nice to just have this integrated into 'make check' so we
don't need to remember to run a special command.
The CI will run it, but 'make check' doesn't. To add it to make check,
I need to figure out how to insert a venv-building step into 'make
check' such that the venv gets deposited into the build dir instead of
the source dir.
I think I may also need yet another set of package dependencies that
pin on precise dependencies for testing purposes to prevent random
regressions during 'make check' when nobody has touched the Python
code.
Overall, I felt like maybe it was more hassle than it was worth if I
can just nudge people touching the python to run a 'make check-dev'
every so often.
Patches welcome, etc. My overall strategy with the python tests so far has been:
(1) Keep python tests fully separate from the QEMU build system to
allow them to be split out into new repositories easily.
(2) Use the pipenv test to lock the very oldest dependencies the code
and tests support, using the very oldest python we support (3.6) This
test is used as the gating test in GitLab CI, as it is very repeatable
and the GitLab CI setup ensures I can always have the exact Python
packages it requires available.
(3) Use the tox test to test against a wide variety of Python
interpreters (3.6 through 3.10 inclusive) using the very latest python
packages to detect regressions on cutting-edge environments
(4) Use the widest possible range of versions for dependent packages
in setup.cfg such that QEMU packages are unlikely to cause versioning
conflicts in environments that decide to integrate our code.
Overall, I test on 3.6 through 3.10, and against the "oldest" and
"newest" dependencies. It's a good, wide matrix.
However, It's #4 there that runs me into trouble with tests that are
guaranteed to pass -- the linters update all the time and cause new
problems. I use pipenv to lock to specific versions, but that tool
wants to run against Python 3.6 *explicitly*, so it isn't suitable for
a generic purpose 'make check' because not everyone will have a Python
3.6 interpreter available. I need something kind of halfway between,
where I can lock against specific versions but not against the Python
interpreter version, and that's what could be used for a decent 'make
check' test.
Of course, I don't want to maintain like 10 versions of a dependent
packages list, either.
(I really, really wish pip had an --use-oldest flag for dependency
resolution, but it doesn't.)
Could we simply use a virtualenv for all QEMU testing tasks (packages
consumed by QEMU tests), and only deal with installed Python packages
for regular non-testing QEMU uses (things exposed via pyqemu that we
want stable)?
