On Thu, Jan 20, 2022 at 02:33:46PM +0100, Philippe Mathieu-Daudé wrote: > On 18/1/22 19:04, John Snow wrote: > > On Tue, Jan 18, 2022 at 5:06 AM Daniel P. Berrangé <berra...@redhat.com> > > wrote: > > > > It would be nice to just have this integrated into 'make check' so we > > > don't need to remember to run a special command. > > > > The CI will run it, but 'make check' doesn't. To add it to make check, > > I need to figure out how to insert a venv-building step into 'make > > check' such that the venv gets deposited into the build dir instead of > > the source dir. > > I think I may also need yet another set of package dependencies that > > pin on precise dependencies for testing purposes to prevent random > > regressions during 'make check' when nobody has touched the Python > > code. > > > > Overall, I felt like maybe it was more hassle than it was worth if I > > can just nudge people touching the python to run a 'make check-dev' > > every so often. > > > > Patches welcome, etc. My overall strategy with the python tests so far has > > been: > > > > (1) Keep python tests fully separate from the QEMU build system to > > allow them to be split out into new repositories easily. > > (2) Use the pipenv test to lock the very oldest dependencies the code > > and tests support, using the very oldest python we support (3.6) This > > test is used as the gating test in GitLab CI, as it is very repeatable > > and the GitLab CI setup ensures I can always have the exact Python > > packages it requires available. > > (3) Use the tox test to test against a wide variety of Python > > interpreters (3.6 through 3.10 inclusive) using the very latest python > > packages to detect regressions on cutting-edge environments > > (4) Use the widest possible range of versions for dependent packages > > in setup.cfg such that QEMU packages are unlikely to cause versioning > > conflicts in environments that decide to integrate our code. > > > > Overall, I test on 3.6 through 3.10, and against the "oldest" and > > "newest" dependencies. It's a good, wide matrix. > > > > However, It's #4 there that runs me into trouble with tests that are > > guaranteed to pass -- the linters update all the time and cause new > > problems. I use pipenv to lock to specific versions, but that tool > > wants to run against Python 3.6 *explicitly*, so it isn't suitable for > > a generic purpose 'make check' because not everyone will have a Python > > 3.6 interpreter available. I need something kind of halfway between, > > where I can lock against specific versions but not against the Python > > interpreter version, and that's what could be used for a decent 'make > > check' test. > > > > Of course, I don't want to maintain like 10 versions of a dependent > > packages list, either. > > > > (I really, really wish pip had an --use-oldest flag for dependency > > resolution, but it doesn't.) > > Could we simply use a virtualenv for all QEMU testing tasks (packages > consumed by QEMU tests), and only deal with installed Python packages > for regular non-testing QEMU uses (things exposed via pyqemu that we > want stable)?
I don't especially like the idea of using virtualenv. It is a good thing that we're testing on the distro python packages, because that is the scenario that our contributors/developers will actually use these tools in. We're got a well defined set of target platforms that QEMU aims for that we need to cover in testing. If we also want to do tests against general python using a virtualenv in CI pipelines thats fine, but doesn't replace the need to testing against our explicit platform targets, its just additive. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
