On 9/21/18 12:06 AM, Eduardo Habkost wrote: > On Thu, Sep 20, 2018 at 11:18:46PM +0200, Philippe Mathieu-Daudé wrote: >> On 9/20/18 10:14 PM, Eduardo Habkost wrote: >>> On Thu, Sep 20, 2018 at 04:00:27PM -0400, Cleber Rosa wrote: >>>> On 9/20/18 2:58 PM, Eduardo Habkost wrote: >>>>> On Thu, Sep 20, 2018 at 11:19:56AM -0400, Cleber Rosa wrote: >>>>>> The acceptance (aka functional, aka Avocado-based) tests are >>>>>> Python files located in "tests/acceptance" that need to be run >>>>>> with the Avocado libs and test runner. >>>>>> >>>>>> Let's provide a convenient way for QEMU developers to run them, >>>>>> by making use of the tests-venv with the required setup. >>>>>> >>>>>> Also, while the Avocado test runner will take care of creating a >>>>>> location to save test results to, it was understood that it's better >>>>>> if the results are kept within the build tree. >>>>>> >>>>>> Signed-off-by: Cleber Rosa <cr...@redhat.com> >>>>>> --- >>>>>> docs/devel/testing.rst | 28 +++++++++++++++++++++++----- >>>>>> tests/Makefile.include | 17 +++++++++++++++-- >>>>>> tests/venv-requirements.txt | 1 + >>>>>> 3 files changed, 39 insertions(+), 7 deletions(-) >>>>>> >>>>>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst >>>>>> index 727c4019b5..0fbf0d0aac 100644 >>>>>> --- a/docs/devel/testing.rst >>>>>> +++ b/docs/devel/testing.rst >>>>>> @@ -545,10 +545,24 @@ Tests based on ``avocado_qemu.Test`` can easily: >>>>>> - >>>>>> http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test >>>>>> - >>>>>> http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html >>>>>> >>>>>> -Installation >>>>>> ------------- >>>>>> +Running tests >>>>>> +------------- >>>>>> >>>>>> -To install Avocado and its dependencies, run: >>>>>> +You can run the acceptance tests simply by executing: >>>>>> + >>>>>> +.. code:: >>>>>> + >>>>>> + make check-acceptance >>>>>> + >>>>>> +This involves the automatic creation of Python virtual environment >>>>>> +within the build tree (at ``tests/venv``) which will have all the >>>>>> +right dependencies, and will save tests results also within the >>>>>> +build tree (at ``tests/results``). >>>>>> + >>>>>> +Manual Installation >>>>>> +------------------- >>>>>> + >>>>>> +To manually install Avocado and its dependencies, run: >>>>>> >>>>>> .. code:: >>>>>> >>>>>> @@ -689,11 +703,15 @@ The exact QEMU binary to be used on QEMUMachine. >>>>>> Uninstalling Avocado >>>>>> -------------------- >>>>>> >>>>>> -If you've followed the installation instructions above, you can easily >>>>>> -uninstall Avocado. Start by listing the packages you have installed:: >>>>>> +If you've followed the manual installation instructions above, you can >>>>>> +easily uninstall Avocado. Start by listing the packages you have >>>>>> +installed:: >>>>>> >>>>>> pip list --user >>>>>> >>>>>> And remove any package you want with:: >>>>>> >>>>>> pip uninstall <package_name> >>>>>> + >>>>>> +If you've used ``make check-acceptance``, the Python virtual >>>>>> environment where >>>>>> +Avocado is installed will be cleaned up as part of ``make check-clean``. >>>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include >>>>>> index 9bb90a83d4..8cef694954 100644 >>>>>> --- a/tests/Makefile.include >>>>>> +++ b/tests/Makefile.include >>>>>> @@ -11,6 +11,7 @@ check-help: >>>>>> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" >>>>>> @echo " $(MAKE) check-block Run block tests" >>>>>> @echo " $(MAKE) check-tcg Run TCG tests" >>>>>> + @echo " $(MAKE) check-acceptance Run all acceptance >>>>>> (functional) tests" >>>>>> @echo " $(MAKE) check-report.html Generates an HTML test >>>>>> report" >>>>>> @echo " $(MAKE) check-venv Creates a Python venv for >>>>>> tests" >>>>>> @echo " $(MAKE) check-clean Clean the tests" >>>>>> @@ -1002,10 +1003,11 @@ check-decodetree: >>>>>> >>>>>> # Python venv for running tests >>>>>> >>>>>> -.PHONY: check-venv >>>>>> +.PHONY: check-venv check-acceptance >>>>>> >>>>>> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv >>>>>> TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt >>>>>> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results >>>>>> >>>>>> $(TESTS_VENV_DIR): >>>>>> $(call quiet-command, \ >>>>>> @@ -1015,8 +1017,19 @@ $(TESTS_VENV_DIR): >>>>>> $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \ >>>>>> PIP, $(TESTS_VENV_REQ)) >>>>>> >>>>>> +$(TESTS_RESULTS_DIR): >>>>>> + $(call quiet-command, mkdir -p $@, \ >>>>>> + MKDIR, $@) >>>>>> + >>>>>> check-venv: $(TESTS_VENV_DIR) >>>>>> >>>>>> +check-acceptance: check-venv $(TESTS_RESULTS_DIR) >>>>>> + $(call quiet-command, \ >>>>>> + $(TESTS_VENV_DIR)/bin/avocado \ >>>>>> + --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) >>>>>> --failfast=on \ >>>>>> + $(SRC_PATH)/tests/acceptance, \ >>>>>> + "AVOCADO", "tests/acceptance") >>>>> >>>>> I think we should provide something easy to use for people who >>>>> already have the right Avocado version installed in their system >>>>> and want to avoid re-downloading Avocado every time. >>>>> >>>> >>>> Right now, people using their own Avocado installation is actually the >>>> documented way. The difference from the currently documented way is >>>> that instead of doing `make check-acceptance`, people will run: >>>> >>>> $ avocado run tests/acceptance >>>> >>>> IMO, for these users, a `alias check-acceptance='avocado run >>>> tests/acceptance'` brings almost the same value. >>>> >>>> About re-downloading: pip caches files by default, so while Avocado will >>>> be installed every time a new venv is created, it should be downloaded >>>> only once. And I should mention that, given the fact that one of the >>>> packaged formats of Avocado is a "Python wheel", the installation is >>>> basically a "tar xf" of sorts. >>> >>> Fair enough. Note that I'm just guessing what other developers >>> would expect here. Maybe most people won't mind having "pip >>> install" running implicitly when they run acceptance tests and >>> this is a non-issue. >>> >>> I'm hoping we can get the attention of more people on this thread >>> so we can get feedback from actual users. If we don't get any >>> feedback about this, I won't mind if we include only the rule you >>> suggested, and improve the system later. >> >> I'm experiencing the 2 cases: >> >> - As a QEMU developer working on a feature, you plan to add some Avocado >> tests. You likely works with bleeding edge QEMU. If system packaged >> Avocado is not updated enough for you, you'll use the venv+pip setup. >> [This is my particular use.] >> >> - As a QEMU maintainer you would run (upstream merged) Avocado tests to >> avoid regressions. The system packages should be sufficient. >> [This is how I'm using it on Travis-CI.] > > By "system packages" you mean packages provided by the OS > distribution, or just that you want to run "pip install" before > the build is executed, as described at > https://docs.travis-ci.com/user/languages/python/#dependency-management > ?
Good point :) Avocado's team provides a Fedora package, so I'm expecting maintainers using Fedora to simply give "dnf install python-avocado" a try. Travis-CI is Ubuntu based, so we have to bootstrap from the python3-pip package, installing python dependencies in the user environment. I'm currently using Debian, and feel much safer installing/testing bleeding edge in trashable virtual environments (venv). 3 cases so far :)
