On Thu, Sep 20, 2018 at 11:18:46PM +0200, Philippe Mathieu-Daudé wrote: > On 9/20/18 10:14 PM, Eduardo Habkost wrote: > > On Thu, Sep 20, 2018 at 04:00:27PM -0400, Cleber Rosa wrote: > >> On 9/20/18 2:58 PM, Eduardo Habkost wrote: > >>> On Thu, Sep 20, 2018 at 11:19:56AM -0400, Cleber Rosa wrote: > >>>> The acceptance (aka functional, aka Avocado-based) tests are > >>>> Python files located in "tests/acceptance" that need to be run > >>>> with the Avocado libs and test runner. > >>>> > >>>> Let's provide a convenient way for QEMU developers to run them, > >>>> by making use of the tests-venv with the required setup. > >>>> > >>>> Also, while the Avocado test runner will take care of creating a > >>>> location to save test results to, it was understood that it's better > >>>> if the results are kept within the build tree. > >>>> > >>>> Signed-off-by: Cleber Rosa <cr...@redhat.com> > >>>> --- > >>>> docs/devel/testing.rst | 28 +++++++++++++++++++++++----- > >>>> tests/Makefile.include | 17 +++++++++++++++-- > >>>> tests/venv-requirements.txt | 1 + > >>>> 3 files changed, 39 insertions(+), 7 deletions(-) > >>>> > >>>> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst > >>>> index 727c4019b5..0fbf0d0aac 100644 > >>>> --- a/docs/devel/testing.rst > >>>> +++ b/docs/devel/testing.rst > >>>> @@ -545,10 +545,24 @@ Tests based on ``avocado_qemu.Test`` can easily: > >>>> - > >>>> http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test > >>>> - > >>>> http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html > >>>> > >>>> -Installation > >>>> ------------- > >>>> +Running tests > >>>> +------------- > >>>> > >>>> -To install Avocado and its dependencies, run: > >>>> +You can run the acceptance tests simply by executing: > >>>> + > >>>> +.. code:: > >>>> + > >>>> + make check-acceptance > >>>> + > >>>> +This involves the automatic creation of Python virtual environment > >>>> +within the build tree (at ``tests/venv``) which will have all the > >>>> +right dependencies, and will save tests results also within the > >>>> +build tree (at ``tests/results``). > >>>> + > >>>> +Manual Installation > >>>> +------------------- > >>>> + > >>>> +To manually install Avocado and its dependencies, run: > >>>> > >>>> .. code:: > >>>> > >>>> @@ -689,11 +703,15 @@ The exact QEMU binary to be used on QEMUMachine. > >>>> Uninstalling Avocado > >>>> -------------------- > >>>> > >>>> -If you've followed the installation instructions above, you can easily > >>>> -uninstall Avocado. Start by listing the packages you have installed:: > >>>> +If you've followed the manual installation instructions above, you can > >>>> +easily uninstall Avocado. Start by listing the packages you have > >>>> +installed:: > >>>> > >>>> pip list --user > >>>> > >>>> And remove any package you want with:: > >>>> > >>>> pip uninstall <package_name> > >>>> + > >>>> +If you've used ``make check-acceptance``, the Python virtual > >>>> environment where > >>>> +Avocado is installed will be cleaned up as part of ``make check-clean``. > >>>> diff --git a/tests/Makefile.include b/tests/Makefile.include > >>>> index 9bb90a83d4..8cef694954 100644 > >>>> --- a/tests/Makefile.include > >>>> +++ b/tests/Makefile.include > >>>> @@ -11,6 +11,7 @@ check-help: > >>>> @echo " $(MAKE) check-qapi-schema Run QAPI schema tests" > >>>> @echo " $(MAKE) check-block Run block tests" > >>>> @echo " $(MAKE) check-tcg Run TCG tests" > >>>> + @echo " $(MAKE) check-acceptance Run all acceptance > >>>> (functional) tests" > >>>> @echo " $(MAKE) check-report.html Generates an HTML test > >>>> report" > >>>> @echo " $(MAKE) check-venv Creates a Python venv for > >>>> tests" > >>>> @echo " $(MAKE) check-clean Clean the tests" > >>>> @@ -1002,10 +1003,11 @@ check-decodetree: > >>>> > >>>> # Python venv for running tests > >>>> > >>>> -.PHONY: check-venv > >>>> +.PHONY: check-venv check-acceptance > >>>> > >>>> TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv > >>>> TESTS_VENV_REQ=$(BUILD_DIR)/tests/venv-requirements.txt > >>>> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results > >>>> > >>>> $(TESTS_VENV_DIR): > >>>> $(call quiet-command, \ > >>>> @@ -1015,8 +1017,19 @@ $(TESTS_VENV_DIR): > >>>> $(TESTS_VENV_DIR)/bin/pip -q install -r $(TESTS_VENV_REQ), \ > >>>> PIP, $(TESTS_VENV_REQ)) > >>>> > >>>> +$(TESTS_RESULTS_DIR): > >>>> + $(call quiet-command, mkdir -p $@, \ > >>>> + MKDIR, $@) > >>>> + > >>>> check-venv: $(TESTS_VENV_DIR) > >>>> > >>>> +check-acceptance: check-venv $(TESTS_RESULTS_DIR) > >>>> + $(call quiet-command, \ > >>>> + $(TESTS_VENV_DIR)/bin/avocado \ > >>>> + --show=none run --job-results-dir=$(TESTS_RESULTS_DIR) > >>>> --failfast=on \ > >>>> + $(SRC_PATH)/tests/acceptance, \ > >>>> + "AVOCADO", "tests/acceptance") > >>> > >>> I think we should provide something easy to use for people who > >>> already have the right Avocado version installed in their system > >>> and want to avoid re-downloading Avocado every time. > >>> > >> > >> Right now, people using their own Avocado installation is actually the > >> documented way. The difference from the currently documented way is > >> that instead of doing `make check-acceptance`, people will run: > >> > >> $ avocado run tests/acceptance > >> > >> IMO, for these users, a `alias check-acceptance='avocado run > >> tests/acceptance'` brings almost the same value. > >> > >> About re-downloading: pip caches files by default, so while Avocado will > >> be installed every time a new venv is created, it should be downloaded > >> only once. And I should mention that, given the fact that one of the > >> packaged formats of Avocado is a "Python wheel", the installation is > >> basically a "tar xf" of sorts. > > > > Fair enough. Note that I'm just guessing what other developers > > would expect here. Maybe most people won't mind having "pip > > install" running implicitly when they run acceptance tests and > > this is a non-issue. > > > > I'm hoping we can get the attention of more people on this thread > > so we can get feedback from actual users. If we don't get any > > feedback about this, I won't mind if we include only the rule you > > suggested, and improve the system later. > > I'm experiencing the 2 cases: > > - As a QEMU developer working on a feature, you plan to add some Avocado > tests. You likely works with bleeding edge QEMU. If system packaged > Avocado is not updated enough for you, you'll use the venv+pip setup. > [This is my particular use.] > > - As a QEMU maintainer you would run (upstream merged) Avocado tests to > avoid regressions. The system packages should be sufficient. > [This is how I'm using it on Travis-CI.]
By "system packages" you mean packages provided by the OS distribution, or just that you want to run "pip install" before the build is executed, as described at https://docs.travis-ci.com/user/languages/python/#dependency-management ? > [...] -- Eduardo
