On 10/12/18 5:44 PM, Philippe Mathieu-Daudé wrote:
> On 12/10/2018 18:53, Cleber Rosa wrote:
>> TL;DR
>> =====
>>
>> Allow acceptance tests to be run with `make check-acceptance`.
>>
>> Details
>> =======
>>
>> This introduces a Python virtual environment that will be setup within
>> the QEMU build directory, that will contain the exact environment that
>> tests may require.
>>
>> There's one current caveat: it requires Python 3, as it's based on the
>> venv module. This was based on some discussions and perception about
>> standardizing on Python 3, but can easily be made to accommodate Python
>> 2 as well.
>>
>> Example of bootstrap and test execution on Travis-CI:
>>
>> https://travis-ci.org/qemu/qemu/jobs/439331028#L2508
>
> If you activate Travis on your github account, you can test that in your
> namespace without having to open zombie pull requests there... A simple
> push to your repository will trigger a full Travis build.
>
> This is how I use it btw, canceling the jobs I'm not interested in, to
> quickly run the others.
> i.e. https://travis-ci.org/philmd/qemu/jobs/439573299#L5600
>
Yep, I've done that shortly after sending this. :) Thanks for the tip,
though.
One thing I miss is the ability, without editing the .travis.yaml, to
"choose" just one/few jobs in a build. Or maybe even the order they run
(Of course I want the acceptance tests job first).
Do you know anything about that?
- Cleber.
>>
>> ...
>> VENV /home/travis/build/qemu/qemu/tests/venv
>> MKDIR /home/travis/build/qemu/qemu/tests/results
>> PIP /home/travis/build/qemu/qemu/tests/venv-requirements.txt
>> AVOCADO tests/acceptance
>> JOB ID : 920e4fcf55a1782f1ae77bee64b20ccdc2e1111d
>> JOB LOG :
>> /home/travis/build/qemu/qemu/tests/results/job-2018-10-09T21.42-920e4fc/job.log
>> (1/6)
>> /home/travis/build/qemu/qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test:
>> PASS (3.57 s)
>> (2/6)
>> /home/travis/build/qemu/qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version:
>> PASS (0.04 s)
>> (3/6)
>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: PASS
>> (0.04 s)
>> (4/6)
>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password:
>> PASS (0.04 s)
>> (5/6)
>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password:
>> PASS (0.04 s)
>> (6/6)
>> /home/travis/build/qemu/qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password:
>> PASS (0.04 s)
>> RESULTS : PASS 6 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 |
>> CANCEL 0
>> JOB TIME : 3.90 s
>> ...
>>
>> Changes from v3:
>> ================
>>
>> * Fixed typo in commit message (s/requiment/requirement/). (Eric)
>>
>> Changes from v2:
>> ================
>>
>> * Make the $(TESTS_VENV_DIR) target depend on the
>> venv-requirements.txt file, and touch $(TESTS_VENV_DIR) after venv
>> runs. With this, updates on the file are reflected on the
>> venv. (Philippe)
>>
>> * Run pip with "python -m pip". It may have been installed reusing
>> the system wide packages, and then the script may not be available
>> on the venv. (Philippe)
>>
>> * Dropped Python version on Travis, and using the version supplied
>> by the distro (3.4). (Philippe)
>>
>> * Added "python3.4-venv" package requirement on Travis. (Philippe)
>>
>> * Added variable (AVOCADO_SHOW) with logging streams to be shown
>> while running the acceptance tests. By default it's set to none,
>> the equivalent of the quiet mode used on previous versions.
>> (Philippe)
>>
>> * On Travis, set the AVOCADO_SHOW variable to "app", so that the
>> individual test results can be easily seen. (Philippe)
>>
>> Ideas discussed, but not implemented:
>>
>> * Run pip with "$(PYTHON) -m pip -q install ..." because it points
>> to the system wide Python installation. (Philippe)
>>
>> * Drop the "--system-site-packages" flag. Waiting on another round
>> of tests to determine if they are really the cause of some package
>> installation problems.
>>
>> Changes from v1:
>> ================
>>
>> * TESTS_VENV_REQ (the path of "venv-requirements.txt") now points to
>> the source path ($SRC_PATH instead of $BUILD_DIR)
>>
>> * Create the venv with "--system-site-packages", which allows the
>> reuse of packages (and no additional downloads) in case there's a
>> package installed system wide providing the same package and
>> version.
>>
>> * Run Avocado with "python -m avocado". It may have been installed
>> reusing the system wide packages, and then the script may not
>> be available on the venv.
>>
>> * Improved documentation describing the Python 3, venv and pip
>> requirements.
>>
>> * Updated avocado-framework requirement to latest released version
>> (65.0)
>>
>> * (New commit) Added support for running the acceptance tests on
>> Travis.
>>
>> Ideas discussed, but not implemented:
>>
>> * Install external packages such as python3-pip on Debian based
>> systems, deemed too invasive on developer's systems.
>>
>> * Allow the use of Python 2, and consequently the "virtualenv"
>> module.
>>
>> Cleber Rosa (3):
>> Bootstrap Python venv for tests
>> Acceptance tests: add make rule for running them
>> Travis support for the acceptance tests
>>
>> .travis.yml | 5 +++++
>> docs/devel/testing.rst | 35 ++++++++++++++++++++++++++++++-----
>> tests/Makefile.include | 37 +++++++++++++++++++++++++++++++++++++
>> tests/venv-requirements.txt | 4 ++++
>> 4 files changed, 76 insertions(+), 5 deletions(-)
>> create mode 100644 tests/venv-requirements.txt
>>
>
--
Cleber Rosa
[ Sr Software Engineer - Virtualization Team - Red Hat ]
[ Avocado Test Framework - avocado-framework.github.io ]
[ 7ABB 96EB 8B46 B94D 5E0F E9BB 657E 8D33 A5F2 09F3 ]
