This introduces a convention, under which tests with known open
issues, will be automatically excluded from execution via "make
check-acceptance". It should by itself improve the reliability
of test jobs, even those with rare failures.
Those tests can still be run with:
./tests/venv/bin/avocado run $TEST_ID
With the test identification are obtainable via:
./tests/venv/bin/avocado list tests/acceptance
And for tests with known issues:
./tests/venv/bin/avocado list -t issue tests/acceptance
A list of tests that are enabled (that is, match the currently
configured targets, and do not have known issues) can be obtained
with:
make list-acceptance
As a future improvement, a tool can be added to sync the "issue" tags
with issues that end up being resolved closed.
Signed-off-by: Cleber Rosa <cr...@redhat.com>
---
docs/devel/testing.rst | 28 ++++++++++++++++++++++++++
tests/Makefile.include | 3 ++-
tests/acceptance/boot_linux_console.py | 3 +++
tests/acceptance/machine_rx_gdbsim.py | 3 +++
4 files changed, 36 insertions(+), 1 deletion(-)
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 2818c58ff8..f93a9bed3f 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -1147,6 +1147,34 @@ variable as shown on the code snippet below to skip the
test:
def test(self):
do_something()
+Disabling tests
+---------------
+
+If you want to have a test temporarily disabled (from ``make
+check-acceptance``), you can add an Avocado ``issue`` tag to the test.
Example::
+
+ class Feature(Test):
+
+ def test(self):
+ """
+ :avocado: tags=issue
+ """
+ do_something()
+
+Rather than simply adding an ``issue`` tag, it may be helpful to point
+to an actual issue on the project's issue tracker::
+
+ class Feature(Test):
+
+ def test(self):
+ """
+ Reason for disabling this test is documented and tracked at
+ https://gitlab.com/qemu-project/qemu/-/issues/9999
+
+ :avocado: tags=issue:9999
+ """
+ do_something()
+
Uninstalling Avocado
--------------------
diff --git a/tests/Makefile.include b/tests/Makefile.include
index d1f90572a7..eb3e5d2aa6 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -94,7 +94,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
# information please refer to "avocado --help".
AVOCADO_SHOW=app
AVOCADO_RUNNER=runner
-AVOCADO_TAGS=$(patsubst %-softmmu,-t arch:%, $(filter %-softmmu,$(TARGETS)))
+comma := ,
+AVOCADO_TAGS=$(patsubst %-softmmu,--filter-by-tags=-issue$(comma)arch:%,
$(filter %-softmmu,$(TARGETS)))
$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
$(call quiet-command, \
diff --git a/tests/acceptance/boot_linux_console.py
b/tests/acceptance/boot_linux_console.py
index 0a49c0e276..4c824bc161 100644
--- a/tests/acceptance/boot_linux_console.py
+++ b/tests/acceptance/boot_linux_console.py
@@ -485,6 +485,9 @@ def test_arm_raspi2_initrd(self):
"""
:avocado: tags=arch:arm
:avocado: tags=machine:raspi2b
+
+ # https://gitlab.com/qemu-project/qemu/-/issues/636
+ :avocado: tags=issue:636
"""
deb_url = ('http://archive.raspberrypi.org/debian/'
'pool/main/r/raspberrypi-firmware/'
diff --git a/tests/acceptance/machine_rx_gdbsim.py
b/tests/acceptance/machine_rx_gdbsim.py
index 32b737b6d8..62893a9989 100644
--- a/tests/acceptance/machine_rx_gdbsim.py
+++ b/tests/acceptance/machine_rx_gdbsim.py
@@ -54,6 +54,9 @@ def test_linux_sash(self):
:avocado: tags=arch:rx
:avocado: tags=machine:gdbsim-r5f562n7
:avocado: tags=endian:little
+
+ # https://gitlab.com/qemu-project/qemu/-/issues/507
+ :avocado: tags=issue:507
"""
dtb_url = ('https://acc.dl.osdn.jp/users/23/23887/rx-virt.dtb')
dtb_hash = '7b4e4e2c71905da44e86ce47adee2210b026ac18'
--
2.31.1
