Add python script with new logic of searching for tests:
Old behavior:
- tests are named [0-9][0-9][0-9]
- tests must be registered in group file (even if test doesn't belong
to any group, like 142)
New behavior:
- group file is dropped
- tests are searched by file-name instead of group file, so it's not
needed more to "register the test", just create it with name
test-*. Old names like [0-9][0-9][0-9] are supported too, but not
recommended for new tests
- groups are parsed from '# group: ' line inside test files
- optional file group.local may be used to define some additional
groups for downstreams
- 'disabled' group is used to temporary disable tests. So instead of
commenting tests in old 'group' file you now can add them to
disabled group with help of 'group.local' file
Benefits:
- no rebase conflicts in group file on patch porting from branch to
branch
- no conflicts in upstream, when different series want to occupy same
test number
- meaningful names for test files
For example, with digital number, when some person wants to add some
test about block-stream, he most probably will just create a new
test. But if there would be test-block-stream test already, he will
at first look at it and may be just add a test-case into it.
And anyway meaningful names are better and test-* notation is
already used in tests directory.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsement...@virtuozzo.com>
---
docs/devel/testing.rst | 51 +++++-
tests/qemu-iotests/check | 20 +--
tests/qemu-iotests/find_tests.py | 72 ++++++++
tests/qemu-iotests/group | 298 -------------------------------
4 files changed, 132 insertions(+), 309 deletions(-)
create mode 100755 tests/qemu-iotests/find_tests.py
delete mode 100644 tests/qemu-iotests/group
diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index 770a987ea4..5532f1abe4 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -153,7 +153,7 @@ check-block
-----------
``make check-block`` runs a subset of the block layer iotests (the tests that
-are in the "auto" group in ``tests/qemu-iotests/group``).
+are in the "auto" group).
See the "QEMU iotests" section below for more information.
GCC gcov support
@@ -267,6 +267,55 @@ another application on the host may have locked the file,
possibly leading to a
test failure. If using such devices are explicitly desired, consider adding
``locking=off`` option to disable image locking.
+Test case groups
+----------------
+
+Test may belong to some groups, you may define it in the comment inside the
+test. By convention, test groups are listed in the second line of the test
+file, after "#!/..." line, like this:
+
+.. code::
+
+ #!/usr/bin/env python3
+ # group: auto quick
+ #
+ ...
+
+Additional way of defining groups is creating tests/qemu-iotests/group.local
+file. This should be used only for downstream (this file should never appear
+in upstream). This file may be used for defining some downstream test groups
+or for temporary disable tests, like this:
+
+.. code::
+
+ # groups for some company downstream process
+ #
+ # ci - tests to run on build
+ # down - our downstream tests, not for upstream
+ #
+ # Format of each line is:
+
+ 013 ci
+ 210 disabled
+ 215 disabled
+ test-our-ugly-workaround down ci
+
+The following groups are defined:
+
+- quick : Tests in this group should finish within some few seconds.
+
+- img : Tests in this group can be used to excercise the qemu-img tool.
+
+- auto : Tests in this group are used during "make check" and should be
+ runnable in any case. That means they should run with every QEMU binary
+ (also non-x86), with every QEMU configuration (i.e. must not fail if
+ an optional feature is not compiled in - but reporting a "skip" is ok),
+ work at least with the qcow2 file format, work with all kind of host
+ filesystems and users (e.g. "nobody" or "root") and must not take too
+ much memory and disk space (since CI pipelines tend to fail otherwise).
+
+- disabled : Tests in this group are disabled and ignored by check.
+
.. _docker-ref:
Docker based tests
diff --git a/tests/qemu-iotests/check b/tests/qemu-iotests/check
index f7a2d3d6c3..09b2ced2f0 100755
--- a/tests/qemu-iotests/check
+++ b/tests/qemu-iotests/check
@@ -168,9 +168,7 @@ do
if $group
then
# arg after -g
- group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e
"/^[0-9][0-9][0-9].* $r /"'{
-s/ .*//p
-}')
+ group_list=$(./find_tests.py "$r")
if [ -z "$group_list" ]
then
echo "Group \"$r\" is empty or not defined?"
@@ -193,10 +191,8 @@ s/ .*//p
then
# arg after -x
# Populate $tmp.list with all tests
- awk '/^[0-9]{3,}/ {print $1}' "${source_iotests}/group" > $tmp.list
2>/dev/null
- group_list=$(sed -n <"$source_iotests/group" -e 's/$/ /' -e
"/^[0-9][0-9][0-9].* $r /"'{
-s/ .*//p
-}')
+ ./find_tests.py > $tmp.list 2>/dev/null
+ group_list=$(./find_tests.py "$r")
if [ -z "$group_list" ]
then
echo "Group \"$r\" is empty or not defined?"
@@ -486,10 +482,14 @@ testlist options
fi
;;
- *)
+ [0-9]*)
start=$r
end=$r
;;
+ *)
+ echo $r >> $tmp.list
+ xpand=false
+ ;;
esac
@@ -504,7 +504,7 @@ testlist options
BEGIN { for (t='$start'; t<='$end'; t++) printf "%03d\n",t }' \
| while read id
do
- if grep -s "^$id\( \|\$\)" "$source_iotests/group" >/dev/null
+ if (./find_tests.py | grep "$id")
then
# in group file ... OK
echo $id >>$tmp.list
@@ -566,7 +566,7 @@ else
touch $tmp.list
else
# no test numbers, do everything from group file
- sed -n -e '/^[0-9][0-9][0-9]*/s/^\([0-9]*\).*/\1/p'
<"$source_iotests/group" >$tmp.list
+ find_tests.py > $tmp.list
fi
fi
diff --git a/tests/qemu-iotests/find_tests.py b/tests/qemu-iotests/find_tests.py
new file mode 100755
index 0000000000..5de0615ebc
--- /dev/null
+++ b/tests/qemu-iotests/find_tests.py
@@ -0,0 +1,72 @@
+#!/usr/bin/env python3
+
+import os
+import glob
+from collections import defaultdict
+
+
+class TestFinder:
+ def __init__(self):
+ self.groups = defaultdict(set)
+ self.all_tests = glob.glob('[0-9][0-9][0-9]')
+
+ self.all_tests += [f for f in glob.iglob('test-*')
+ if not f.endswith('.out')]
+
+ for t in self.all_tests:
+ with open(t) as f:
+ for line in f:
+ if line.startswith('# group: '):
+ for g in line.split()[2:]:
+ self.groups[g].add(t)
+
+ def add_group_file(self, fname):
+ with open(fname) as f:
+ for line in f:
+ line = line.strip()
+
+ if (not line) or line[0] == '#':
+ continue
+
+ words = line.split()
+ test_file = words[0]
+ groups = words[1:]
+
+ if test_file not in self.all_tests:
+ print('Warning: {}: "{}" test is not found. '
+ 'Skip.'.format(fname, test_file))
+ continue
+
+ for g in groups:
+ self.groups[g].add(test_file)
+
+ def find_tests(self, group=None):
+ if group is None:
+ tests = self.all_tests
+ elif group not in self.groups:
+ tests = []
+ elif group != 'disabled' and 'disabled' in self.groups:
+ tests = self.groups[group] - self.groups['disabled']
+ else:
+ tests = self.groups[group]
+
+ return sorted(tests)
+
+
+if __name__ == '__main__':
+ import sys
+
+ if len(sys.argv) > 2:
+ print("Usage ./find_tests.py [group]")
+ sys.exit(1)
+
+ tf = TestFinder()
+ if os.path.isfile('group'):
+ tf.add_group_file('group')
+
+ if len(sys.argv) == 2:
+ tests = tf.find_tests(sys.argv[1])
+ else:
+ tests = tf.find_tests()
+
+ print('\n'.join(tests))
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
deleted file mode 100644
index 79c6dfc85d..0000000000
--- a/tests/qemu-iotests/group
+++ /dev/null
@@ -1,298 +0,0 @@
-#
-# QA groups control file
-# Defines test groups
-#
-# Some notes about the groups:
-#
-# - do not start group names with a digit
-#
-# - quick : Tests in this group should finish within some few seconds.
-#
-# - img : Tests in this group can be used to excercise the qemu-img tool.
-#
-# - auto : Tests in this group are used during "make check" and should be
-# runnable in any case. That means they should run with every QEMU binary
-# (also non-x86), with every QEMU configuration (i.e. must not fail if
-# an optional feature is not compiled in - but reporting a "skip" is ok),
-# work at least with the qcow2 file format, work with all kind of host
-# filesystems and users (e.g. "nobody" or "root") and must not take too
-# much memory and disk space (since CI pipelines tend to fail otherwise).
-#
-
-#
-# test-group association ... one line per test
-#
-001 rw auto quick
-002 rw auto quick
-003 rw auto
-004 rw auto quick
-005 img auto quick
-# 006 was removed, do not reuse
-007 snapshot auto
-008 rw auto quick
-009 rw auto quick
-010 rw auto quick
-011 rw auto quick
-012 auto quick
-013 rw auto
-014 rw
-015 rw snapshot
-# 016 was removed, do not reuse
-017 rw backing auto quick
-018 rw backing auto quick
-019 rw backing auto quick
-020 rw backing auto quick
-021 io auto quick
-022 rw snapshot auto
-023 rw
-024 rw backing auto quick
-025 rw auto quick
-026 rw blkdbg
-027 rw auto quick
-028 rw backing quick
-029 rw auto quick
-030 rw auto backing
-031 rw auto quick
-032 rw auto quick
-033 rw auto quick
-034 rw auto backing quick
-035 rw auto quick
-036 rw auto quick
-037 rw auto backing quick
-038 rw auto backing quick
-039 rw auto quick
-040 rw auto
-041 rw auto backing
-042 rw auto quick
-043 rw auto backing
-044 rw
-045 rw quick
-046 rw auto aio quick
-047 rw auto quick
-048 img auto quick
-049 rw auto
-050 rw auto backing quick
-051 rw
-052 rw auto backing quick
-053 rw auto quick
-054 rw auto quick
-055 rw
-056 rw backing
-057 rw
-058 rw quick
-059 rw quick
-060 rw auto quick
-061 rw auto
-062 rw auto quick
-063 rw auto quick
-064 rw quick
-065 rw quick
-066 rw auto quick
-067 rw quick
-068 rw quick
-069 rw auto quick
-070 rw quick
-071 rw auto quick
-072 rw auto quick
-073 rw auto quick
-074 rw auto quick
-075 rw quick
-076 io
-077 rw quick
-078 rw quick
-079 rw auto
-080 rw auto
-081 rw quick
-082 rw quick
-083 rw
-084 img quick
-085 rw
-086 rw auto quick
-087 rw quick
-088 rw quick
-089 rw auto quick
-090 rw auto quick
-091 rw migration
-092 rw quick
-093 throttle
-094 rw quick
-095 rw quick
-096 rw quick
-097 rw auto backing
-098 rw auto backing quick
-099 rw auto quick
-# 100 was removed, do not reuse
-101 rw quick
-102 rw quick
-103 rw auto quick
-104 rw auto
-105 rw auto quick
-106 rw quick
-107 rw auto quick
-108 rw auto quick
-109 rw
-110 rw auto backing quick
-111 rw auto quick
-112 rw
-113 rw quick
-114 rw auto quick
-115 rw
-116 rw quick
-117 rw auto
-118 rw
-119 rw quick
-120 rw auto quick
-121 rw
-122 rw
-123 rw quick
-124 rw backing
-125 rw
-126 rw auto backing
-127 rw auto backing quick
-128 rw quick
-129 rw quick
-130 rw quick
-131 rw quick
-132 rw quick
-133 auto quick
-134 rw auto quick
-135 rw
-136 rw
-137 rw auto
-138 rw auto quick
-139 rw quick
-140 rw auto quick
-141 rw auto quick
-142
-143 auto quick
-144 rw quick
-145 quick
-146 quick
-147 img
-148 rw quick
-149 rw sudo
-150 rw auto quick
-151 rw
-152 rw quick
-153 rw quick
-154 rw auto backing quick
-155 rw
-156 rw auto quick
-157 quick
-158 rw auto quick
-159 rw auto quick
-160 rw quick
-161 rw auto quick
-162 quick
-163 rw
-165 rw quick
-169 rw quick migration
-170 rw auto quick
-171 rw quick
-172 auto
-173 rw
-174 auto
-175 quick
-176 rw auto backing
-177 rw auto quick
-178 img
-179 rw auto quick
-181 rw auto migration
-182 rw quick
-183 rw migration
-184 rw auto quick
-185 rw
-186 rw auto
-187 rw auto
-188 rw quick
-189 rw
-190 rw auto quick
-191 rw auto
-192 rw auto quick
-194 rw migration quick
-195 rw auto quick
-196 rw quick migration
-197 rw quick
-198 rw
-199 rw migration
-200 rw
-201 rw migration
-202 rw quick
-203 rw auto migration
-204 rw quick
-205 rw quick
-206 rw
-207 rw
-208 rw quick
-209 rw quick
-210 rw
-211 rw quick
-212 rw quick
-213 rw quick
-214 rw auto
-215 rw quick
-216 rw quick
-217 rw auto quick
-218 rw quick
-219 rw
-220 rw auto
-221 rw quick
-222 rw quick
-223 rw quick
-224 rw quick
-225 rw quick
-226 auto quick
-227 quick
-228 rw quick
-229 auto quick
-231 quick
-232 quick
-233 quick
-234 quick migration
-235 quick
-236 quick
-237 rw quick
-238 quick
-239 rw quick
-240 quick
-241 rw quick
-242 rw quick
-243 rw quick
-244 rw auto quick
-245 rw
-246 rw quick
-247 rw quick
-248 rw quick
-249 rw auto quick
-250 rw quick
-251 rw auto quick
-252 rw auto backing quick
-253 rw quick
-254 rw backing quick
-255 rw quick
-256 rw auto quick
-257 rw
-258 rw quick
-259 rw auto quick
-260 rw quick
-261 rw
-262 rw quick migration
-263 rw quick
-264 rw
-265 rw auto quick
-266 rw quick
-267 rw auto quick snapshot
-268 rw auto quick
-270 rw backing quick
-272 rw
-273 backing quick
-277 rw quick
-279 rw backing quick
-280 rw migration quick
-281 rw quick
-282 rw img quick
-283 auto quick
-284 rw
-286 rw quick
-288 quick
-289 rw quick
--
2.21.0
