On 24/04/2019 14.50, Daniel P. Berrangé wrote: > On Wed, Apr 24, 2019 at 02:37:02PM +0200, Thomas Huth wrote: >> On 24/04/2019 13.25, Daniel P. Berrangé wrote: >>> On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote: >>>> Tests in this group are supposed to run in every possible QEMU >>>> configuration. >>>> That means they should run with every QEMU binary (also non-x86), without >>>> dependencies on an optional features, they must work at least with the >>>> qcow2 >>>> image format and be able to run on all kind of host filesystems and users >>>> (i.e. also as "nobody" or "root"). >>>> >>>> The initial list has been created as a subset of the "quick" group, where >>>> I've disabled all tests that are failing with qemu-system-aarch64 or >>>> qemu-system-tricore or in one of our CI pipelines. >>>> >>>> Signed-off-by: Thomas Huth <th...@redhat.com> >>>> --- >>>> tests/qemu-iotests/group | 194 ++++++++++++++++++++------------------- >>>> 1 file changed, 102 insertions(+), 92 deletions(-) >>>> >>>> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group >>>> index bae7718380..2ed42dcc14 100644 >>>> --- a/tests/qemu-iotests/group >>>> +++ b/tests/qemu-iotests/group >>>> @@ -4,63 +4,73 @@ >>>> # - do not start group names with a digit >>>> # >>>> >>>> +# >>>> +# Some notes about the groups: >>>> +# - Tests in the "quick" group should finish within some few seconds >>>> +# - Tests in the "ci" group are suitable for running in CI systems. That >>>> +# means they should run with every QEMU binary (also non-x86), with >>>> +# every QEMU configuration (i.e. no dependency on an optional feature), >>>> +# at least with the qcow2 image format and all kind of host filesystems >>>> +# and users (i.e. also as "nobody" or "root"). >>> >>> No dependancy on optional features feels a bit restrictive from my POV. >>> >>> We have quite alot of testing coverage of stuff that depends on the >>> crypto libraries that is important for us to run in "CI". This includes >>> LUKS block drivers, qcow2 with LUKS, NBD with TLS. Personally I expect >>> all of those to be tested by CI systems. >>> >>> IOW for a "CI" group, the bar should be higher than "no optional features" >> >> Ok, I think I just did not write it properly. What I meant was: the test >> should not *fail* because of a missing feature (what some tests are >> unfortunately doing). It's fine if the test detects the missing feature >> and then marks the test as *skip*. > > Ok, this brings back the topic I don't think we ever resolved around how > to best "detect" features in the iotests. eg if we want to know whether > we can use TLS features, how do we decide this ? > > One option is to do a dummy launch of QEMU using TLS & check if it fails, > but that's kind of fragile for CI. A genuine bug in QEMU might cause this > dummy launch to fail, and thus mistakenly skip all the tests instead of > failing.
Maybe we need a "--dump-features" option for QEMU (or a QMP command)? (somewhat similar to "gcc -dumpspecs" for example?) Thomas
