On Wed, Apr 24, 2019 at 02:37:02PM +0200, Thomas Huth wrote:
> On 24/04/2019 13.25, Daniel P. Berrangé wrote:
> > On Wed, Apr 24, 2019 at 12:37:43PM +0200, Thomas Huth wrote:
> >> Tests in this group are supposed to run in every possible QEMU
> >> configuration.
> >> That means they should run with every QEMU binary (also non-x86), without
> >> dependencies on an optional features, they must work at least with the
> >> qcow2
> >> image format and be able to run on all kind of host filesystems and users
> >> (i.e. also as "nobody" or "root").
> >>
> >> The initial list has been created as a subset of the "quick" group, where
> >> I've disabled all tests that are failing with qemu-system-aarch64 or
> >> qemu-system-tricore or in one of our CI pipelines.
> >>
> >> Signed-off-by: Thomas Huth <th...@redhat.com>
> >> ---
> >> tests/qemu-iotests/group | 194 ++++++++++++++++++++-------------------
> >> 1 file changed, 102 insertions(+), 92 deletions(-)
> >>
> >> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> >> index bae7718380..2ed42dcc14 100644
> >> --- a/tests/qemu-iotests/group
> >> +++ b/tests/qemu-iotests/group
> >> @@ -4,63 +4,73 @@
> >> # - do not start group names with a digit
> >> #
> >>
> >> +#
> >> +# Some notes about the groups:
> >> +# - Tests in the "quick" group should finish within some few seconds
> >> +# - Tests in the "ci" group are suitable for running in CI systems. That
> >> +# means they should run with every QEMU binary (also non-x86), with
> >> +# every QEMU configuration (i.e. no dependency on an optional feature),
> >> +# at least with the qcow2 image format and all kind of host filesystems
> >> +# and users (i.e. also as "nobody" or "root").
> >
> > No dependancy on optional features feels a bit restrictive from my POV.
> >
> > We have quite alot of testing coverage of stuff that depends on the
> > crypto libraries that is important for us to run in "CI". This includes
> > LUKS block drivers, qcow2 with LUKS, NBD with TLS. Personally I expect
> > all of those to be tested by CI systems.
> >
> > IOW for a "CI" group, the bar should be higher than "no optional features"
>
> Ok, I think I just did not write it properly. What I meant was: the test
> should not *fail* because of a missing feature (what some tests are
> unfortunately doing). It's fine if the test detects the missing feature
> and then marks the test as *skip*.
Ok, this brings back the topic I don't think we ever resolved around how
to best "detect" features in the iotests. eg if we want to know whether
we can use TLS features, how do we decide this ?
One option is to do a dummy launch of QEMU using TLS & check if it fails,
but that's kind of fragile for CI. A genuine bug in QEMU might cause this
dummy launch to fail, and thus mistakenly skip all the tests instead of
failing.
We can't look at config-host.{h,mak} either as iotests are supposed to be
runnable outside of a build tree.
We could look at whether gnutls is linked to QEMU but again that is going
to be fragile if a bug in configure causes us to mistakenly stop enabling
gnutls.
For developers, dynamically probing features & skipping is reasonable.
For automated CI, I think we should set clear desired featureset upfront
based on what we know we are supposde to have available and not silently
skip tests
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
