On 5/7/19 10:22 AM, Thomas Huth wrote: > On 07/05/2019 15.22, Markus Armbruster wrote: >> Thomas Huth <th...@redhat.com> writes: >> >>> Currently, all tests are in the "auto" group. This is a little bit >>> pointless. >>> OTOH, we need a group for the tests that we can automatically run during >>> "make check" each time, too. Tests in this new group are supposed to run >>> with every possible QEMU configuration, for example they must run with every >>> QEMU binary (also non-x86), without failing when an optional features is >>> missing (but reporting "skip" is ok), and be able to run on all kind of host >>> filesystems and users (i.e. also as "nobody" or "root"). >>> So let's use the "auto" group for this class of tests now. The initial >>> list has been determined by running the iotests with non-x86 QEMU targets >>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >>> macOS and FreeBSD). >> >> I wonder whether we should additionally limit "make check" to "quick" >> tests. How slow are the non-quick auto tests for you? > > I already sorted out some of the tests that run veeeery long, since the > run time on gitlab, cirrus-ci and travis is limited. "make check-block" > currently takes 3 minutes on my laptop, I think that's still ok? > > When I run the tests from the auto group that are not in the quick > group, I currently get: >
My personal threshold is about 5 seconds for quick, so: > 003 1s ... > 007 2s ... Should these be moved to quick? > 013 5s ... this one is borderline > 014 15s ... > 015 9s ... Definitely not quick, but if you think they are still okay for auto, I can live with that. > 022 1s ... Another candidate for quick? > 023 18s ... Even longer than 14. Okay for auto? etc. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3226 Virtualization: qemu.org | libvirt.org
signature.asc
Description: OpenPGP digital signature
