On 07/05/2019 17.50, Eric Blake wrote: > On 5/7/19 10:22 AM, Thomas Huth wrote: >> On 07/05/2019 15.22, Markus Armbruster wrote: >>> Thomas Huth <th...@redhat.com> writes: >>> >>>> Currently, all tests are in the "auto" group. This is a little bit >>>> pointless. >>>> OTOH, we need a group for the tests that we can automatically run during >>>> "make check" each time, too. Tests in this new group are supposed to run >>>> with every possible QEMU configuration, for example they must run with >>>> every >>>> QEMU binary (also non-x86), without failing when an optional features is >>>> missing (but reporting "skip" is ok), and be able to run on all kind of >>>> host >>>> filesystems and users (i.e. also as "nobody" or "root"). >>>> So let's use the "auto" group for this class of tests now. The initial >>>> list has been determined by running the iotests with non-x86 QEMU targets >>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. including >>>> macOS and FreeBSD). >>> >>> I wonder whether we should additionally limit "make check" to "quick" >>> tests. How slow are the non-quick auto tests for you? >> >> I already sorted out some of the tests that run veeeery long, since the >> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> currently takes 3 minutes on my laptop, I think that's still ok? >> >> When I run the tests from the auto group that are not in the quick >> group, I currently get: >> > > My personal threshold is about 5 seconds for quick, so: > >> 003 1s ... >> 007 2s ... > > Should these be moved to quick?
I'll leave that decision up to the blocklayer folks ... I thought that there might have been a different reason that these have not been put into "quick" yet...? >> 013 5s ... > > this one is borderline > >> 014 15s ... >> 015 9s ... > > Definitely not quick, but if you think they are still okay for auto, I > can live with that. > >> 022 1s ... > > Another candidate for quick? > >> 023 18s ... > > Even longer than 14. Okay for auto? I think I'd give it a try. If people are complaining later that "make check" is running now way too long, we still can refine the list later. Thomas
signature.asc
Description: OpenPGP digital signature
