Kevin Wolf <kw...@redhat.com> writes: > Am 10.05.2019 um 10:55 hat Thomas Huth geschrieben: >> On 08/05/2019 07.47, Thomas Huth wrote: >> > On 07/05/2019 17.50, Eric Blake wrote: >> >> On 5/7/19 10:22 AM, Thomas Huth wrote: >> >>> On 07/05/2019 15.22, Markus Armbruster wrote: >> >>>> Thomas Huth <th...@redhat.com> writes: >> >>>> >> >>>>> Currently, all tests are in the "auto" group. This is a little bit >> >>>>> pointless. >> >>>>> OTOH, we need a group for the tests that we can automatically run >> >>>>> during >> >>>>> "make check" each time, too. Tests in this new group are supposed to >> >>>>> run >> >>>>> with every possible QEMU configuration, for example they must run with >> >>>>> every >> >>>>> QEMU binary (also non-x86), without failing when an optional features >> >>>>> is >> >>>>> missing (but reporting "skip" is ok), and be able to run on all kind >> >>>>> of host >> >>>>> filesystems and users (i.e. also as "nobody" or "root"). >> >>>>> So let's use the "auto" group for this class of tests now. The initial >> >>>>> list has been determined by running the iotests with non-x86 QEMU >> >>>>> targets >> >>>>> and with our CI pipelines on Gitlab, Cirrus-CI and Travis (i.e. >> >>>>> including >> >>>>> macOS and FreeBSD). >> >>>> >> >>>> I wonder whether we should additionally limit "make check" to "quick" >> >>>> tests. How slow are the non-quick auto tests for you? >> >>> >> >>> I already sorted out some of the tests that run veeeery long, since the >> >>> run time on gitlab, cirrus-ci and travis is limited. "make check-block" >> >>> currently takes 3 minutes on my laptop, I think that's still ok? >> >>> >> >>> When I run the tests from the auto group that are not in the quick >> >>> group, I currently get: >> >>> >> >> >> >> My personal threshold is about 5 seconds for quick, so: >> >> >> >>> 003 1s ... >> >>> 007 2s ... >> >> >> >> Should these be moved to quick? >> > >> > I'll leave that decision up to the blocklayer folks ... I thought that >> > there might have been a different reason that these have not been put >> > into "quick" yet...? >> > >> >>> 013 5s ... >> >> >> >> this one is borderline >> >> >> >>> 014 15s ... >> >>> 015 9s ... >> >> >> >> Definitely not quick, but if you think they are still okay for auto, I >> >> can live with that. >> >> >> >>> 022 1s ... >> >> >> >> Another candidate for quick? >> >> >> >>> 023 18s ... >> >> >> >> Even longer than 14. Okay for auto? >> > >> > I think I'd give it a try. If people are complaining later that "make >> > check" is running now way too long, we still can refine the list later. >> >> Thinking about this again, "make check" now runs quite a bit longer >> indeed. So I now rather tend to remove the tests that run longer than 5s >> from the auto group instead... I think I'll send a v4 of this patch >> where I'll remove them from the auto group. > > I don't think time is everything. We should also consider how much > the tests contribute to basic code coverage. There is no point in > removing a test from the list because it takes 10 seconds, but if I > split it in two tests taking each 5 seconds, you would include both > halves. > > For example, 030, 040 and 041 are not that quick (14/11/42 seconds, > respectively), but they are the most important tests for block jobs and > covering a lot. Sure, 42 seconds is a lot, but I'd keep 030 and 040 at > least.
Yes, we want block jobs covered. However, 42 seconds is a lot indeed. Can you think of ways to get a useful part of the full coverage in five seconds or less?
