On Thu, May 09, 2019 at 06:40:40AM +0200, Gerd Hoffmann wrote: > Hi, > > > > Tests can also timeout due to slow downloads of test kernels. > > > Any chance to run the downloads without timeout? > > > > I acknowledge this is an issue, and have thought about two possible > > ways to solve it: > > > > 1) Downloading/caching/checking all the test assets in a job "pre-tests" > > plugin. > > > 2) Report the test phase (say, setUp()) to the test runner, which > > would allow the runner to: > > > I'm very much interested in your opinion so we can evolve the idea into > > implementation. > > (1) is the better approach I think. That way you can give better > feedback to the user, along the lines of "download 2 of 5 in progress". > Also it allows for better management of the assets, you can have tools > to list them / fetch them / copy them from one to another test machine > / find & cleanup obsolete assets (i.e. Fedora 28 kernel after switching > tests to Fedora 29). > > (2) is probably still useful, to report phases of longer running > tests and maybe have separate timeouts for each phase. But for > assets I think approach (1) is better than a "download asset" > phase. >
I also think that approach #1 is simpler and saner, but thinking about where we're going with the test runner development, I started to have doubts about it. The reason is that we're adding parallel and multi environment (process, machine, container) execution capabilities to the runner. Dogfood version is here: https://github.com/avocado-framework/avocado/pull/3111 By doing all download/caching before any single test starts, we may be wasting a lot of CPU time that could be used for running tests, making the overall job execution much longer. And with regards to test phases, the runner could distinguish between setup and test phases, possibly re-running the setup phase a number of times until sucessfull completion, improving even further the reliability of the *test* results (excluding the setup phase). Anyway, this is only brainstorm at this point, so I'll let you know when this moves forward. > cheers, > Gerd > > FIY: one of the new runner planned features, is based on a question use case you mention, about how to run tests built into a minimal image, such as an initrd. Regards, - Cleber.
