On Wed, 6 Feb 2019 12:02:55 -0500 Cleber Rosa <cr...@redhat.com> wrote:
> On 2/6/19 9:36 AM, Cornelia Huck wrote: > > On Fri, 1 Feb 2019 19:55:52 -0500 > > Cleber Rosa <cr...@redhat.com> wrote: > > > >> The current version of the "check-acceptance" target will only show > >> one line for execution of all tests. That's probably OK if the tests > >> to be run are quick enough and they're always the same. > >> > >> But, there's already one test alone that takes on average ~5 seconds > >> to run, we intend to adapt the list of tests to match the user's build > >> environment (among other choices). > > > > Btw: What are our expectations regarding execution time for tests? > > Especially if we continue adding tests, and architecture-specific tests > > are bound to be slower if run on a foreign architecture via tcg. > > > > Would a make check-acceptance-quick command make sense? ("I only want > > to verify quickly that I didn't break too much, so run the quicker > > tests only, probably only for my host architecture") > > > > Yes, it definitely makes sense. Now, let me know if the following also > makes sense to you: > > 1) Because these tests focus on functional testing, the default > target/shortcut ("make check-acceptance") should run the complete set of > test cases (including the slow ones). Nod. > > 2) Requirements vary greatly from user to user, to while adding a > "check-acceptance-quick" is fine, you just mentioned one extra test > execution variation ("for my host architecture"). For those, the idea > is that: > > a) "make check-acceptance[-quick]" will adapt to the build environment > (if you only built s390x targets, that's all it's going to use) Yes. However, I usually build some extra targets (just to verify that files that e.g. include headers I modify don't break for other environments), but don't necessarily want to run all acceptance tests for them (as I don't expect functional changes for them). That's probably something I only want to do when I run the full set anyway. > > b) "avocado" command line interface *should* be easy enough to fulfill > other requirements, and not necessarily require a "make" target. For > instance, if you're only interested in your host arch and one specific > machine type, a command line such as the following should do the trick: > > $ make check-venv > $ ./tests/venv/bin/avocado run -t arch:`uname -m` -t > machine:WHAT_I_CARE_ABOUT tests/acceptance/ > > How does that sound? Yes, that looks quite usable (I can easily define aliases locally for shortcuts) and solves the case I mentioned above. So, to summarize: - add 'make check-acceptance-quick' that excludes tests marked as 'slow' - use the avocado command line interface to further narrow down architectures and machines, if wanted ?