On Wed, Oct 17, 2018 at 03:25:34PM -0400, Cleber Rosa wrote: > > > On 10/17/18 3:09 PM, Eduardo Habkost wrote: > > On Wed, Oct 17, 2018 at 07:40:51PM +0100, Peter Maydell wrote: > >> On 17 October 2018 at 18:38, Cleber Rosa <cr...@redhat.com> wrote: > >>> > >>> > >>> On 10/17/18 12:29 PM, Eduardo Habkost wrote: > >>>> On Wed, Oct 17, 2018 at 01:34:41PM +0100, Peter Maydell wrote: > >>>>> So, why does the test code need to care? It's not clear > >>>>> from the patch... My expectation would be that you'd > >>>>> just test all the testable target architectures, > >>>>> regardless of what the host architecture is. > >>>> > >>>> I tend to agree. Maybe the right solution is to get rid of the > >>>> os.uname(). I think the default should be testing all QEMU > >>>> binaries that were built, and the host architecture shouldn't > >>>> matter. > >> > >> Yes, looking at os.uname() also seems like an odd thing > >> for the tests to be doing here. The test framework > >> should be as far as possible host-architecture agnostic. > >> (For some of the KVM cases there probably is a need to > >> care, but those are exceptions, not the rule.) > >> > >>> I'm in favor of exercising all built targets, but that seems to me to be > >>> on another layer, above the test themselves. This change is about the > >>> behavior of a test when not told about the target arch (and thus binary) > >>> it should use. > >> > >> At that level, I think the right answer is "tell the user > >> they need to specify the qemu executable they are trying to test". > >> In particular, there is no guarantee that the user has actually > >> built the executable for the target that corresponds to the > >> host, so it doesn't work to try to default to that anyway. > > > > Agreed. However, I don't see when exactly this message would be > > triggered. Cleber, on which use cases do you expect > > pick_default_qemu_bin() to be called? > > > > When a test is run ad-hoc. You even suggested that tests could/should > be executable. > > > In an ideal world, any testing runner/tool should be able to > > automatically test all binaries by default. Can Avocado help us > > do that? (If not, we could just do it inside a > > ./tests/acceptance/run script). > > > > Avocado can do that indeed. But I'm afraid that's not the main issue. > Think of the qemu-iotests: do we want a "check" command to run all > tests with all binaries?
Good question. That would be my first expectation, but I'm not sure. Pro: testing all binaries by default would cause less confusion than picking a random QEMU binary. Con: testing all binaries may be inconvenient for quickly checking if a test case works. (I'm not convinced this is a problem. If you don't want to test everything, you probably already have a short target list in your ./configure line?) Pro: testing a single binary using uname() is already implemented. Con: making `avocado run` automatically generate variants of a test case may take some effort? -- Eduardo
