On 28/06/2022 15.53, Ani Sinha wrote:
On Tue, Jun 28, 2022 at 19:15 Peter Maydell <peter.mayd...@linaro.org
<mailto:peter.mayd...@linaro.org>> wrote:
On Tue, 28 Jun 2022 at 14:23, Ani Sinha <a...@anisinha.ca
<mailto:a...@anisinha.ca>> wrote:
> On Tue, Jun 28, 2022 at 6:25 PM Daniel P. Berrangé
<berra...@redhat.com <mailto:berra...@redhat.com>> wrote:
> > This proposed biosbits test also involves a considerable download.
>
> I do not think 50 MB is "considerable" . Last time I tried to run
> avocado tests, my laptop ran out of disk space!
I think 50MB is pretty big. It might be smaller than some other
avocado tests, but it's not exactly the "no binary involved"
that most qtests are.
Well bios-tables-test uses the binary blobs of the acpi tables. Only
difference is that in this case, we could maintain them within the qemu
tree. In this case the blob in slightly larger and comes from a third party.
That is the difference.
"slightly larger" ... it apparently contains already a complete grub and
python 2.7 interpreter ... I'd consider that as *much* larger than the ~ 2k
loc bios-tables-test ;-)
Very few 'make check' tests even run code in the guest.
So bits test is similar here. It runs code in the guest vm.
The qtests that run some TCG code only use some few lines of code. The tests
that run third party firmware images are the avocado tests.
There are definitely some awkwardnesses with 'check-avocado',
but we should work on fixing those, not use them as a reason
to refuse to put tests into the avocado tests if that's where
they fit best.
I think this test fits best in the qtrst not with the integration test
framework.
I disagree. Third party binary blob tests are certainly nothing for the
qtest directory.
Very few path developers will ever run it and wrote new tests for
it if we have it there. I would be terribly discouraged if that’s where this
test landed.
I see your point - I'm also hardly ever running whole "make check-avocado"
since it's too slow/fat/broken. But using this as a reason to stick your
test into the qtest framework also does not work/scale: Being one of the
s390x maintainers - What if I now also want my s390x related tests to be
executed by everybody? The kernel + initrd images from
tests/avocado/machine_s390_ccw_virtio.py are also not that big, ca. 50 MiB
each. Should I now move that out of the avocade directory, too, and force
everybody to consume it via a submodule? Then who's next? QEMU has 21 target
architectures, if every maintainers adds a 50 MiB test to the tree, we're at
more than a gigabyte of data already that you'd have to download before you
were able to run "make check". This simply does not scale.
So the avocado framework is currently still the best place that we have for
such tests. You just have to get used to consume it via "tags" instead of
running the whole "make check-avocado" suite. For example, being a s390x
maintainer, I'm only running:
make check-venv
./tests/venv/bin/avocado run -t arch:s390x tests/avocado/
and that finishes within some few minutes and only downloads some few
hundreds of megabytes. You could do something similar with acpi. You could
even wrap it in a "make check-avocado-acpi" target if you don't like to
execute avocado directly.
I even wouldn't mind if you put your python stuff in a new directory like
tests/pytests/ for example, as long as it downloads your binaries separately
- as I wrote in another mail, the avocado framework rather looks like an
oddball in our test framework nowadays since it uses a separate test runner
and not the meson test harness, so having a new approach for python-based
tests is maybe even a good idea. I just really want to avoid that this goes
into tests/qtest (since it really does not belong there), and please don't
add more external stuff via git submodules, that's really the wrong approach
for this.
Thomas
