On 12/28/2011 09:01 AM, Avi Kivity wrote:
On 12/28/2011 04:27 PM, Anthony Liguori wrote:
Maybe I've used the wrong wording. I got the feeling that, besides
testing qemu
the way you need it, keeping qemu-test simple was really important.
Simple is always important. In the case of qemu-test, there are some
important trade offs made to keep things simple and fast. It doesn't
try to work with arbitrary guests which means the tests need to handle
only one environment. The guest is pre-made to have exactly what is
needed for the tests so there is no setup required. The guest is as
small as possible such that the test can run as quickly as possible.
In fact using linux as a guest negates that. First of all, which linux
version? if it's fixed, you'll eventually miss functionality and need to
migrate. If it keeps changing, so does your test, and it will keep
breaking.
The kernel is a git submodule so it's a very specific version. Yes, we may need
to bump the version down the road and obviously, if we have to change any tests
in the process, we can.
Using Linux as a guest allows you to do system tests (ping was an
example) but doesn't allow you to do unit tests (test regression where
if this bit in that register was set, but if them bits in thar registers
were clear, then the packet would be encoded with ROT-26 before being
sent out).
I think qemu-test needs to use its own drivers which allow full control
over what you do with the hardware. Otherwise, a regression that only
shows up in non-Linux guests will not be testable with qemu-test.
I think you're advocating for qtest. This is another important part of my
testing strategy. I haven't received a lot of input on that RFC...
http://mid.gmane.org/1322765012-3164-1-git-send-email-aligu...@us.ibm.com
But there's certain things that I still consider to be unit testing (like basic
networking tests) that I don't want to have to write with qtest. I'm not up for
writing a TCP/IP stack in Python...
There is no need to have a single tool that meets every possible
need. In fact, the Unix tradition is to have separate single purposed
tools.
Those single purpose tools, if successful, tend to grow more purposes
(cf. qemu), and if unsuccessful, tend to lose purpose.
Having two test tools it not a bad thing provided that the overlap
isn't significant.
This is important, if the boundary isn't clear, then it will grow more
fuzzy in time.
I suggest the following:
- qemu-test: qemu unit tests
- kvm-unit-tests: kvm unit tests
- kvm-autotest: unit test drivers + system tests
I would counter with:
- gtester unit tests (test-visitor, test-qobject, etc., qemu-iotest)
- qtest: low level, single device functional tests
- kvm-unit-tests: low level, instruction-set level functional tests
- qemu-test: higher level functional/coverage tests (multiple device
interaction)
- kvm-autotest: unit/functional test drivers + acceptance testing
Note that the line I'm drawing is acceptance vs. functional testing, not unit
vs. integration testing. Technically, our unit tests are things like
test-visitor. Everything else is an integration test.
But the separation between kvm-autotest is acceptance testing vs. functional
testing.
Acceptance testing is, "does Windows boot", "can I create three virtio-serial
devices".
Obviously, part of acceptance testing is, "does this set of functional tests
pass".
We shouldn't be discussing whether it's possible to merge the two
tools, but rather what the technical benefits of doing so would be.
Since at this point, there is almost no overlap between the two, I
don't see any actual technical benefit to merging them. I see benefit
to autotest executing qemu-test, of course.
I'd say that running a ping test is a weak version of kvm-autotest's
system tests.
Consider the Linux kernel to be our library of functionality to write our unit
tests. I don't want to write a TCP/IP stack. We aren't just grabbing a random
distro kernel. We're building one from scratch configured in a specific way.
In theory, we could even host the kernel source on git.qemu.org and fork it to
add more interesting things in the kernel (although I'd prefer not to do this,
obviously).
Running a synthetic test that pokes values into memory
and mmio and sees a packet coming out is a unit test (the latter can in
fact be executed without a guest at all, just a table driving calls to
the memory and irq APIs).
Just putting the code in kvm-autotest.git in a directory doesn't make
sense to me. Beyond the lack of a technical reason to do so,
logistically, it makes it harder for me to ask people to submit test
cases with a patch series if I can't actually apply the test case when
I'm applying the patches to qemu.git.
If qemu-test didn't use large submodules (like linux.git), I would
have made qemu-test part of qemu.git. As far as I'm concerned,
qemu-test.git is just an extension to qemu.git.
Why not just put it in qemu.git?
Actually, having thought a bit about it, I think we can move qemu-test into
qemu.git and just build the initramfs in a separate repository. That will make
it easier to host binary packages of the later on qemu.org.
Regards,
Anthony Liguori
