On Thu, Mar 08, 2012 at 08:07:27PM -0300, Lucas Meneghel Rodrigues wrote: > On 03/08/2012 06:24 PM, Anthony Liguori wrote: > >> > >>Cons: > >>- Lot of code will be duplicated to cover the main code paths: > >>writting tests will require writting/supporting considerable > >>ammount of code (that already exists in autotest). > > > >Again, existence proof that this isn't true. > > Case in point, the virtio test (that uses an auxiliary script to > send data to the host). Can you tell me if both tests cover even > remotely the same amount of functionality? > > https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py > > https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py > > Here is the qemu-test version > > http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master > > What the qemu-test version covers: > * host starts qemu with one virtio console device backed by a file > * guest verifies if the name of the device is correct > * guest writes to the console device > * host verifies if guest wrote to the virtio console > > What the virtio-console covers: > > * Sends data between host and guest back and forth, validates the > data being sent, for both small and large amounts of data, both > random or sequential. > * Tests write/send in blocking, polling, selecting mode, with port > mode sync/async > * Verifies if the maximum amount of ports was created and it's > available in the guest > * Tries lseek on the device > * Verifies if concomitant access to a single port passes or fails > * Verifies data throughput and resource utilization > * Repeats the data transfer with live migration to see if the port > connections survive > * Keeps an eye on the guest OS to see if we have kernel panics, and > if it does, it'll clean up and put the guest in a working state so > other functionality can be checked > * Probably something else I'm forgetting right now. >
Thanks for the example Lucas. :) BTW, this is a QE-level test. Simpler tests may be sufficient for developers and for TDD, but, if one implements even a much simpler test using libautotest, he gets these benefits for free: 1. QE or somebody else may extend the test using the autotest goodies the original developer was not aware of. Patches from QE will flow to qemu.git. That's the integration with QE I was talking about. 2. The test can run in the autotest grid, where a large matrix of configurations and usage scenarios are used. That's possible because we don't have much stuff hardcoded. 3. Autotest has a lot of checks for robustness. One of the main problems of keeping thousands of tests running is to keep them stable, without false positives. Lucas: could you ellaborate more on what we get "for free" when we use libautotest, even when writting a very simple developer-level test? > Good luck implementing that in a shell script. I'd love to see how > you implement that amount of coverage in less lines in shell. > > So, more coverage, more code. It's as simple as that. We don't write > code just for the sake of it. > > >>- QE will be alienated from the qemu test effort. There will be > >>no integration between the QE efforts and the maintenance of > >>the qemu developer-level tests. > > > >I think we're a pretty friendly and open community :-) There is no > >reason that QE should be "alienated" unless folks are choosing not to > >participate upstream. > > > >>- You don't get the goodies from autotest (standardized system > >>info collection, video recording, grid support, etc). > >>- The new tests will work only on the master branch. > > > >This last one is a legitimate point that I have considered myself. But I > >think the value of having tests in HEAD outweigh the cost here. > > > >>3. A mix of (1) and (2): we "move" everything under qemu.git, but > >>keep the compatibility layer and provide a "libqemutest" for > >>third-parties. > >> > >>Anybody writting a test that interacts with qemu will use this > >>library: qemu, libvirt, spice, ovirt, QE tests, etc. > > > >I really see this all as over architecting to be honest. > > > >Can someone explain in clear terms (without appealing to maturity, > >flexibility, or any other qualitative aspect) why it takes anything more > >than a few dozen shell functions to write all of the tests that are in > >kvm-autotest test today? > > Clearly as your requirements are different than the ones we had when > the project was written, qemu-test doesn't need any of the stuff > present there and you can do it all with a handful of shell script > functions. > > But to do cover the same things covered in autotest today with the > same requirements, I really doubt you can do the same with the same > handful of shell script functions. See the example about the virtio > test above, not to mention other functionality we have with the > tests, such as make possible to do tests run with a migration thread > going on the background, while it's plugging/unplugging devices, > running a stress test or ltp, or a benchmark, or measuring the time > drift experienced by the guest. -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
