Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote: > On 03/08/2012 11:59 AM, Ademar Reis wrote: > > >>I expect QEMU to grow tests for anything that involves launching > >>QEMU directly. Where I would not see QEMU growing tests for is > >>things like launching QEMU through libvirt. > > > >Agree. For QEMU developers, libvirt should not be on the way, the > >interaction should be minimal or non-existent. > > > >That's an area which will require some work in libautotest, > >because due to previous QE requirements, it now invokes libvirt > >methods instead of QEMU directly for a lot of stuff. But it can > >be done and is part of the plan. > > If you're talking about libautotest as an abstraction layer for > launching QEMU, I don't think that's something we should use in > upstream QEMU. Abstraction layers are okay when the things you are > abstracting are well defined and relatively static. We're talking > about testing the tip of a development tree though and coordinating > changes with an abstraction layer would be counter productive. > Valid point. If I'm a qemu developer and I want to make huge changes, ideally I shouldn't have to submit patches to libautotest because the tests in qemu.org will break. I don't know how likely such scenario is (Lucas is maitaining kvm-autotest for years now and the integration hardly breaks) But we do have a conflict of interests, so lets evaluate a few options: My main interests are: - remove barriers for developers to write tests - share the testing effort between multiple projects (qemu, libvirt, ovirt, spice, QE, etc...) Now the options: 1. We keep the invocation stuff in libautotest and when writting a complex test in qemu.git, the test writter can choose to use it because of the goodies there. Pros: - Lots of codepaths implemented both in python and as cmd-line utilities: less lines of code to write a test, smaller barrier for the developer. - Mature code in the test library. - Goodies such as video-recording and system info collection in a standard way for different projects. - Code shared with other teams. - The same test code can run on old platforms because libautotest has a compatibility layer, thus enabling a larger grid of tests. - QE teams will be closer to the development, sharing some of their testing code and maintenance burden. - It can co-exist with qemu-test. Cons: - You depend on an external library for some of your tests (in a way, a circular dependency) 2. We "move" (or implement) all the libautotest stuff related to qemu invocation and interaction into qemu.git, forbidding the dependency of autotest in qemu.git. In a way, that's what tends to happen with qemu-test if things keep going on AS IS in qemu.org (I'm absolutely sure that you'll replicate a lot of what autotest does as you increase the test coverage and complexity). Pros: - Everything under qemu.git control: you break it, you fix it. - Organic growth of the test infra-structure in QEMU Cons: - Lot of code will be duplicated to cover the main code paths: writting tests will require writting/supporting considerable ammount of code (that already exists in autotest). - QE will be alienated from the qemu test effort. There will be no integration between the QE efforts and the maintenance of the qemu developer-level tests. - You don't get the goodies from autotest (standardized system info collection, video recording, grid support, etc). - The new tests will work only on the master branch. 3. A mix of (1) and (2): we "move" everything under qemu.git, but keep the compatibility layer and provide a "libqemutest" for third-parties. Anybody writting a test that interacts with qemu will use this library: qemu, libvirt, spice, ovirt, QE tests, etc. There's current code written in python (which can be encapsulated into cmd line utilities), but the library can provide stuff in shell, perl, whatever. You just have to provide an API for third-parties. To avoid the burden of the qemu-devel list for collaborators, a maintainer collects the contributions from third-parties. Pros: - Everything under qemu.git control. - API available for upper layers in the stack (libvirt, QE, autotest, ovirt, spice, etc), keeping an unified effort on testing. - Reuse the current code from autotest (refactorings are planned anyway). Cons: - A lot of work to be done by qemu developers. - A lot of work to be done by autotest: third-parties will keep using autotest instead of libqemutest for a while. (1) requires work only on autotest by now... you'll wait and see how far we can go (just raise your concerns now if you plan to block the autotest dependency in qemu.git). In par
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote: > On 03/08/2012 03:02 PM, Ademar Reis wrote: > >On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote: > >>On 03/08/2012 11:59 AM, Ademar Reis wrote: > > > > > > > >> > >>>>I expect QEMU to grow tests for anything that involves launching > >>>>QEMU directly. Where I would not see QEMU growing tests for is > >>>>things like launching QEMU through libvirt. > >>> > >>>Agree. For QEMU developers, libvirt should not be on the way, the > >>>interaction should be minimal or non-existent. > >>> > >>>That's an area which will require some work in libautotest, > >>>because due to previous QE requirements, it now invokes libvirt > >>>methods instead of QEMU directly for a lot of stuff. But it can > >>>be done and is part of the plan. > >> > >>If you're talking about libautotest as an abstraction layer for > >>launching QEMU, I don't think that's something we should use in > >>upstream QEMU. Abstraction layers are okay when the things you are > >>abstracting are well defined and relatively static. We're talking > >>about testing the tip of a development tree though and coordinating > >>changes with an abstraction layer would be counter productive. > >> > > > >Valid point. If I'm a qemu developer and I want to make huge > >changes, ideally I shouldn't have to submit patches to > >libautotest because the tests in qemu.org will break. > > > >I don't know how likely such scenario is (Lucas is maitaining > >kvm-autotest for years now and the integration hardly breaks) > >But we do have a conflict of interests, so lets evaluate a > >few options: > > Because kvm-autotest doesn't have thousands of tests and isn't part > of 200+ developers fast paths :-) > > > > >My main interests are: > > - remove barriers for developers to write tests > > This is my number one priority. > > > - share the testing effort between multiple projects (qemu, > > libvirt, ovirt, spice, QE, etc...) > > This is my number 99 priority. You have the right to consider the interests of qemu only, but qemu is used by several other teams who depend on it to implement their own tests. Some collaboration would be extremely beneficial for everybody. Collaboration goes both ways: you get benefits from the interaction and code reuse with different projects and teams, but it may require a few tradeoffs. > > I think you could achieve practical code sharing by decomposing > kvm-autotest into a set of useful, independent tools. For > instance > > >Now the options: > > > >1. We keep the invocation stuff in libautotest and when writting > >a complex test in qemu.git, the test writter can choose to use it > >because of the goodies there. > > > >Pros: > > - Lots of codepaths implemented both in python and as cmd-line > > utilities: less lines of code to write a test, smaller > > barrier for the developer. > > I've got an existence proof that this is not true. The qemu-test > tests are smaller than the corresponding autotest tests. You're comparing developer-level tests with the existent QA-level tests (much more complex). We're talking about an approach where you'll need just a few lines of code to instantiate a VM and interact with it. > > > - Mature code in the test library. > > - Goodies such as video-recording and system info collection > > in a standard way for different projects. > > Why does video-recording have to be part of libautotest? Why can't > you just have a simply utility that connects to a VNC session and > records it? Presumably that's all it's doing here. > > Likewise, there are dozens of "system info collection" such as > sosreport.. why fold this all into libautotest? For standardization: think outside of the qemu community. A lot of our developers contribute code to multiple projects. It gets easier if there's a standard way to achieve the same results when writting tests for different projects. Anyway, it's a minor issue. > > > - Code shared with other teams. > > - The same test code can run on old platforms because > > libautotest has a compatibility layer, thus enabling a > > larger grid of tests. > > This is not a requirement from a QEMU perspective nor do I think it's useful. Well, it's not a requirement, but you can't expect QE teams to spend their time
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 04:46:09PM +0100, Kevin Wolf wrote: > Am 08.03.2012 15:56, schrieb Anthony Liguori: > >> I particularly agreed with basically everything you said on that > >> discussion regarding test simplification (I had just joined the > >> team back then). To me, autotest has been focusing on QE-level, > >> leaving the developer-level test requirements out. Now we're > >> attacking this new front, and a lot of the requirements are > >> indeed from that discussion. > > > > If you want to talk about this in terms of "requirements", my requirement > > is for > > "developer-level" tests to live in qemu.git and be integrated into make > > check. > > Actually, I think make check should be something that runs _really_ > quickly. We'll probably want another make check-full or something that > runs more extensive tests. I like make check because it's part of make, but I would suggest: $ test-runner | ... This would allow use-cases such as: [qemu]$ test-runner tests.d/block/ (run all block tests) [qemu]$ test-runner tests.d/core/whatever-i-want-to-test.sh (run one individual test) [qemu]$ test-runner tests.d/ (run all tests) [qemu]$ test-runner tests.d/devices/*pci* (you get the idea) And, of course: [qemu]$ test-runner --remote=autotest.qemu.org tests.d/block (a test path/name is its ID to both the local test runner and the remote server) Cheers, - Ademar > > That's the reason why the patch I proposed earlier today runs only a few > qemu-iotests cases during make check. The whole thing would already take > too long for everyone to use it. If people run some quick sanity tests > for all of qemu plus the full tests in the subsystem in which they're > making changes, I think that would be the best we can realistically expect. > > Kevin -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 08:48:33AM -0600, Anthony Liguori wrote: > On 03/08/2012 08:01 AM, Lucas Meneghel Rodrigues wrote: > >On 03/08/2012 10:36 AM, Anthony Liguori wrote: > > > >>>Virt/qemu tests: Minimal guest images > >>>- > >>> > >>>In order to make development level test possible, we need the tests to > >>>run fast. > >>>In order to do that, a set of minimal guest images is being developed > >>>and we > >>>have a version for x86_64 ready and functional: > >>> > >>>https://github.com/autotest/buildroot-autotest > >> > >>I'm really not a fan of buildroot. Note that in order to ship binaries, > >>full source needs to be provided in order to comply with the GPL. The > >>FSF at least states that referring to another website for source that's > >>not under your control doesn't satisfy the requirements of the GPL. > > > >We have a full clone of the buildroot repository that points out to the > >sources, > >if it's necessary to have a clone of all the projects needed host there in > >order > >to be able to publish a binary image to help people, then we can do it. > > This is harder than I think you anticipate but okay.. > > > > >>Just out of curiosity, did you try to use qemu-test? Is there a reason > >>you created something different? > > > >I did, and it does what it proposes to. Nothing against it, but we have code > >that can do more things, that has been developed for longer time. > > > >It's similar to qemu-jeos vs buildbot, you have written scripts to create an > >image, which happens to be precisely why buildroot was written more than 10 > >years ago and it works very well, allowing me to put things on the image that > >are not possible with qemu-jeos. If the problem is to point out to all sub > >modules as git repos, we can make that happen too, rather than re-writing > >stuff > >that works. > > > >For a long time I would like to see people working on a single code base, > > I agree, we just disagree on what that code base should be :-) > > That code base should be qemu.git. This discussion isn't about > improving third-party QE--at least not to me. Third party QE is a > solved problem thanks to all of your wonderful work with > kvm-autotest. I'm sure you're looking for more > participation/developers, but even if you had twice the developers > working on kvm-autotest, I don't think it would fundamentally change > our quality. You got it wrong: we don't want people to work on autotest, we want people to use autotest because we believe there's a lot of benefit on using it. And we *know* that the current autotest is not good enough for developers, which is why we're now working on new usage scenarios. > > I'm interested in driving our development process toward test driven > development such that all 200+ people that write patches for QEMU > for a given release write and run tests as part of their normal > development process. The requirements to achieve this are different > than the requirements you have been working against up until now. Fully agree, these are new requirements. > > Every barrier that we put up to writing and running tests will > reduce than number of 200+ to something lower. > > Submitting a patch to a different project than qemu.git is a > barrier. Now instead of getting a single set of feedback, you've > got to deal with feedback from two projects. > Fully agree, please check my previous email with the plans for the new architecture of autotest. > Having to use setup another framework (that runs as root) is another > barrier. I change a file in QEMU, run make, then run make check. I > don't install anything, I don't sudo anything. The whole process is > relatively quick and painless. Fully agree, this is one of the requirement we're going after. > > Having to make a change to autotest, then install autotest, relaunch > it, etc, is just too complicated to be part of a developers fast > path. Fully agree, which is why we're not planning any of it. > > Now I think we should talk about how to make tests that live in > qemu.git and run as part of make check easily harnessed by > autotest.. But I think the primary focus of future test work needs > to be within qemu.git. Fully agree, please check our previous e-mails with the plans for the new architecture. So as you can see, there are no disagreements. :-) Please keep the feedback and requirements coming. Cheers, - Ademar -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote: > On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote: > >Hi guys. For a while we have been discussing ways to make the virtualization > >tests written on top of autotest useful for development level testing. > > > >One of our main goals is to provide useful tools for the qemu community, > >since > >we have a good number of tests and libraries written to perform > >integration/QA > >testing for that tool, being successfuly used by a number of QA teams that > >work > >on qemu. Also, we recently provided a subset of that infrastructure to test > >libvirt, one of our virtualization projects of interest. > > > >We realized that some (admittedly not very radical) changes have to be made > >on > >autotest itself, so we're inviting other users of autotest to give this a > >good > >read. This same document lives in the autotest wiki: > > > >https://github.com/autotest/autotest/wiki/FuturePlans > > > >Please note that splitting the virt tests from autotest is not discarded at > >the > >moment, and it's not incompatible with the plan outlined below. > > > > > >Virt tests and autotest future goals > > > > > >In order to make the autotest infrastructure, and the virt tests developed on > >top of autotest more useful for the people working on developing linux, virt > >and > >other platform software, as well as the QA teams, we are working on a number > >of > >goals, to streamline, simplify and make the available tools appropriate for > >*development level testing*. > > > >Executing tests appropriate for *QA level testing* will continue to be > >supported, as it's one of the biggest strenghts of autotest. > > > >The problem > >--- > > > >Autotest provides as of today a local engine, used to run tests on your local > >machine (your laptop or a remote server). Currently, it runs tests properly > >wrapped and packaged under the autotest client/tests/ folder, using specific > >autotest rules. > > > >For the virt tests that live inside autotest, we have even more rules to > >follow, > >causing a lot of frustration for people that are not used to how things are > >structured and how to execute and select tests. > > > >The proposed solution > >- > > > >A solution is needed for both scenarios (virt and other general purpose > >tests). > >The idea is to create specialized tools can run simple tests without > >packaging, > >code that: > > > >* Knows nothing about the underlying infrastructure > >* Written in any language (shell script, c, perl, you name it) > > > >It'll be up to the test runner to make sense of the results, provided that > >the > >test writer follows some simple common sense principles while writing the > >code: > > > >1) Make the program to return 0 on success, !=0 on failure > >2) Make the program use a test output format, mainly TAP > > We're using gtest in QEMU, not TAP--for better or worse. If > autotest could use the gtest protocol, it would integrate much > better with QEMU's tests. > > Within QEMU, everything should be gtest when possible. > > >For simple tests, we believe that option 1) will be far more popular. > >Autotest > >will harness the execution of the test and put the results under the test > >results directory, with all the sysinfo collection and other instrumentation > >transparently available for the user. > > > >At some point, the test writer might want to start the framework features > >that > >need to be enabled explicitly, then he/she might want to learn how to use the > >python API to do so, but it'll not be a requirement. > > > >More about the test runner > >-- > > > >The test runner for both general and virt cases should have very simple and > >minimal output: > > > >:: > > > >Results stored in /path/to/autotest/results/default > >my-full-test.py -- PASS > >my-other-test.py -- PASS > >look-mom-i-can-use-shell.sh -- PASS > >look-mom-i-can-use-perl.pl -- FAIL > >test-name-is-the-description.sh -- PASS > >my-yet-another-test.sh -- SKIPPED > >i-like-python.py -- PASS > >whatever-test.pl -- PASS > > > >Both will be specialized tools that use the infrastructure of > >client/bin/autotest-local, but with special code to attend to the output spec > >above. They will know how to handle dependencies, and skip tests if needed. > > > >Directory structure > >--- > > > >This is just to give a rough idea of how we won't depend the tests to be in > >the > >autotest source code folder: > > > >:: > > > >/path/to/autotest -> top level dir, that will make the autotest libs > >available > >- client/bin -> Contains the test runners and auxiliary scripts > >- client/virt/tests: Contains the virt tests that still live in autotest > >- client/tests/kvm/tests: Contains the qemu tests that still live in autotest > > > >/any/path/test1: Contains tests for software foo > >/any/path/test2: C
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 11:54:31AM +, Stefan Hajnoczi wrote: > On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi wrote: > > On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues > > wrote: > >> One of our main goals is to provide useful tools for the qemu community, > >> since we have a good number of tests and libraries written to perform > >> integration/QA testing for that tool, being successfuly used by a number of > >> QA teams that work on qemu. Also, we recently provided a subset of that > >> infrastructure to test libvirt, one of our virtualization projects of > >> interest. > > > > Thanks for sharing. > > One thing I should have added is that my message is about what it > would take for me to use autotest and contribute tests. But now I > realize that you might be going for a different model: > > If you're aiming for a different model where autotest integrates > external test suites (i.e. tests wouldn't be written in autotest.git, > instead autotest.git would contain snapshots of external test suites), > then this proposal seems fine. Upstream projects like QEMU would > develop their own test suite and it would be dropped into autotest or > a specific autotest instance. > Yes, that's the idea. We want autotest to be the framework, not (just a) collection of tests. We also want each development team to implement and maintain their own set of tests, using (or not) the goodies from autotest at their discretion. In summary, autotest is (or is going to be) a framework that provides: - A test runner, with grid/cluster support and advanced instrumentation - A devel library and set of utilities for test writers - A set of pre-built images (JeOS – Just Enough OS) for test writers (attached is a picture showing what we want to achieve) If a project has an internal library or set of utilities that can be of general use, they can be submitted to autotest.git for inclusion, thus reaching a broader audience. A short summary of the plans: - Tests can live anywhere and each devel team implements and maintains their own set of tests - Usage of the autotest library by test writers is optional - Tests are scripts returning 0 or error (any language) - Tests can be run individually or in sets - Tests should run fast, our target is seconds or a few minutes - The test runner is smart and “just works” by default - Trivial standard output (FAIL, PASS, SKIPPED) - Collect logs, OS data and other stuff (e.g. --record-video!) - Skip unsupported tests based on the environment they're run - Multiplex configurations / platforms when on the grid - Support to run tests “in the cloud” There are also lots of small changes and usability improvements in the pipeline (and feedback right now is very valuable). Thanks, - Ademar -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote: > On 03/08/2012 08:49 AM, Ademar Reis wrote: > >On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote: > >>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote: > >>>Virt/qemu tests: Minimal guest images > >>>- > >>> > >>>In order to make development level test possible, we need the tests to run > >>>fast. > >>>In order to do that, a set of minimal guest images is being developed and > >>>we > >>>have a version for x86_64 ready and functional: > >>> > >>>https://github.com/autotest/buildroot-autotest > >> > >>I'm really not a fan of buildroot. Note that in order to ship > >>binaries, full source needs to be provided in order to comply with > >>the GPL. The FSF at least states that referring to another website > >>for source that's not under your control doesn't satisfy the > >>requirements of the GPL. > >> > >>Just out of curiosity, did you try to use qemu-test? Is there a > >>reason you created something different? > >> > >>I think it's good that you're thinking about how to make writing > >>tests easier, but we have a growing test infrastructure in QEMU and > >>that's what I'd prefer people focused on. > >> > > > >You probably remember the long thread we had back in December on > >qemu-devel on this topic. Back then our message was "we have a > >growing test infrastructure in s/QEMU/autotest/ and that's what > >we'd prefer people focused on". :-) > > > > From Dor: > > > >(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html) > > > >""" > >If you wish, you can challenge Lucas and Cleber w/ these type of > >requirements and we'll all improve as a result. > >""" > > > >Your response was: > > > >""" > >Well consider qemu-test the challenge. It's an existence proof > >that we can have a very easy to use framework for testing that > >runs extremely fast and is very easy to write tests for. > >""" > > > >http://knowyourmeme.com/memes/challenge-accepted ;-) > > > >I particularly agreed with basically everything you said on that > >discussion regarding test simplification (I had just joined the > >team back then). To me, autotest has been focusing on QE-level, > >leaving the developer-level test requirements out. Now we're > >attacking this new front, and a lot of the requirements are > >indeed from that discussion. > > If you want to talk about this in terms of "requirements", my > requirement is for "developer-level" tests to live in qemu.git and > be integrated into make check. > > Just as we've been discussing and working on since the previous set of > discussions. > > >By simplifying the design and bringing barriers down, we hope to > >reach a broader audience and help developers write and maintain > >tests, benefiting from all the instrumentation that autotest > >brings. It's not going to be just about qemu (check the new test > >examples). > > > >We have a team fully dedicated to autotest and it's used not only > >by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc > >(these all have code contributions in autotest) > > > >That said, the current qemu-tests will probably be easily > >integrated into (the new) autotest and we hope that, given enough > >time, autotest will be good enough to relieve qemu from the > >framework maintenance and code duplication with other projects. > > autotest should not be the focal point for integration. qemu.git should be. > > I'd be perfectly happy to review patches submitting the test > infrastructure from kvm-autotest into qemu.git (provided it didn't > have unreasonable external dependencies and fit into QEMU). > > Developer-level tests need to live where the developers live. The > developers live in qemu.git. See my other response on this thread > for the explanation of why this is so important. > Excelent, we're in the same page then. This was my number 1 requirement when I was discussing the changes with Lucas and Cleber. For convenience, I'll repeat here what I wrote in a previous e-mail (no qemu-devel archive available yet to use as a reference). In summary, autotest is (or is going to be) a framework that provides: - A test runner, with grid/
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote: > On 03/08/2012 09:07 AM, Ademar Reis wrote: > >On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote: > >>On 03/08/2012 08:49 AM, Ademar Reis wrote: > >>>On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote: > >>>>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote: > >>>>>Virt/qemu tests: Minimal guest images > >>>>>- > >>>>> > >>>>>In order to make development level test possible, we need the tests to > >>>>>run fast. > >>>>>In order to do that, a set of minimal guest images is being developed > >>>>>and we > >>>>>have a version for x86_64 ready and functional: > >>>>> > >>>>>https://github.com/autotest/buildroot-autotest > >>>> > >>>>I'm really not a fan of buildroot. Note that in order to ship > >>>>binaries, full source needs to be provided in order to comply with > >>>>the GPL. The FSF at least states that referring to another website > >>>>for source that's not under your control doesn't satisfy the > >>>>requirements of the GPL. > >>>> > >>>>Just out of curiosity, did you try to use qemu-test? Is there a > >>>>reason you created something different? > >>>> > >>>>I think it's good that you're thinking about how to make writing > >>>>tests easier, but we have a growing test infrastructure in QEMU and > >>>>that's what I'd prefer people focused on. > >>>> > >>> > >>>You probably remember the long thread we had back in December on > >>>qemu-devel on this topic. Back then our message was "we have a > >>>growing test infrastructure in s/QEMU/autotest/ and that's what > >>>we'd prefer people focused on". :-) > >>> > >>> From Dor: > >>> > >>>(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html) > >>> > >>>""" > >>>If you wish, you can challenge Lucas and Cleber w/ these type of > >>>requirements and we'll all improve as a result. > >>>""" > >>> > >>>Your response was: > >>> > >>>""" > >>>Well consider qemu-test the challenge. It's an existence proof > >>>that we can have a very easy to use framework for testing that > >>>runs extremely fast and is very easy to write tests for. > >>>""" > >>> > >>>http://knowyourmeme.com/memes/challenge-accepted ;-) > >>> > >>>I particularly agreed with basically everything you said on that > >>>discussion regarding test simplification (I had just joined the > >>>team back then). To me, autotest has been focusing on QE-level, > >>>leaving the developer-level test requirements out. Now we're > >>>attacking this new front, and a lot of the requirements are > >>>indeed from that discussion. > >> > >>If you want to talk about this in terms of "requirements", my > >>requirement is for "developer-level" tests to live in qemu.git and > >>be integrated into make check. > >> > >>Just as we've been discussing and working on since the previous set of > >>discussions. > >> > >>>By simplifying the design and bringing barriers down, we hope to > >>>reach a broader audience and help developers write and maintain > >>>tests, benefiting from all the instrumentation that autotest > >>>brings. It's not going to be just about qemu (check the new test > >>>examples). > >>> > >>>We have a team fully dedicated to autotest and it's used not only > >>>by Qemu but also libvirt, Google, Xen, Fedora, Twitter, etc, etc > >>>(these all have code contributions in autotest) > >>> > >>>That said, the current qemu-tests will probably be easily > >>>integrated into (the new) autotest and we hope that, given enough > >>>time, autotest will be good enough to relieve qemu from the > >>>framework maintenance and code duplication with other projects. > >> > >>autotest should not be the focal point for integration. qemu.git should be. > >> > >&
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 11:03:54AM -0600, Anthony Liguori wrote: > On 03/08/2012 10:05 AM, Ademar Reis wrote: > >On Thu, Mar 08, 2012 at 09:14:02AM -0600, Anthony Liguori wrote: > >>On 03/08/2012 09:07 AM, Ademar Reis wrote: > >>>On Thu, Mar 08, 2012 at 08:56:23AM -0600, Anthony Liguori wrote: > >>>>On 03/08/2012 08:49 AM, Ademar Reis wrote: > >>>>>On Thu, Mar 08, 2012 at 07:36:11AM -0600, Anthony Liguori wrote: > >>>>>>On 03/07/2012 10:00 PM, Lucas Meneghel Rodrigues wrote: > >>>>>>>Virt/qemu tests: Minimal guest images > >>>>>>>- > >>>>>>> > >>>>>>>In order to make development level test possible, we need the tests to > >>>>>>>run fast. > >>>>>>>In order to do that, a set of minimal guest images is being developed > >>>>>>>and we > >>>>>>>have a version for x86_64 ready and functional: > >>>>>>> > >>>>>>>https://github.com/autotest/buildroot-autotest > >>>>>> > >>>>>>I'm really not a fan of buildroot. Note that in order to ship > >>>>>>binaries, full source needs to be provided in order to comply with > >>>>>>the GPL. The FSF at least states that referring to another website > >>>>>>for source that's not under your control doesn't satisfy the > >>>>>>requirements of the GPL. > >>>>>> > >>>>>>Just out of curiosity, did you try to use qemu-test? Is there a > >>>>>>reason you created something different? > >>>>>> > >>>>>>I think it's good that you're thinking about how to make writing > >>>>>>tests easier, but we have a growing test infrastructure in QEMU and > >>>>>>that's what I'd prefer people focused on. > >>>>>> > >>>>> > >>>>>You probably remember the long thread we had back in December on > >>>>>qemu-devel on this topic. Back then our message was "we have a > >>>>>growing test infrastructure in s/QEMU/autotest/ and that's what > >>>>>we'd prefer people focused on". :-) > >>>>> > >>>>> From Dor: > >>>>> > >>>>>(http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg03024.html) > >>>>> > >>>>>""" > >>>>>If you wish, you can challenge Lucas and Cleber w/ these type of > >>>>>requirements and we'll all improve as a result. > >>>>>""" > >>>>> > >>>>>Your response was: > >>>>> > >>>>>""" > >>>>>Well consider qemu-test the challenge. It's an existence proof > >>>>>that we can have a very easy to use framework for testing that > >>>>>runs extremely fast and is very easy to write tests for. > >>>>>""" > >>>>> > >>>>>http://knowyourmeme.com/memes/challenge-accepted ;-) > >>>>> > >>>>>I particularly agreed with basically everything you said on that > >>>>>discussion regarding test simplification (I had just joined the > >>>>>team back then). To me, autotest has been focusing on QE-level, > >>>>>leaving the developer-level test requirements out. Now we're > >>>>>attacking this new front, and a lot of the requirements are > >>>>>indeed from that discussion. > >>>> > >>>>If you want to talk about this in terms of "requirements", my > >>>>requirement is for "developer-level" tests to live in qemu.git and > >>>>be integrated into make check. > >>>> > >>>>Just as we've been discussing and working on since the previous set of > >>>>discussions. > >>>> > >>>>>By simplifying the design and bringing barriers down, we hope to > >>>>>reach a broader audience and help developers write and maintain > >>>>>tests, benefiting from all the instrumentation that autotest > >>>>>brings. It's not going to be just about qemu (check the new test > >>>>>examples). >
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 09:17:42AM -0300, Ademar Reis wrote: > On Thu, Mar 08, 2012 at 11:54:31AM +, Stefan Hajnoczi wrote: > > On Thu, Mar 8, 2012 at 11:44 AM, Stefan Hajnoczi wrote: > > > On Thu, Mar 8, 2012 at 4:00 AM, Lucas Meneghel Rodrigues > > > wrote: > > >> One of our main goals is to provide useful tools for the qemu community, > > >> since we have a good number of tests and libraries written to perform > > >> integration/QA testing for that tool, being successfuly used by a number > > >> of > > >> QA teams that work on qemu. Also, we recently provided a subset of that > > >> infrastructure to test libvirt, one of our virtualization projects of > > >> interest. > > > > > > Thanks for sharing. > > > > One thing I should have added is that my message is about what it > > would take for me to use autotest and contribute tests. But now I > > realize that you might be going for a different model: > > > > If you're aiming for a different model where autotest integrates > > external test suites (i.e. tests wouldn't be written in autotest.git, > > instead autotest.git would contain snapshots of external test suites), > > then this proposal seems fine. Upstream projects like QEMU would > > develop their own test suite and it would be dropped into autotest or > > a specific autotest instance. > > > > Yes, that's the idea. We want autotest to be the framework, not > (just a) collection of tests. We also want each development team > to implement and maintain their own set of tests, using (or not) > the goodies from autotest at their discretion. > > In summary, autotest is (or is going to be) a framework that > provides: > > - A test runner, with grid/cluster support and advanced >instrumentation > - A devel library and set of utilities for test writers > - A set of pre-built images (JeOS – Just Enough OS) for >test writers > > (attached is a picture showing what we want to achieve) Facepalm right after pressing 'y'. > > If a project has an internal library or set of utilities that can > be of general use, they can be submitted to autotest.git for > inclusion, thus reaching a broader audience. > > A short summary of the plans: > > - Tests can live anywhere and each devel team implements and >maintains their own set of tests > - Usage of the autotest library by test writers is optional > - Tests are scripts returning 0 or error (any language) > - Tests can be run individually or in sets > - Tests should run fast, our target is seconds or a few minutes > - The test runner is smart and “just works” by default >- Trivial standard output (FAIL, PASS, SKIPPED) >- Collect logs, OS data and other stuff (e.g. --record-video!) >- Skip unsupported tests based on the environment they're run >- Multiplex configurations / platforms when on the grid >- Support to run tests “in the cloud” > > There are also lots of small changes and usability improvements > in the pipeline (and feedback right now is very valuable). > > Thanks, > - Ademar > > -- > Ademar de Souza Reis Jr. > Red Hat > > ^[:wq! -- Ademar de Souza Reis Jr. Red Hat ^[:wq! <>
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 10:36:42AM -0600, Anthony Liguori wrote: > On 03/08/2012 10:34 AM, Kevin Wolf wrote: > >Am 08.03.2012 17:10, schrieb Anthony Liguori: > >>>And, of course: > >>>[qemu]$ test-runner --remote=autotest.qemu.org tests.d/block > >> > >>I don't understand what this would do. > > > > From the previous discussions on this topic, I suppose it would task the > >autotest instance at autotest.qemu.org to run the block tests on my git > >tree and send me an email with the results, or something like that. For > >running full tests this would be a great thing to have, I don't want my > >own development boxes to be busy for several hours. Not sure if it can > >work for upstream, though. > > I don't think that's realistic from an upstream PoV. We don't have > the infrastructure today to host that and the companies that do have > that infrastructure behind their firewall I imagine. > Sure, that's a goal for the long long run. We don't have anything similar in autotest either. the host autotest.qemu.org was just an arbitrary example. > > > > >And if you want to translate the syntax, I guess it would be make > >remote=autotest.qemu.org check-block in your proposal. I'm fine with > >either syntax. Sure, make check-whatever looks good. It's less granular, but equally good (and we can easily create the make targets if the tool is flexible enough) -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote: > On 03/08/2012 04:24 PM, Ademar Reis wrote: > >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote: > >>On 03/08/2012 03:02 PM, Ademar Reis wrote: > >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote: > >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote: > >>> > >>> > >>> > >>>> > >>>>>>I expect QEMU to grow tests for anything that involves launching > >>>>>>QEMU directly. Where I would not see QEMU growing tests for is > >>>>>>things like launching QEMU through libvirt. > >>>>> > >>>>>Agree. For QEMU developers, libvirt should not be on the way, the > >>>>>interaction should be minimal or non-existent. > >>>>> > >>>>>That's an area which will require some work in libautotest, > >>>>>because due to previous QE requirements, it now invokes libvirt > >>>>>methods instead of QEMU directly for a lot of stuff. But it can > >>>>>be done and is part of the plan. > >>>> > >>>>If you're talking about libautotest as an abstraction layer for > >>>>launching QEMU, I don't think that's something we should use in > >>>>upstream QEMU. Abstraction layers are okay when the things you are > >>>>abstracting are well defined and relatively static. We're talking > >>>>about testing the tip of a development tree though and coordinating > >>>>changes with an abstraction layer would be counter productive. > >>>> > >>> > >>>Valid point. If I'm a qemu developer and I want to make huge > >>>changes, ideally I shouldn't have to submit patches to > >>>libautotest because the tests in qemu.org will break. > >>> > >>>I don't know how likely such scenario is (Lucas is maitaining > >>>kvm-autotest for years now and the integration hardly breaks) > >>>But we do have a conflict of interests, so lets evaluate a > >>>few options: > >> > >>Because kvm-autotest doesn't have thousands of tests and isn't part > >>of 200+ developers fast paths :-) > >> > >>> > >>>My main interests are: > >>> - remove barriers for developers to write tests > >> > >>This is my number one priority. > >> > >>> - share the testing effort between multiple projects (qemu, > >>> libvirt, ovirt, spice, QE, etc...) > >> > >>This is my number 99 priority. > > > >You have the right to consider the interests of qemu only, but > >qemu is used by several other teams who depend on it to implement > >their own tests. Some collaboration would be extremely beneficial > >for everybody. > > > >Collaboration goes both ways: you get benefits from the > >interaction and code reuse with different projects and teams, but > >it may require a few tradeoffs. > > I think the problem here effectively boils down to "code sharing at any cost". > I may agree with the "sharing at any cost" if you reference parts of the discussion from December. But if you re-read that thread and compare with what we're proposing now, you'll see that we're proposing something very different, based on the requirements previously raised. > Let's look at it in purely practical terms. How many lines of code > would be saved here and what does that code do? > > If the only sharing is some infrastructure bits in libautotest, then > I suspect we're talking in the low 10ks of lines at the absolute > maximum (and I don't think it's even close to that in reality). > Launching QEMU and connecting to QMP isn't that hard. It's in the > hundreds of lines of code. What else is there that's need to write > a test? > > I don't think this is sufficiently compelling to add an > unconditional autotest dependency to QEMU. > > >>I think you could achieve practical code sharing by decomposing > >>kvm-autotest into a set of useful, independent tools. For > >>instance > >> > >>>Now the options: > >>> > >>>1. We keep the invocation stuff in libautotest and when writting > >>>a complex test in qemu.git, the test writter can choose to use it > >>>because of the goodies there. > >>> > >>>Pros: > >>> - L
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Thu, Mar 08, 2012 at 08:07:27PM -0300, Lucas Meneghel Rodrigues wrote: > On 03/08/2012 06:24 PM, Anthony Liguori wrote: > >> > >>Cons: > >>- Lot of code will be duplicated to cover the main code paths: > >>writting tests will require writting/supporting considerable > >>ammount of code (that already exists in autotest). > > > >Again, existence proof that this isn't true. > > Case in point, the virtio test (that uses an auxiliary script to > send data to the host). Can you tell me if both tests cover even > remotely the same amount of functionality? > > https://github.com/autotest/autotest/blob/master/client/tests/kvm/tests/virtio_console.py > > https://github.com/autotest/autotest/blob/master/client/virt/scripts/virtio_console_guest.py > > Here is the qemu-test version > > http://git.qemu.org/?p=qemu-test.git;a=blob;f=tests/virtio-serial.sh;h=e95ae6e0b63758262919702d51a9c83bebe2fb08;hb=master > > What the qemu-test version covers: > * host starts qemu with one virtio console device backed by a file > * guest verifies if the name of the device is correct > * guest writes to the console device > * host verifies if guest wrote to the virtio console > > What the virtio-console covers: > > * Sends data between host and guest back and forth, validates the > data being sent, for both small and large amounts of data, both > random or sequential. > * Tests write/send in blocking, polling, selecting mode, with port > mode sync/async > * Verifies if the maximum amount of ports was created and it's > available in the guest > * Tries lseek on the device > * Verifies if concomitant access to a single port passes or fails > * Verifies data throughput and resource utilization > * Repeats the data transfer with live migration to see if the port > connections survive > * Keeps an eye on the guest OS to see if we have kernel panics, and > if it does, it'll clean up and put the guest in a working state so > other functionality can be checked > * Probably something else I'm forgetting right now. > Thanks for the example Lucas. :) BTW, this is a QE-level test. Simpler tests may be sufficient for developers and for TDD, but, if one implements even a much simpler test using libautotest, he gets these benefits for free: 1. QE or somebody else may extend the test using the autotest goodies the original developer was not aware of. Patches from QE will flow to qemu.git. That's the integration with QE I was talking about. 2. The test can run in the autotest grid, where a large matrix of configurations and usage scenarios are used. That's possible because we don't have much stuff hardcoded. 3. Autotest has a lot of checks for robustness. One of the main problems of keeping thousands of tests running is to keep them stable, without false positives. Lucas: could you ellaborate more on what we get "for free" when we use libautotest, even when writting a very simple developer-level test? > Good luck implementing that in a shell script. I'd love to see how > you implement that amount of coverage in less lines in shell. > > So, more coverage, more code. It's as simple as that. We don't write > code just for the sake of it. > > >>- QE will be alienated from the qemu test effort. There will be > >>no integration between the QE efforts and the maintenance of > >>the qemu developer-level tests. > > > >I think we're a pretty friendly and open community :-) There is no > >reason that QE should be "alienated" unless folks are choosing not to > >participate upstream. > > > >>- You don't get the goodies from autotest (standardized system > >>info collection, video recording, grid support, etc). > >>- The new tests will work only on the master branch. > > > >This last one is a legitimate point that I have considered myself. But I > >think the value of having tests in HEAD outweigh the cost here. > > > >>3. A mix of (1) and (2): we "move" everything under qemu.git, but > >>keep the compatibility layer and provide a "libqemutest" for > >>third-parties. > >> > >>Anybody writting a test that interacts with qemu will use this > >>library: qemu, libvirt, spice, ovirt, QE tests, etc. > > > >I really see this all as over architecting to be honest. > > > >Can someone explain in clear terms (without appealing to maturity, > >flexibility, or any other qualitative aspect) why it takes anything more > >than a few dozen shell functions to write all of the tests that are in > >kvm-autotest test today? > > Clearly as your requirements are different than the ones we had when > the project was written, qemu-test doesn't need any of the stuff > present there and you can do it all with a handful of shell script > functions. > > But to do cover the same things covered in autotest today with the > same requirements, I really doubt you can do the same with the same > handful of shell script functions. See the example about the virtio > test above, not to mention other functionality we have with the > tests, such as make possible to do tes
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Fri, Mar 09, 2012 at 12:59:16AM +0100, Andreas Färber wrote: > Hi, > > Am 08.03.2012 16:00, schrieb Ademar Reis: > > Fully agree, please check my previous email with the plans for > > the new architecture of autotest. > [...] > > Fully agree, please check our previous e-mails with the plans for > > the new architecture. > > FYI your mails are arriving hours late on qemu-devel. Thanks for the heads up. It appears to be normal now, as I see the message I've just sent in the archives. Anyway, the message I was talking about is this one: http://lists.nongnu.org/archive/html/qemu-devel/2012-03/msg01312.html http://lists.nongnu.org/archive/html/qemu-devel/2012-03/msg01313.html (the last one contains the intended attachment) Cheers, - Ademar -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Fri, Mar 09, 2012 at 09:41:05AM +, Stefan Hajnoczi wrote: > On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis wrote: > > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote: > >> On 03/08/2012 04:24 PM, Ademar Reis wrote: > >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote: > >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote: > >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote: > >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote: > >> >>> - QE will be alienated from the qemu test effort. There will be > >> >>> no integration between the QE efforts and the maintenance of > >> >>> the qemu developer-level tests. > >> >> > >> >>I think we're a pretty friendly and open community :-) There is no > >> >>reason that QE should be "alienated" unless folks are choosing not > >> >>to participate upstream. > >> > > >> >For the exact same reasons you as a developer don't want to > >> >implement tests inside autotest, QE won't want to implement tests > >> >for qemu.git. It's out of their comfort zone, just put yourself > >> >on their shoes. > >> > >> This is a really, really poor argument and I hope I don't need to go > >> into details of why. If the primary reason for libautotest is so > >> the people writing tests for QEMU can avoid actually working with > >> the developers of QEMU... we've got a problem. > > > > No, one of the benefits of having libautotest is to *collaborate* > > with QE. I'll explain again: > > > > - As a qemu developer, I don't want to spend my time learning and > > getting involved in autotest, which is a complex QE project > > (I heard this numerous times). > > > > - As a Quality Engineer, I don't want to invest my time learning > > and getting involved into upstream qemu to test HEAD. > > I think this is the key point of the whole discussion - most of the > other topics have been distractions. Both communities do testing but > we test different things and have different priorities. > > For me this has been the big realization from this discussion. I felt > kvm-autotest and qemu should share tests. I was pushing for that but > after following this thread I don't think it makes sense, here's why: > > The Quality Engineer you describe is not a QEMU upstream QE, instead > the QE has a broader and more downstream focus. (This is why > comparisons with WebKit or other upstream projects doing testing are > not valid comparisons.) Lucas, Cleber and the others red-hatters should remembers this from my internal presentation, it was the first point I made: QE and Developers have very different goals and interests. Which is why we're pushing all these changes in autotest. We see opportunities for collaboration, but we do realize the difference. And look: Lucas and Cleber are not QE, they're developers working on the autotest framework/library/whatever. We'll need similar positions inside qemu as the test infra-structure grows. > > There is not enough in common between upstream QEMU testing and > downstream KVM QE to make convergence a win-win. libautotest sounds > like a technical solution to a people problem - the problem is that we > have different priorities. We overlap but at the end of the day we do > different things. We can make a best effort to converge but I don't > see incentives that will make this a success. Creating an abstraction > library will be sub-optimal for both communities. This is the part I don't agree, but in the end it's a matter of opinion. > > I think what's much more valuable is for qemu.git tests to be easily > hooked into autotest. That way you get access to the testing that the > qemu community is doing for free. We get this by default in our plans: any application that can be run and returns 0/error can be run as a test inside autotest. What we're asking is the *possibility* of a developer using libautotest when writting a complex test. - We're not asking everybody to use autotest - We're not saying autotest is better for everything - We're not requiring you to install autotest on your machine (even though the process will be trivial) - We're asking that, if a developer is going to write a test together with his patch to submit to qemu.git, he should be allowed to use libautotest at his own discretion... - Ditto for using the test-runner: even if tests are simple scripts, there will be benefits in using our optional test-r
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Fri, Mar 09, 2012 at 08:13:45AM -0600, Anthony Liguori wrote: > On 03/09/2012 06:48 AM, Lucas Meneghel Rodrigues wrote: > > Look at how this discussion started. We've been discussing testing > on qemu-devel at excruciating length and detail and have finally > come to something of a consensus. AFAIK, no one from autotest has > participated in those discussions which is fair as I'm sure ya'll > don't read qemu-devel religiously. The discussion started back in December and this is a follow-up RFC based on what was said back then. (http://lists.nongnu.org/archive/html/qemu-devel/2011-12/msg02434.html) > > Then we see this note that more or less declares, this is how QEMU > should do all of its testing. What reaction did you really expect > there to be? :-) This is a clear misunderstanding: we've repeated numerous times that libautotest and the test runner are optional. I don't know how much I have to stress this out. Technically speaking, the use of autotest would be enabled by "import autotest", "source $AUTOTESTDIR/libautotest.sh" or something similar. This can be easily detected by the test runner, which could skip such tests if autotest is not available. We're also offering a global test-runner that we believe will bring advantages to anybody running tests that follow the simple requirement of being a script returning 0/error. > > >It's just that this extra stuff is potentially not interesting to the goal of > >doing developer level regression testing of qemu alone. > > I think we need to focus this discussion on concrete technical > proposals. If the proposal is, QEMU should use libautotest, we need > to start with an awful lot more detail about libautotest does and > what functions it provides. The discussion diverged long ago. So much that the initial test examples, bootstrap procedure, directory structure, test runner output, and our request for requiements was all kind of forgotten. Anyway, we'll keep working on autotest targeting the other projects which will benefit from it. As I said in the other e-mail, we'll probably revisit this topic in the KVM forum in person, with more code to show and hopefully with some beers around us. :-) Cheers, - Ademar -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
Re: [Qemu-devel] [RFC] Future goals for autotest and virtualization tests
On Fri, Mar 09, 2012 at 02:54:23PM +, Stefan Hajnoczi wrote: > On Fri, Mar 9, 2012 at 2:00 PM, Ademar Reis wrote: > > On Fri, Mar 09, 2012 at 09:41:05AM +, Stefan Hajnoczi wrote: > >> On Thu, Mar 8, 2012 at 11:51 PM, Ademar Reis wrote: > >> > On Thu, Mar 08, 2012 at 05:21:44PM -0600, Anthony Liguori wrote: > >> >> On 03/08/2012 04:24 PM, Ademar Reis wrote: > >> >> >On Thu, Mar 08, 2012 at 03:24:15PM -0600, Anthony Liguori wrote: > >> >> >>On 03/08/2012 03:02 PM, Ademar Reis wrote: > >> >> >>>On Thu, Mar 08, 2012 at 01:16:58PM -0600, Anthony Liguori wrote: > >> >> >>>>On 03/08/2012 11:59 AM, Ademar Reis wrote: > >> >> >>> - QE will be alienated from the qemu test effort. There will be > >> >> >>> no integration between the QE efforts and the maintenance of > >> >> >>> the qemu developer-level tests. > >> >> >> > >> >> >>I think we're a pretty friendly and open community :-) There is no > >> >> >>reason that QE should be "alienated" unless folks are choosing not > >> >> >>to participate upstream. > >> >> > > >> >> >For the exact same reasons you as a developer don't want to > >> >> >implement tests inside autotest, QE won't want to implement tests > >> >> >for qemu.git. It's out of their comfort zone, just put yourself > >> >> >on their shoes. > >> >> > >> >> This is a really, really poor argument and I hope I don't need to go > >> >> into details of why. If the primary reason for libautotest is so > >> >> the people writing tests for QEMU can avoid actually working with > >> >> the developers of QEMU... we've got a problem. > >> > > >> > No, one of the benefits of having libautotest is to *collaborate* > >> > with QE. I'll explain again: > >> > > >> > - As a qemu developer, I don't want to spend my time learning and > >> > getting involved in autotest, which is a complex QE project > >> > (I heard this numerous times). > >> > > >> > - As a Quality Engineer, I don't want to invest my time learning > >> > and getting involved into upstream qemu to test HEAD. > >> > >> I think this is the key point of the whole discussion - most of the > >> other topics have been distractions. Both communities do testing but > >> we test different things and have different priorities. > >> > >> For me this has been the big realization from this discussion. I felt > >> kvm-autotest and qemu should share tests. I was pushing for that but > >> after following this thread I don't think it makes sense, here's why: > >> > >> The Quality Engineer you describe is not a QEMU upstream QE, instead > >> the QE has a broader and more downstream focus. (This is why > >> comparisons with WebKit or other upstream projects doing testing are > >> not valid comparisons.) > > > > Lucas, Cleber and the others red-hatters should remembers this > > from my internal presentation, it was the first point I made: > > QE and Developers have very different goals and interests. > > > > Which is why we're pushing all these changes in autotest. We see > > opportunities for collaboration, but we do realize the difference. > > > > And look: Lucas and Cleber are not QE, they're developers working > > on the autotest framework/library/whatever. We'll need similar > > positions inside qemu as the test infra-structure grows. > > I don't understand this last paragraph. If qemu.git upstream was > doing full-scale QE it would work fine because the differences that > I've described and you also have pointed out would be absent. > In order to have QEMU working in full "TDD Mode" (a current goal), I predict developers assigned to the maintenance of the in-house test infrastructure (qemu-test) will be needed, on positions similar to what Lucas and Cleber currently do with autotest. Only time will tell. -- Ademar de Souza Reis Jr. Red Hat ^[:wq!
Re: [Qemu-devel] [PATCH 0/2] kvm_stat: add man page and tui column headers
On Mon, Mar 02, 2015 at 05:29:05PM -0600, Stefan Hajnoczi wrote: > This series makes kvm_stat easier to learn for newcomers by adding a man page > and text UI column headers. > > Stefan Hajnoczi (2): > kvm_stat: add column headers to text UI > kvm_stat: add kvm_stat.1 man page > > Makefile | 9 > scripts/kvm/kvm_stat | 5 - > scripts/kvm/kvm_stat.texi | 55 > +++ > 3 files changed, 68 insertions(+), 1 deletion(-) > create mode 100644 scripts/kvm/kvm_stat.texi > Patches are close to trivial, but anyway, I reviewed and tested them. Reviewed-by: Ademar Reis -- Ademar Reis Red Hat ^[:wq!
Re: [RFC PATCH-for-5.2] gitlab-ci: Do not automatically run Avocado integration tests anymore
On Mon, Nov 30, 2020 at 10:31:09AM +, Daniel P. Berrangé wrote: > On Fri, Nov 27, 2020 at 07:29:31PM +0100, Thomas Huth wrote: > > On 27/11/2020 18.57, Philippe Mathieu-Daudé wrote: > > > On 11/27/20 6:47 PM, Thomas Huth wrote: > > >> On 27/11/2020 18.41, Philippe Mathieu-Daudé wrote: > > >>> We lately realized that the Avocado framework was not designed > > >>> to be regularly run on CI environments. Therefore, as of 5.2 > > >>> we deprecate the gitlab-ci jobs using Avocado. To not disrupt > > >>> current users, it is possible to keep the current behavior by > > >>> setting the QEMU_CI_INTEGRATION_JOBS_PRE_5_2_RELEASE variable > > >>> (see [*]). > > >>> From now on, using these jobs (or adding new tests to them) > > >>> is strongly discouraged. > > >>> > > >>> Tests based on Avocado will be ported to new job schemes during > > >>> the next releases, with better documentation and templates. > > >> > > >> Why should we disable the jobs by default as long as there is no > > >> replacement > > >> available yet? > > > > > > Why keep it enabled if it is failing randomly > > > > We can still disable single jobs if they are not stable, but that's no > > reason to disable all of them by default, is it? > > Agreed, the jobs which are known to be broken or unreliable should > be unconditonally disabled in QEMU as a whole. This isn't specific > to gitlab config - the qemu build makefiles/mesonfiles should disable > the problem tests entirely, as we don't want developers wasting time > running them locally either if they're known to be broken/unreliable. > The problem is identifying when a test is broken/unreliable. No complex test is 100% reliable: change the environment, (configuration, build options, platform, etc) and any test complex enough will start to fail. I've worked in projects orders of magnitude simpler than QEMU and that was a golden rule. Testing QEMU is *HARD*. Which is why I defend test-automation separated from CI: * Have a stable CI with tests cherry-picked by whoever is maintaining a particular CI runner (we shouldn't have orphan runners). * Have as many tests as possible in the git repo: maintained, reviewed and run (outside of a CI) by people who care about them. I absolutely agree with you that maintainers and developers should care and our goal should be a gating CI. The question is how to create a strategy and a plan to get there. Forcing people to care rarely works. Thanks. - Ademar -- Ademar Reis Jr Red Hat ^[:wq!
