On May 23, 2019 3:45 PM, "Cleber Rosa" <cr...@redhat.com> wrote: > > > > ----- Original Message ----- > > From: "Aleksandar Markovic" <aleksandar.m.m...@gmail.com> > > To: "Cleber Rosa" <cr...@redhat.com> > > Cc: "Wainer dos Santos Moschetta" <waine...@redhat.com>, "Aleksandar Markovic" <amarko...@wavecomp.com>, > > qemu-devel@nongnu.org, "Aleksandar Rikalo" <arik...@wavecomp.com>, "Eduardo Habkost" <ehabk...@redhat.com>, > > "Aurelien Jarno" <aurel...@aurel32.net>, "Philippe Mathieu-Daudé" < f4...@amsat.org> > > Sent: Wednesday, May 22, 2019 6:43:54 PM > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests > > > > On May 22, 2019 11:46 PM, "Cleber Rosa" <cr...@redhat.com> wrote: > > > > > > > > > > > > ----- Original Message ----- > > > > From: "Eduardo Habkost" <ehabk...@redhat.com> > > > > To: "Philippe Mathieu-Daudé" <f4...@amsat.org> > > > > Cc: qemu-devel@nongnu.org, "Aleksandar Rikalo" <arik...@wavecomp.com >, > > "Aleksandar Markovic" > > > > <aleksandar.m.m...@gmail.com>, "Aleksandar Markovic" < > > amarko...@wavecomp.com>, "Cleber Rosa" <cr...@redhat.com>, > > > > "Aurelien Jarno" <aurel...@aurel32.net>, "Wainer dos Santos Moschetta" < > > waine...@redhat.com> > > > > Sent: Wednesday, May 22, 2019 5:12:30 PM > > > > Subject: Re: [Qemu-devel] [PATCH 0/4] mips: Add more Avocado tests > > > > > > > > On Tue, May 21, 2019 at 01:19:06AM +0200, Philippe Mathieu-Daudé wrote: > > > > > Hi, > > > > > > > > > > It was a rainy week-end here, so I invested it to automatize some > > > > > of my MIPS tests. > > > > > > > > > > The BootLinuxSshTest is not Global warming friendly, it is not > > > > > meant to run on a CI system but rather on a workstation previous > > > > > to post a pull request. > > > > > It can surely be improved, but it is a good starting point. > > > > > > > > Until we actually have a mechanism to exclude the test case on > > > > travis-ci, I will remove patch 4/4 from the queue. Aleksandar, > > > > please don't merge patch 4/4 yet or it will break travis-ci. > > > > > > > > Cleber, Wainer, is it already possible to make "avocado run" skip > > > > tests tagged with "slow"? > > > > > > > > > > The mechanism exists, but we haven't tagged any test so far as slow. > > > > > > > Cleber, > > > > For the test from patch 4/4, there is no dilemma - it should be in the > > “slow” group, as Philippe envisioned and said, so that it is not humpered > > with stricter requirements for “fast” (default) group. Could you explain us > > how to do it, so that we can hopefully finally proceed? > > > > Hi Aleksandar, > > The point is that there's no "group" definition at this point. This is the > core of the discussion. > > I think we're close to converging to something simple and effective. Please > let us know what you think of the proposals given. > > Thanks! > - Cleber. >
Cleber, hi. Thanks for responding. My views are very similar to Philippe's, but I will provide you with more details of our (mips) perspective. As far as black/whitelist issues that is a moot point for us - we only want to be able to have a way to tag a test within the test itself (so, without updating some common files, external lists,etc.) In general, we would like to have a test environment where we would be able to test what WE deem suitable for us, without feeling that we bother you or anybody else, or that we are bothered by others. Let me give you a little extreme example: Let's say we want a complex test that downloads components from let's say fifty internet location, executes zillion test cases, and last two days. I wouldn't like anybody to ask me “Why would you that?” or tell me “You can't do this.” or say “No, we did not anticipate such tests, patch rejected.” I (we, people from mips) should be able to define what I (we) need. Having such test would be a big deal for me, not only that I could run it manually or automatically every weekend, but I could ask submitters of critical changes: “Did you run this test that we have in Avocado dir?”, without specifying test details, procedures, etc. All this is a BIG deal for me. On the other hand, I agree that certain group of tests (envisioned for daily or so Travis CI) should have some stricter limitations and structure. But right now I feel humpered by it, and this is counterproductive. So, we want freedom, responsibility and ownersheep of our tests. Please give us the opportunity to get down on business and start writing tests and start testing. Yours, Aleksandar > > Gratefully, > > Aleksandar > > > > > Should we define/document a criteria for a test to be slow? Given > > > that this is highly subjective, we have to think of: > > > > > > * Will we consider the average or maximum run time (the timeout > > > definition)? > > > > > > * For a single test, what is "slow"? Some rough numbers from Travis > > > CI[1] to help us with guidelines: > > > - boot_linux_console.py:BootLinuxConsole.test_x86_64_pc: PASS (6.04 s) > > > - boot_linux_console.py:BootLinuxConsole.test_arm_virt: PASS (2.91 s) > > > - > > linux_initrd.py:LinuxInitrd.test_with_2gib_file_should_work_with_linux_v4_16: > > PASS (18.14 s) > > > - boot_linux.py:BootLinuxAarch64.test_virt: PASS (396.88 s) > > > > > > * Do we want to set a maximum job timeout? This way we can skip > > > tests after a given amount of time has passed. Currently we interrupt > > > the test running when the job timeout is reached, but it's possible > > > to add a option so that no new tests will be started, but currently > > > running ones will be waited on. > > > > > > Regards, > > > - Cleber. > > > > > > [1] - https://travis-ci.org/clebergnu/qemu/jobs/535967210#L3518 > > > > > > > -- > > > > Eduardo > > > > > >
