This is another example of achieving the same result (exclusion from a list): https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element s/tempest/tests2skip.py https://git.openstack.org/cgit/openstack/tripleo-image-elements/tree/element s/tempest/tests2skip.txt
andrea -----Original Message----- From: Matthew Treinish [mailto:mtrein...@kortar.org] Sent: 27 February 2014 15:49 To: OpenStack Development Mailing List (not for usage questions) Subject: Re: [openstack-dev] [QA] The future of nosetests with Tempest On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote: > > > On 2/12/2014 1:57 PM, Matthew Treinish wrote: > >On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote: > >> > >> > >>On 1/17/2014 8:34 AM, Matthew Treinish wrote: > >>>On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote: > >>>>On 01/16/2014 10:56 PM, Matthew Treinish wrote: > >>>>>Hi everyone, > >>>>> > >>>>>With some recent changes made to Tempest compatibility with > >>>>>nosetests is going away. We've started using newer features that > >>>>>nose just doesn't support. One example of this is that we've > >>>>>started using testscenarios and we're planning to do this in more places moving forward. > >>>>> > >>>>>So at Icehouse-3 I'm planning to push the patch out to remove > >>>>>nosetests from the requirements list and all the workarounds and > >>>>>references to nose will be pulled out of the tree. Tempest will > >>>>>also start raising an unsupported exception when you try to run > >>>>>it with nose so that there isn't any confusion on this moving > >>>>>forward. We talked about doing this at summit briefly and I've > >>>>>brought it up a couple of times before, but I believe it is time > >>>>>to do this now. I feel for tempest to move forward we need to do this now so that there isn't any ambiguity as we add even more features and new types of testing. > >>>>I'm with you up to here. > >>>>> > >>>>>Now, this will have implications for people running tempest with > >>>>>python 2.6 since up until now we've set nosetests. There is a > >>>>>workaround for getting tempest to run with python 2.6 and testr see: > >>>>> > >>>>>https://review.openstack.org/#/c/59007/1/README.rst > >>>>> > >>>>>but essentially this means that when nose is marked as > >>>>>unsupported on tempest python 2.6 will also be unsupported by > >>>>>Tempest. (which honestly it basically has been for while now just > >>>>>we've gone without making it official) > >>>>The way we handle different runners/os can be categorized as > >>>>"tested in gate", "unsupported" (should work, possibly some hacks > >>>>needed), and "hostile". At present, both nose and py2.6 I would > >>>>say are in the unsupported category. The title of this message and > >>>>the content up to here says we are moving nose to the hostile > >>>>category. With only 2 months to feature freeze I see no > >>>>justification in moving > >>>>py2.6 to the hostile category. I don't see what new testing > >>>>features scheduled for the next two months will be enabled by > >>>>saying that tempest cannot and will not run on 2.6. It has been > >>>>agreed I think by all projects that py2.6 will be dropped in J. It > >>>>is OK that py2.6 will require some hacks to work and if in the > >>>>next few months it needs a few more then that is ok. If I am > >>>>missing another connection between the py2.6 and nose issues, please explain. > >>>> > >>> > >>>So honestly we're already at this point in tempest. Nose really > >>>just doesn't work with tempest, and we're adding more features to > >>>tempest, your negative test generator being one of them, that > >>>interfere further with nose. I've seen several > >> > >>I disagree here, my team is running Tempest API, CLI and scenario > >>tests every day with nose on RHEL 6 with minimal issues. I had to > >>workaround the negative test discovery by simply sed'ing that out of > >>the tests before running it, but that's acceptable to me until we > >>can start testing on RHEL 7. Otherwise I'm completely OK with > >>saying py26 isn't really supported and isn't used in the gate, and > >>it's a buyer beware situation to make it work, which includes > >>pushing up trivial patches to make it work (which I did a few of > >>last week, and they were small syntax changes or usages of > >>testtools). > >> > >>I don't understand how the core projects can be running unit tests > >>in the gate on py26 but our functional integration project is going > >>to actively go out and make it harder to run Tempest with py26, that > >>sucks. > >> > >>If we really want to move the test project away from py26, let's > >>make the concerted effort to get the core projects to move with it. > > > >So as I said before the python 2.6 story for tempest remains the same > >after this change. The only thing that we'll be doing is actively > >preventing nose from working with tempest. > > > >> > >>And FWIW, I tried the discover.py patch with unittest2 and > >>testscenarios last week and either I botched it, it's not documented > >>properly on how to apply it, or I screwed something up, but it > >>didn't work for me, so I'm not convinced that's the workaround. > >> > >>What's the other option for running Tempest on py26 (keeping RHEL 6 > >>in mind)? Using tox with testr and pip? I'm doing this all > >>single-node. > > > >Yes, that is what the discover patch is used to enable. By disabling > >nose the only path to run tempest with py2.6 is to use testr. (which > >is what it always should have been) > > > >Attila confirmed it was working here: > >http://fpaste.org/76651/32143139/ > >in that example he applies 2 patches the second one is currently in > >the gate for tempest. (https://review.openstack.org/#/c/72388/ ) So > >all that needs to be done is to apply that discover patch: > > > >https://code.google.com/p/unittest-ext/issues/detail?id=79 > > > >(which I linked to before) > > > >Then tempest should run more or less the same between 2.7 and 2.6. > >(The only difference I've seen is in how skips are handled) > > > >> > >>>patches this cycle that attempted to introduce incorrect behavior > >>>while trying to fix compatibility with nose. That's why I think we > >>>need a clear message on this sooner than later. Which is why I'm > >>>proposing actively raising an error when things are run with nose > >>>upfront so there isn't any illusion that things are expected to work. > >>> > >>>This doesn't necessarily mean we're moving python 2.6 to the hostile category. > >>>Nose support is independent of python 2.6 support. Py26 I would > >>>still consider to be unsupported, the issue is that the hack to > >>>make py26 work is outside of tempest. This is why we've recommended > >>>that people using python 2.6 run with nose, which really is no > >>>longer an option. Attila's abandoned patch that I linked above > >>>documents points to this bug with a patch to discover which is need to get python 2.6 working with tempest and testr: > >>> > >>>https://code.google.com/p/unittest-ext/issues/detail?id=79 > > > >_______________________________________________ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > One question I had was is there an easy way to setup a config file to > specify the test bucket and what can be excluded, like you can with > nose.cfg and nose? We used that for filtering out API tests that > didn't work with the PowerVM driver in Nova but I'm not aware of > something similar with testr. So I'm not sure if it's exactly what you're looking for but testr has the --load-list option which you can use to specify a file which lists the tests you want to run. I don't think there is a method to exclude tests besides using a regex filter right now. There is a bug open about this: https://bugs.launchpad.net/testrepository/+bug/1208610 So I can see you doing this 2 ways either writing a little script that will generate a list file by doing something like: 1. testr list-tests > file 2. remove excludes from file 3. testr run --load-list file or making a long unwieldy regex that excludes the tests you need to. Something like what I did here: https://review.openstack.org/#/c/51275/4/tox.ini -Matt Treinish _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
