On Tue, Feb 25, 2014 at 07:46:23PM -0600, Matt Riedemann wrote: > > > On 2/12/2014 1:57 PM, Matthew Treinish wrote: > >On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote: > >> > >> > >>On 1/17/2014 8:34 AM, Matthew Treinish wrote: > >>>On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote: > >>>>On 01/16/2014 10:56 PM, Matthew Treinish wrote: > >>>>>Hi everyone, > >>>>> > >>>>>With some recent changes made to Tempest compatibility with nosetests is > >>>>>going > >>>>>away. We've started using newer features that nose just doesn't support. > >>>>>One > >>>>>example of this is that we've started using testscenarios and we're > >>>>>planning to > >>>>>do this in more places moving forward. > >>>>> > >>>>>So at Icehouse-3 I'm planning to push the patch out to remove nosetests > >>>>>from the > >>>>>requirements list and all the workarounds and references to nose will be > >>>>>pulled > >>>>>out of the tree. Tempest will also start raising an unsupported > >>>>>exception when > >>>>>you try to run it with nose so that there isn't any confusion on this > >>>>>moving > >>>>>forward. We talked about doing this at summit briefly and I've brought > >>>>>it up a > >>>>>couple of times before, but I believe it is time to do this now. I feel > >>>>>for > >>>>>tempest to move forward we need to do this now so that there isn't any > >>>>>ambiguity > >>>>>as we add even more features and new types of testing. > >>>>I'm with you up to here. > >>>>> > >>>>>Now, this will have implications for people running tempest with python > >>>>>2.6 > >>>>>since up until now we've set nosetests. There is a workaround for getting > >>>>>tempest to run with python 2.6 and testr see: > >>>>> > >>>>>https://review.openstack.org/#/c/59007/1/README.rst > >>>>> > >>>>>but essentially this means that when nose is marked as unsupported on > >>>>>tempest > >>>>>python 2.6 will also be unsupported by Tempest. (which honestly it > >>>>>basically has > >>>>>been for while now just we've gone without making it official) > >>>>The way we handle different runners/os can be categorized as "tested > >>>>in gate", "unsupported" (should work, possibly some hacks needed), > >>>>and "hostile". At present, both nose and py2.6 I would say are in > >>>>the unsupported category. The title of this message and the content > >>>>up to here says we are moving nose to the hostile category. With > >>>>only 2 months to feature freeze I see no justification in moving > >>>>py2.6 to the hostile category. I don't see what new testing features > >>>>scheduled for the next two months will be enabled by saying that > >>>>tempest cannot and will not run on 2.6. It has been agreed I think > >>>>by all projects that py2.6 will be dropped in J. It is OK that py2.6 > >>>>will require some hacks to work and if in the next few months it > >>>>needs a few more then that is ok. If I am missing another connection > >>>>between the py2.6 and nose issues, please explain. > >>>> > >>> > >>>So honestly we're already at this point in tempest. Nose really just > >>>doesn't > >>>work with tempest, and we're adding more features to tempest, your > >>>negative test > >>>generator being one of them, that interfere further with nose. I've seen > >>>several > >> > >>I disagree here, my team is running Tempest API, CLI and scenario > >>tests every day with nose on RHEL 6 with minimal issues. I had to > >>workaround the negative test discovery by simply sed'ing that out of > >>the tests before running it, but that's acceptable to me until we > >>can start testing on RHEL 7. Otherwise I'm completely OK with > >>saying py26 isn't really supported and isn't used in the gate, and > >>it's a buyer beware situation to make it work, which includes > >>pushing up trivial patches to make it work (which I did a few of > >>last week, and they were small syntax changes or usages of > >>testtools). > >> > >>I don't understand how the core projects can be running unit tests > >>in the gate on py26 but our functional integration project is going > >>to actively go out and make it harder to run Tempest with py26, that > >>sucks. > >> > >>If we really want to move the test project away from py26, let's > >>make the concerted effort to get the core projects to move with it. > > > >So as I said before the python 2.6 story for tempest remains the same after > >this > >change. The only thing that we'll be doing is actively preventing nose from > >working with tempest. > > > >> > >>And FWIW, I tried the discover.py patch with unittest2 and > >>testscenarios last week and either I botched it, it's not documented > >>properly on how to apply it, or I screwed something up, but it > >>didn't work for me, so I'm not convinced that's the workaround. > >> > >>What's the other option for running Tempest on py26 (keeping RHEL 6 > >>in mind)? Using tox with testr and pip? I'm doing this all > >>single-node. > > > >Yes, that is what the discover patch is used to enable. By disabling nose the > >only path to run tempest with py2.6 is to use testr. (which is what it always > >should have been) > > > >Attila confirmed it was working here: > >http://fpaste.org/76651/32143139/ > >in that example he applies 2 patches the second one is currently in the gate > >for > >tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs to be > >done > >is to apply that discover patch: > > > >https://code.google.com/p/unittest-ext/issues/detail?id=79 > > > >(which I linked to before) > > > >Then tempest should run more or less the same between 2.7 and 2.6. (The only > >difference I've seen is in how skips are handled) > > > >> > >>>patches this cycle that attempted to introduce incorrect behavior while > >>>trying > >>>to fix compatibility with nose. That's why I think we need a clear message > >>>on > >>>this sooner than later. Which is why I'm proposing actively raising an > >>>error > >>>when things are run with nose upfront so there isn't any illusion that > >>>things > >>>are expected to work. > >>> > >>>This doesn't necessarily mean we're moving python 2.6 to the hostile > >>>category. > >>>Nose support is independent of python 2.6 support. Py26 I would still > >>>consider > >>>to be unsupported, the issue is that the hack to make py26 work is outside > >>>of > >>>tempest. This is why we've recommended that people using python 2.6 run > >>>with > >>>nose, which really is no longer an option. Attila's abandoned patch that I > >>>linked above documents points to this bug with a patch to discover which is > >>>need to get python 2.6 working with tempest and testr: > >>> > >>>https://code.google.com/p/unittest-ext/issues/detail?id=79 > > > >_______________________________________________ > >OpenStack-dev mailing list > >OpenStack-dev@lists.openstack.org > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > One question I had was is there an easy way to setup a config file > to specify the test bucket and what can be excluded, like you can > with nose.cfg and nose? We used that for filtering out API tests > that didn't work with the PowerVM driver in Nova but I'm not aware > of something similar with testr.
So I'm not sure if it's exactly what you're looking for but testr has the --load-list option which you can use to specify a file which lists the tests you want to run. I don't think there is a method to exclude tests besides using a regex filter right now. There is a bug open about this: https://bugs.launchpad.net/testrepository/+bug/1208610 So I can see you doing this 2 ways either writing a little script that will generate a list file by doing something like: 1. testr list-tests > file 2. remove excludes from file 3. testr run --load-list file or making a long unwieldy regex that excludes the tests you need to. Something like what I did here: https://review.openstack.org/#/c/51275/4/tox.ini -Matt Treinish _______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
