On Feb 26, 2014 7:37 PM, "Matt Riedemann" <mrie...@linux.vnet.ibm.com> wrote: > > > > On 2/25/2014 7:46 PM, Matt Riedemann wrote: >> >> >> >> On 2/12/2014 1:57 PM, Matthew Treinish wrote: >>> >>> On Wed, Feb 12, 2014 at 11:32:39AM -0700, Matt Riedemann wrote: >>>> >>>> >>>> >>>> On 1/17/2014 8:34 AM, Matthew Treinish wrote: >>>>> >>>>> On Fri, Jan 17, 2014 at 08:32:19AM -0500, David Kranz wrote: >>>>>> >>>>>> On 01/16/2014 10:56 PM, Matthew Treinish wrote: >>>>>>> >>>>>>> Hi everyone, >>>>>>> >>>>>>> With some recent changes made to Tempest compatibility with >>>>>>> nosetests is going >>>>>>> away. We've started using newer features that nose just doesn't >>>>>>> support. One >>>>>>> example of this is that we've started using testscenarios and >>>>>>> we're planning to >>>>>>> do this in more places moving forward. >>>>>>> >>>>>>> So at Icehouse-3 I'm planning to push the patch out to remove >>>>>>> nosetests from the >>>>>>> requirements list and all the workarounds and references to nose >>>>>>> will be pulled >>>>>>> out of the tree. Tempest will also start raising an unsupported >>>>>>> exception when >>>>>>> you try to run it with nose so that there isn't any confusion on >>>>>>> this moving >>>>>>> forward. We talked about doing this at summit briefly and I've >>>>>>> brought it up a >>>>>>> couple of times before, but I believe it is time to do this now. I >>>>>>> feel for >>>>>>> tempest to move forward we need to do this now so that there isn't >>>>>>> any ambiguity >>>>>>> as we add even more features and new types of testing. >>>>>> >>>>>> I'm with you up to here. >>>>>>> >>>>>>> >>>>>>> Now, this will have implications for people running tempest with >>>>>>> python 2.6 >>>>>>> since up until now we've set nosetests. There is a workaround for >>>>>>> getting >>>>>>> tempest to run with python 2.6 and testr see: >>>>>>> >>>>>>> https://review.openstack.org/#/c/59007/1/README.rst >>>>>>> >>>>>>> but essentially this means that when nose is marked as unsupported >>>>>>> on tempest >>>>>>> python 2.6 will also be unsupported by Tempest. (which honestly it >>>>>>> basically has >>>>>>> been for while now just we've gone without making it official) >>>>>> >>>>>> The way we handle different runners/os can be categorized as "tested >>>>>> in gate", "unsupported" (should work, possibly some hacks needed), >>>>>> and "hostile". At present, both nose and py2.6 I would say are in >>>>>> the unsupported category. The title of this message and the content >>>>>> up to here says we are moving nose to the hostile category. With >>>>>> only 2 months to feature freeze I see no justification in moving >>>>>> py2.6 to the hostile category. I don't see what new testing features >>>>>> scheduled for the next two months will be enabled by saying that >>>>>> tempest cannot and will not run on 2.6. It has been agreed I think >>>>>> by all projects that py2.6 will be dropped in J. It is OK that py2.6 >>>>>> will require some hacks to work and if in the next few months it >>>>>> needs a few more then that is ok. If I am missing another connection >>>>>> between the py2.6 and nose issues, please explain. >>>>>> >>>>> >>>>> So honestly we're already at this point in tempest. Nose really just >>>>> doesn't >>>>> work with tempest, and we're adding more features to tempest, your >>>>> negative test >>>>> generator being one of them, that interfere further with nose. I've >>>>> seen several >>>> >>>> >>>> I disagree here, my team is running Tempest API, CLI and scenario >>>> tests every day with nose on RHEL 6 with minimal issues. I had to >>>> workaround the negative test discovery by simply sed'ing that out of >>>> the tests before running it, but that's acceptable to me until we >>>> can start testing on RHEL 7. Otherwise I'm completely OK with >>>> saying py26 isn't really supported and isn't used in the gate, and >>>> it's a buyer beware situation to make it work, which includes >>>> pushing up trivial patches to make it work (which I did a few of >>>> last week, and they were small syntax changes or usages of >>>> testtools). >>>> >>>> I don't understand how the core projects can be running unit tests >>>> in the gate on py26 but our functional integration project is going >>>> to actively go out and make it harder to run Tempest with py26, that >>>> sucks. >>>> >>>> If we really want to move the test project away from py26, let's >>>> make the concerted effort to get the core projects to move with it. >>> >>> >>> So as I said before the python 2.6 story for tempest remains the same >>> after this >>> change. The only thing that we'll be doing is actively preventing nose >>> from >>> working with tempest. >>> >>>> >>>> And FWIW, I tried the discover.py patch with unittest2 and >>>> testscenarios last week and either I botched it, it's not documented >>>> properly on how to apply it, or I screwed something up, but it >>>> didn't work for me, so I'm not convinced that's the workaround. >>>> >>>> What's the other option for running Tempest on py26 (keeping RHEL 6 >>>> in mind)? Using tox with testr and pip? I'm doing this all >>>> single-node. >>> >>> >>> Yes, that is what the discover patch is used to enable. By disabling >>> nose the >>> only path to run tempest with py2.6 is to use testr. (which is what it >>> always >>> should have been) >>> >>> Attila confirmed it was working here: >>> http://fpaste.org/76651/32143139/ >>> in that example he applies 2 patches the second one is currently in >>> the gate for >>> tempest. (https://review.openstack.org/#/c/72388/ ) So all that needs >>> to be done >>> is to apply that discover patch: >>> >>> https://code.google.com/p/unittest-ext/issues/detail?id=79 >>> >>> (which I linked to before) >>> >>> Then tempest should run more or less the same between 2.7 and 2.6. >>> (The only >>> difference I've seen is in how skips are handled) >>> >>>> >>>>> patches this cycle that attempted to introduce incorrect behavior >>>>> while trying >>>>> to fix compatibility with nose. That's why I think we need a clear >>>>> message on >>>>> this sooner than later. Which is why I'm proposing actively raising >>>>> an error >>>>> when things are run with nose upfront so there isn't any illusion >>>>> that things >>>>> are expected to work. >>>>> >>>>> This doesn't necessarily mean we're moving python 2.6 to the hostile >>>>> category. >>>>> Nose support is independent of python 2.6 support. Py26 I would >>>>> still consider >>>>> to be unsupported, the issue is that the hack to make py26 work is >>>>> outside of >>>>> tempest. This is why we've recommended that people using python 2.6 >>>>> run with >>>>> nose, which really is no longer an option. Attila's abandoned patch >>>>> that I >>>>> linked above documents points to this bug with a patch to discover >>>>> which is >>>>> need to get python 2.6 working with tempest and testr: >>>>> >>>>> https://code.google.com/p/unittest-ext/issues/detail?id=79 >>> >>> >>> _______________________________________________ >>> OpenStack-dev mailing list >>> OpenStack-dev@lists.openstack.org >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> One question I had was is there an easy way to setup a config file to >> specify the test bucket and what can be excluded, like you can with >> nose.cfg and nose? We used that for filtering out API tests that didn't >> work with the PowerVM driver in Nova but I'm not aware of something >> similar with testr. >> > > Another question I guess, and I know this is too late now, but the nosexcover package allowed us to get xunit xml results for the test runs and then jenkins has a nice way to process that xml and keep a history of failures and dig into the stack traces, how long the tests take, trends, etc. I'm assuming there is nothing like that available with testr? > > > -- > > Thanks, > > Matt Riedemann > > > _______________________________________________ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
There is a subunit to xunit converter bundled with python-subunit. subunit2xunit maybe? On a phone so hard to check. Also fwiw using Jenkins to mangle that data doesn't scale well which is why we don't do it. This may or may not be a problem. In particular the xunit plugin has a global lock and jobs that use it can't finish until the previous job has had its xunit processed. This is terrible. Clark
_______________________________________________ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
