On Tue, Jul 10, 2018 at 03:16:14PM -0400, Matthew Treinish wrote: > On Tue, Jul 10, 2018 at 10:16:37AM +0100, Chris Dent wrote: > > On Mon, 9 Jul 2018, Matthew Treinish wrote: > > > > > It's definitely a bug, and likely a bug in stestr (or one of the lower > > > level > > > packages like testtools or python-subunit), because that's what's > > > generating > > > the return code. Tox just looks at the return code from the commands to > > > figure > > > out if things were successful or not. I'm a bit surprised by this though I > > > thought we covered the unxsuccess and xfail cases because I would have > > > expected > > > cdent to file a bug if it didn't. Looking at the stestr tests we don't > > > have > > > coverage for the unxsuccess case so I can see how this slipped through. > > > > This was reported on testrepository some years ago and a bit of > > analysis was done: https://bugs.launchpad.net/testrepository/+bug/1429196 > > > > This actually helps a lot, because I was seeing the same issue when I tried > writing a quick patch to address this. When I manually poked the TestResult > object it didn't have anything in the unxsuccess list. So instead of relying > on that I wrote this patch: > > https://github.com/mtreinish/stestr/pull/188 > > which uses the output filter's internal function for counting results to > find unxsuccess tests. It's still not perfect though because if someone > runs with the --no-subunit-trace flag it still doesn't work (because that > call path never gets run) but it's at least a starting point. I've > marked it as WIP for now, but I'm thinking we could merge it as is and > leave the --no-subunit-trace and unxsuccess as a known issues for now, > since xfail and unxsuccess are pretty uncommon in practice. (gabbi is the > only thing I've seen really use it) > > > > > So yeah, I did file a bug but it fell off the radar during those > > dark times. > > >
Just following up here, after digging some more and getting a detailed bug filed by electrofelix [1] I was able to throw together a different patch that should solve this in a better way: https://github.com/mtreinish/stestr/pull/190 Once that lands I can push a bugfix release to get it out there so people can actually use the fix. -Matt Treinish
signature.asc
Description: PGP signature
__________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev