If the "known_bugs" list isn't being well received, how about this:
# TODO(ewanm): Enable once bug #21212 is fixed
if False:
assert(something)
And then put a comment on bug #21212 saying "please also enable the following
unit tests when you fix this bug".
Ewan.
From: Justin Santa Barbara [mailto:jus...@fathomdb.com]
Sent: 28 February 2011 15:50
To: openstack@lists.launchpad.net
Cc: Clay Gerrard; Dan Prince; Tim Simpson; Ewan Mellor
Subject: Re: [Openstack] How to deal with 'tangential' bugs?
I'm really talking about tests that degrade to work around bugs; I agree that
we shouldn't blindly skip whole tests (although when a test is skipped using
@unittest.skip, I think it gives a nice indication that it was skipped vs
passed?)
For example, consider the OpenStack API authentication issue I gave.
To get a passing unit test with OpenStack authentication, my best bet is to set
all three values (username, api_key, api_secret) to the same value. This,
however, is a truly terrible test case. Having "known_bugs" marks my unit test
as being suboptimal; it lets me provide better code in the same place
(controlled by the known_bugs setting); and when the bug fixer comes to fix it
they easily get a failing unit test that they can use for TDD.
In code, you'd have something like this:
auth_name = random_string(8)
auth_key = random_string(8)
auth_secret = random_string(8)
if 'bug12345' in known_bugs:
# Set all 3 values the same, so that we can still auth even though secret &
key aren't what we'd intuitively expect
auth_secret = auth_name
auth_key = auth_name
create_user(auth_name, auth_key, auth_secret)
So I have a weak test case (because it's the best I can do until bug12345 is
fixed). When someone removes 'bug12345' from the known_bug list, the same test
case becomes strong. If someone is ready to fix the OpenStack auth issue, they
need only remove bug12345 from the known_bugs list and the tests cases will
then fail, so they have ready-made TDD.
Another example: My unit test might skip over an assertion, even though the
spec says it should be the case, because there's a filed bug about it. I have
the choice between not testing it at all (not writing a good unit test), or
writing a good unit test with a 'guard' that means that it temporarily degrades
to a weaker unit test.
Another example: I encountered an issue with fake_carrot. Vish fixed it very
quickly, but in the meantime my best bet was to use "real" RabbitMQ. Switching
between fake_carrot and "real" RabbitMQ could be driven by a known_bugs flag.
This is probably more controversial, but because fake_carrot is supposed to
behave the same as carrot, it did mean I could satisfy myself & anyone else
that the bug was (probably) with fake_carrot, because it wasn't behaving like
the system it was supposed to be faking.
I think in reality there are two alternatives to putting this into a really
obvious file like known_bugs.py:
1) Hide a similar flag in the file with the unit test, or comment out the 'good
test code'
or
2) Simply not write a thorough test case.
I think both of these alternatives are worse. I don't love known_bugs.py
either, but I think it's the best we've got. If anyone has a better idea, I'd
love to hear it!
Ewan: I think the idea of skipUnless(datetime.now()...) is brilliant. I'm just
thinking of the havoc that could be caused by sneaking an obfuscated version of
that into the unit test suite, particularly one that only failed if the
username was 'hudson' :)
Justin
On Mon, Feb 28, 2011 at 1:57 PM, Tim Simpson
<tim.simp...@rackspace.com<mailto:tim.simp...@rackspace.com>> wrote:
The Skip plugin for nose offers similar functionality which can be used in
Python 2.6:
http://somethingaboutorange.com/mrl/projects/nose/0.11.1/plugins/skip.html
Using this you can write decorators that raise SkipTest if a certain criteria
isn't met.
________________________________________
From:
openstack-bounces+tim.simpson=rackspace.com<http://rackspace.com>@lists.launchpad.net<http://lists.launchpad.net>
[openstack-bounces+tim.simpson=rackspace.com<http://rackspace.com>@lists.launchpad.net<http://lists.launchpad.net>]
on behalf of Clay Gerrard
[clay.gerr...@rackspace.com<mailto:clay.gerr...@rackspace.com>]
Sent: Monday, February 28, 2011 3:09 PM
To: Dan Prince; Justin Santa Barbara
Cc: openstack@lists.launchpad.net<mailto:openstack@lists.launchpad.net>
Subject: Re: [Openstack] How to deal with 'tangential' bugs?
Unittest2 lets you define a test case that is expected to fail:
http://docs.python.org/library/unittest.html#unittest.expectedFailure
new in 2.7, but it could be possible to backport - or do something similar...
May have issues with nose:
http://code.google.com/p/python-nose/issues/detail?id=325
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to :
openstack@lists.launchpad.net<mailto:openstack@lists.launchpad.net>
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
_______________________________________________
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
