Jay and I have been having an interesting discussion about how to deal with bugs that mean that unit tests _should_ fail. So, if I find a bug, I should write a failing unit test first, and fix it (in one merge). However, if I can't fix it, I can't get a failing unit test merged into the trunk (because it fails). It may be that I can't get what I'm actually working on merged with good unit tests until this 'tangential' bug is fixed.
(The discussion is here: https://code.launchpad.net/~justin-fathomdb/nova/bug724623/+merge/51227) I suggested that we introduce a "known_bugs" collection. It would have a set of values to indicate bugs that are known but not yet fixed. Ideally these would be linked to bug reports (we could mandate this). When a developer wants to write a test or behavior to work around a particular bug, they can control it based on testing this collection ("if 'bug12345' in known_bugs:") When someone is ready to fix the bug, they remove the bug from the collection, the unit tests then fail, they fix the code and commit with the known_bugs item removed. This would let people that find bugs but can't or don't want to fix them still contribute unit tests. This could be a QA person that can write tests but not necessarily code the fix. This could be a developer who simply isn't familiar with the particular system. Or it could be where the fix needs to go through the OpenStack discussion process. Or it could simply be a train-of-thought / 'flow' issue. Take, for example, my favorite OpenStack API authentication issue. To get a passing unit test with OpenStack authentication, my best bet is to set all three values (username, api_key, api_secret) to the same value. This, however, is a truly terrible test case. Having "known_bugs" marks my unit test as being suboptimal; it lets me provide better code in the same place (controlled by the known_bugs setting); and when the bug fixer comes to fix it they easily get a failing unit test that they can use for TDD. Jay (correctly) points out that this is complicated; can cause problems down the line when the bug is fixed in an unexpected way; that known_bugs should always be empty; and that the right thing to do is to fix the bug or get it fixed. I agree, but I don't think that getting the bug fixed before proceeding is realistic in a project with as many stakeholders as OpenStack has. Can we resolve the dilemma? How should we proceed when we find a bug but we're working on something different? Justin
_______________________________________________ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
