Am 09.01.2012 15:35, schrieb Roy Smith:
The classic unittest philosophy says that tests should be independent of each other, which means they should be able to be run in arbitrary order. Some people advocate that the test framework should intentionally randomize the order, to flush out inter-test dependencies that the author didn't realize existed (or intend).
While I agree with the idea, I'd like to add that independence is an illusion. You already have possible dependencies if you run tests in the same process/OS-installation/computer/parallel universe. If you now happen to influence one test with another and the next run randomizes the tests differently, you will never see the fault again. Without this reproducability, you don't gain anything but the bad stomach feeling that something is wrong.
Test independence is a good thing. Many people don't understand this when writing tests, and inadvertently write tests that depend on each other. I've worked with those systems. They're a bear to debug. You've got some test suite that runs for 15 minutes and dies at case 37 of 43. If you try to run case 37 by itself, it won't run, and you can't figure out what state cases 1-36 were supposed to leave the system in to make 37 work. You could sink days or weeks into debugging this kind of crap. BTDT.
I'm sorry to hear that, debugging other peoples' feces is never a task to wish for. That said, there are two kinds of dependencies and in at least this discussion there hasn't been any mentioning of the differences yet, but those differences are important.
Your unfortunate case is where test X creates persistent state that must be present in order for test X+1 to produce meaningful results. This kind of dependency obviously blows, as it means you can't debug test X+1 separately. I'd call this operational dependency.
This kind of dependency is IMHO a bug in the tests themselves. The unit testing framework could help you find those bugs by allowing random order of execution for the test cases.
There is another dependency and that I'd call a logical dependency. This occurs when e.g. test X tests for an API presence and test Y tests the API behaviour. In other words, Y has no chance to succeed if X already failed. Unfortunately, there is no way to express this relation, there is no "@unittest.depends(test_X)" to decorate test_Y with (Not yet!). Each test would be the root of a tree of tests that it depends on. If a dependency fails already, you can either skip the tree or at least mark the following test failures as implicit failures, so that you can easily distinguish them from the root cause.
This kind of dependency is quite normal although not directly supported by the unittest module. As a workaround, being able to define an order allows you to move the dependencies further to the top, so they are tested first.
To sum it up, in order to catch operational dependencies, you need a random order while in order to clearly express logical dependencies in the output, you want a fixed order. Neither is the one true way, though my gut feeling is that the fixed order is overall more useful.
As long as they understand the consequences of their actions, don't try to preach unittest religion to them. They're in the best position to know if what they're trying to do is the best thing for their particular situation.
Amen! Uli -- http://mail.python.org/mailman/listinfo/python-list
