Steve D'Aprano <steve+pyt...@pearwood.info>: > On Wed, 9 Nov 2016 10:01 am, teppo.p...@gmail.com wrote: >> Generally, with testing, it would be optimal to test outputs of the >> system for given inputs without caring how things are implemented. > > I disagree with that statement.
I, OTOH, agree with it. > But in fact, that test suite is completely insufficient. The > implementation of list.sort() uses two different sort algorithms: > insertion sort for short lists, and Timsort for long lists. My > black-box test suite utterly fails to test Timsort. Independent algorithms can be packaged as independent software components and subjected to separate black-box tests -- through their advertised APIs. > Black-box testing is better than nothing, but white-box testing is > much more effective. I've had to deal with a lot of white-box test code. In practice, * It relies on internal reporting by the implementation instead of real, observed behavior. * It hasn't stayed current with the implementation. * It has convoluted the code with scary conditional compilation and other tricks, thus lowering the quality of the implementation. > Certainly. And if you test do_something_important against EVERY > possible combination of inputs, then you don't need to care about the > implementation, since you've tested every single possible case. The space of every imaginable situation is virtually infinite; testing will only scratch the surface no matter how much effort you put into it. I'm thinking of recent bugs that have sneaked into a product. The question was why our tests didn't catch the bugs. The answer was that those particular sequences weren't included in the thousands of test cases that we run regularly. >> class Example: >> def __init__(self, queue=None): >> self._queue = queue or Queue() > > That's buggy. If I pass a queue which is falsey, you replace it with > your own default queue instead of the one I gave you. That's wrong. And your two thousand white-box test cases might miss the bug as well. > The lessen here is: when you want to test for None, TEST FOR NONE. > Don't use "or" when you mean "if obj is None". In the real world, that's among the most benign of bugs that riddle software (and software tests). Programming is too hard for mortals. Junior software developers are full of ambitious ideas -- sky's the limit. Experienced software developers are full of awe if anything actually works. Marko -- https://mail.python.org/mailman/listinfo/python-list
