On 15 Nov, 01:02, "Brendan Miller" <[EMAIL PROTECTED]> wrote: > On Thu, Nov 13, 2008 at 3:54 AM, James Harris > > > > <[EMAIL PROTECTED]> wrote: > > On 11 Nov, 22:59, Brendan Miller <[EMAIL PROTECTED]> wrote: > >> What would heavy python unit testers say is the best framework? > > >> I've seen a few mentions that maybe the built in unittest framework > >> isn't that great. I've heard a couple of good things about py.test and > >> nose. Are there other options? Is there any kind of concensus about > >> the best, or at least how they stack up to each other? > > > You don't mention what you want from testing so it's hard to say which > > is "best" as it depends on one's point of view. > > > For example, I had a requirement to test more than just Python > > programs. I wanted to test code written in any language. None of the > > frameworks I found supported this so I wrote my own tester. It > > interacts with programs via file streams - principally stdin, stdout > > and stderr though others can be added as needed. > > I was speaking to unit tests (tests of individual classes and > functions in isolation of the rest of the program) and a test driven > development approach. I find those to be much more useful, and good > for pinpointing bugs and guiding development. In contrast regression > tests tend to be slow, and are the programmatic equivalent of kicking > the tires and seeing if they fall off.
I agree with your points but would add 1. We should write smaller modules than we generally do. It promotes clear thinking and encourages the production of reusable code. In Python it is easy to add a __main__ section to interface to the classes or functions defined in the code. Such code can then be tested as mentioned. The functions or classes can still be imported into and used in other code as needed. 2. The example given wasn't intended to be the only way forward but to highlight that what makes a test package good depends on your requirements (which I don't think you mentioned). In my case the desire to test arbitrary languages was a killer blow to various Python- only frameworks which I wasn't too impressed with anyway. (At least I saw none that looked worth investing the time to learn.) Similarly an option for you is to write your own tester. It may be a better fit than someone else's package and not too hard to do. On the other hand a pre-written package may be better - depending on what you are looking for. > I've also seen regression tests lead to a "sweep the bugs under the > rug mentality" where developers will code to prevent errors from > crashing the regression test, e.g. by catching and swallowing all > exceptions without fixing the underlying problem. It's easy to fool > regression tests since what it does works at such a high level that > most aspects of program correctness can't be directly checked. This is > very frustrating to me because it actually leads to lower code > quality. If you have dodgy developers then much is wrong, not just testing. If the team works well, IMHO, production of test cases should be a welcome part of the development process. When is it better to find bugs: before or after shipping to a client? Test cases can be put together in various ways. For example, if one has the code and chooses to work this way white-box testing can lead to generation of test cases. On the other hand if an organisation prefers it can produce test cases from the spec - or they can do both. It's their choice but in any case the developers and testers should work together. > > One nice by-product is that test code does not bloat-out the original > > source which remains unchanged. > > That's the main reason most people don't write unit tests. It forces > them to properly decouple their code so that parts can be used > independently of one another. Adding such things to messy ball of mud > code after the fact is an enourmous pain in the butt. Thankfully, > since python is duck typed and doesn't require lots of boilerplate > writing interfaces and abstract factories (since the class object > itself acts as an abstract factory), writing properly decoupled code > in the first place doesn't look nearly as hard as in C++ or Java. You could still import one Python module into another to run test cases such as ----- import test_support import <program_to_be_tested> <tests of program_to_be_tested using test_support> ----- where the tests can interact directly with the components of the code under test. The test support code can handle things such as collating results and reporting as well as providing code to directly support certain tests, handle exceptions etc. I wouldn't be surprised if good testing code were in many cases to be significantly longer than the code being tested. Keeping the tests in a separate file keeps the code being tested shorter which can only improve clarity. It also allows either the developer or someone else to write the tests. Putting code and testing code in the same file almost requires one person to write both parts. A final thought: if you have dodgy developers who engineer their code to avoid tests you're not going to have much success in getting them to write good test code (though, hopefully, the mental process of writing test cases should help them improve the code they write). At the end of the day all I'm saying is that the best test framework for you really depends on your requirements, rather than what others think is "best" or not. -- James -- http://mail.python.org/mailman/listinfo/python-list
