Hi everyone, I would like to discuss adding at least one more testing method to the sage development process than just doctests (e.g. nose).
Doctests are certainly great but they have obvious limitations. Some of them are: 1) You want to keep the source code clean so doctests should be “short”. But some test cases require more complicated code or have long output which you would not like to add to the source code. 2) You don’t want to have certain tests in the documentation of a function which would just distract the user. 3) Some things can’t or cannot easily be tested in doctests. Would like to see for instance are: a) performance tests where we would test against the previous release to make sure that changes being introduced do not affect the performance in a negative way b) randomized test, example: check for a number of randomly generated number fields that arithmetic operations with randomly generated number field elements gives the correct results. Randomized tests help to identify issues that occur with input that no one thought about testing. c) test mathematical correctness more extensively by storing results of a larger set of results that have been verified to be mathematically correct in some way (These tests in particular could run very long of we want to cover large ranges and we should not make these tests run by everyone but rather by some bots in parallel and maybe also not block new releases if untested but they could run continuously and block a new release if a problem is discovered. The data to check against would be publicly available and we could advertise that people install a bot on their machine that runs at scheduled times and just picks some examples that have not been checked with the current development version (or not on their particular architecture or OS version or so.)) d) test unpickling of objects which seems to break rather often and is not covered at all by any of the doctests Maybe not all of these tests would have to be run every time someone submits a patch but should be run before a release comes out. I know that there are many things to discuss and other people have way way more experience with sage development than I have but I feel very strongly about this and really think that this should be a top priority to ensure that sage works reliably. What do you think? Stephan -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
