Big +1. There's the sage/tests folder which seems to be a place where certain developers who really couldn't help themselves put some additional tests. But it doesn't go near as far as what you're proposing.
However, there's obvious issues wrt. ensuring that such tests gets written once in a while, without forcing further bureaucracy on the development process. Currently, the Reviewer's Checklist specifies that you should manually try out the code on various examples. Perhaps the reviewer could be in charge of writing the first germs of such non-doctest testing of the ticket. It wouldn't have to be fancy, just sort of saving for the future the work that the reviewer is doing anyway. Whenever a bug shows up in a module, that's an obvious time to improve the non-doctest testing, when pinning down the bug and testing that it was fixed. > a) performance tests where we would test against the previous release to make > sure that changes being introduced do not affect the performance in a > negative way Vincent Delecroix proposed something like this for GSoC 2016, but no students came forward: https://wiki.sagemath.org/GSoC/2016 > b) randomized test There's the hypothesis library which is a Python-take on Haskell's brilliant QuickCheck. IMO, this kind of declarative property testing on random instances is extremely well suited for testing mathematical software, but there's some challenges in providing a good set of "random generators" for the many types of objects and algorithms (e.g. testing topological sort can be done on huge graphs, but testing travelling salesman algorithm should be done on small graphs. How and where do you "define" small/big?). > c) test mathematical correctness more extensively by storing results of a > larger set of results that have been verified to be mathematically correct in > some way This is a very nice idea. > d) test unpickling of objects which seems to break rather often and is not > covered at all by any of the doctests +1 for test-framework(s) for any part of Sage which is currently untested. Best, Johan stephan.j.eh...@gmail.com writes: > Hi everyone, > > I would like to discuss adding at least one more testing method to the sage > development process than just doctests (e.g. nose). > > Doctests are certainly great but they have obvious limitations. > > Some of them are: > 1) You want to keep the source code clean so doctests should be “short”. But > some test cases require more complicated code or have long output which you > would not like to add to the source code. > 2) You don’t want to have certain tests in the documentation of a function > which would just distract the user. > 3) Some things can’t or cannot easily be tested in doctests. > > Would like to see for instance are: > a) performance tests where we would test against the previous release to make > sure that changes being introduced do not affect the performance in a > negative way > b) randomized test, example: check for a number of randomly generated number > fields that arithmetic operations with randomly generated number field > elements gives the correct results. Randomized tests help to identify issues > that occur with input that no one thought about testing. > c) test mathematical correctness more extensively by storing results of a > larger set of results that have been verified to be mathematically correct in > some way > (These tests in particular could run very long of we want to cover large > ranges and we should not make these tests run by everyone but rather by some > bots in parallel and maybe also not block new releases if untested but they > could run continuously and block a new release if a problem is discovered. > The data to check against would be publicly available and we could advertise > that people install a bot on their machine that runs at scheduled times and > just picks some examples that have not been checked with the current > development version (or not on their particular architecture or OS version or > so.)) > d) test unpickling of objects which seems to break rather often and is not > covered at all by any of the doctests > > Maybe not all of these tests would have to be run every time someone submits > a patch but should be run before a release comes out. > > I know that there are many things to discuss and other people have way way > more experience with sage development than I have but I feel very strongly > about this and really think that this should be a top priority to ensure that > sage works reliably. > > What do you think? > > Stephan -- -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at https://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
