I'm glad to hear that the community has developers who are bug hunting, too!
Regarding tests for invalid input: that also helps to discover bugs (imagine that the invalid input was an output of another buggy routine) However, sometimes this is costly (e.g. test if an ideal is radical) and should be turned on only if a certain assert level is active. Singular recently introduced assumeLevel (individually configurable for each library). Q: Is there a similar concept in sage for this? Test with random input and/or against other implementations/CAS: Q: Is this functionality organized (alongside doctesting) in some framework/package or would that be useful? Many contributors to Sage have only had experience of writing code for > themselves > > Then we have to transfer our knowledge or we are hopelessly lost ;-) In a recent project I'm working on we observed some occasional bugs in routines we used or wrote. After digging deeper into that we experienced that almost each routine in our project was buggy and almost each second routine we did rely on (I won't tell which CAS we are using but you can guess it) Jack Am Dienstag, 13. Mai 2014 14:31:09 UTC+2 schrieb John Cremona: > > David has indeed made some very hepful contributions to this -- I remember > the cremona_letter_code business well (the function just assumed that the > input was valid), and there was another case to do with validating the > input to mwrank, an external C++ program included in Sage, which took a bit > of work but ended up much better than it had been. > > For quite a lot of Sage's number theory functionality many of us do > regularly test against Magma, and this has in the past revealed bugs on > both sides. To a lesser extent tests are possible against the pari > library, but of course that tells you little when the only method > implemented in Sage involvers a call to the pari library. > > Many contributors to Sage have only had experience of writing code for > them,selves before, and there is a lot to learn about in the different > world of open source sofware where you cannot assume that the person > running your code has a PhD in the area.... > > John > > > On 13 May 2014 13:22, Dr. David Kirkby <drki...@gmail.com <javascript:>>wrote: > >> On 13 May 2014 10:48, <kro...@uni-math.gwdg.de <javascript:>> wrote: >> > Hello, >> > >> > are here developers in the group who actively test (the routines they >> use) >> > e.g. by throwing random input at them? >> > (like in ticket http://trac.sagemath.org/ticket/1136 ) >> > >> > If yes, which other techniques do you use to detect bugs? >> > >> > Thanks, >> >> I have done a couple of related things, as I have sometimes had >> concerns about the Sage testing - I think more so in the past than >> now. Some of the code was of very dubious quality in my opinion. I >> recall an email from one developer telling me he did not see much >> point in testing much, as he would just fix bugs when reported !!! >> >> I have several times advocated the use of Sage releases where no new >> functionality is added - only bugs were fixed. There has been little >> appetite among developers for this. Anyway, a couple of things I have >> done. >> >> #1 >> >> I developed some code to send random data to commands and uncovered >> some which don't behave as expected. >> >> cremona_letter_code >> >> for example should only work for non-negative integers, but I found that >> >> sage: cremona_letter_code(23.23) # Invalid input, gives output. >> 'x' >> sage: cremona_letter_code(-3) # Invalid input, hangs using lots of CPU >> time. >> >> That was >> >> http://trac.sagemath.org/ticket/10105 >> >> I found a number of examples of functions which could mis-behave if >> fed invalid input. >> >> #2 >> Some time ago I suggested that we could compare Sage output with >> output from WolframAlpha to use as a note in doc tests, giving the >> WolframAlpha URL that gives the same result. I felt WolframAlpha was >> preferable to Mathematica, since everyone had access to it. Alex >> Ghitza in particular thought we might be breaking the terms of the >> license, and on reading it, he could well have had a valid point. So I >> asked Wolfram Reseach to clarify the license, and if we were permitted >> to use WolrramAlpha to make comparisons with Sage. They basically said >> yes, although we can't make automated querying of their servers. That >> is hardly surprising, as calling their servers thousands or millions >> of times would load their servers. You can read their response here. >> >> https://groups.google.com/forum/#!topic/sage-devel/tijz3svNNiE >> >> I don't know if anyone has actually done this, making notes in doc >> tests of the WolframAlpha / Mathematica output. But it would give some >> confidence that results are the same. >> >> I do recall one function, related to prime testing, where Mathematica >> was a lot faster than Sage. On closer inspection, it was revealed that >> the Mathematica algorithm had been proved write for N < $some_value, >> but there was no proof it worked for all values, whereas the Sage test >> was more thorough. I reported this to Wolfram Research, who said they >> would amend the documentation to reflect this. I can't recall whether >> they did. >> >> I think there is some value to comparing Sage and Mathematica output >> for tests, and perhaps also noting the CPU time used. If Sage is >> significantly slower, it might indicate there are better algorithms in >> existence, and those could be investigated. >> >> >> Dr. David Kirkby G8WRB >> http://www.vnacalibration.co.uk/ >> Economical & accurate VNA calibration kits. >> Coefficients available for HP, Agilent, Anritsu, Rohde & Schwarz and >> VNWA network analyzers. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "sage-devel" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to sage-devel+...@googlegroups.com <javascript:>. >> To post to this group, send email to sage-...@googlegroups.com<javascript:> >> . >> Visit this group at http://groups.google.com/group/sage-devel. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "sage-devel" group. To unsubscribe from this group and stop receiving emails from it, send an email to sage-devel+unsubscr...@googlegroups.com. To post to this group, send email to sage-devel@googlegroups.com. Visit this group at http://groups.google.com/group/sage-devel. For more options, visit https://groups.google.com/d/optout.
