On Nov 3, 5:29 am, Bill Hart <goodwillh...@googlemail.com> wrote: > Hi all,
Hi Bill > Now the MPIR test code is pretty extensive and really ought to have > picked up this bug. We put a lot of time into the test code for that > MPIR release, so this is unfortunate. Bugs are inevitable. Anybody that believes one can write a non-trivial piece of bug-free code is mistaken. NASA can't do it. The aviation industry can't do it. So any thoughts of writing bug-free code are just pointless. It will never happen. The best we can do is to reduce the probability of bugs. > However, the entire Pari test suite and the entire Sage test suite > (with an older version of Pari) passed without picking up this pretty > serious bug in the MPIR division code! > > I think this underscores something I have been saying for a long time. > Sage doesn't test the C libraries it uses well enough. I agree. There tends to be a trust that the upstream developers are good mathematicians and their code is often widely used, so it must be ok. But I'm afraid that in many cases in Sage, the upstream developers skill set is often not in software, but in mathematics. > As a result of > that, it is taking inordinate amounts of developers' time to track > down bugs turned up by Sage doctests when spkg's are updated. In some > cases there is actually woefully inadequate test code in the C library > itself. Agreed. I think however there is now an increased awareness of this, and the situation is improving. I know recently someone proposed adding some code to Sage, and I asked if an audit had been done of the library. It was clear that in that case the developers were clearly quite skilled at writing software. It was well commented, clean etc. I did not check the maths of it, but it passed the first stage in that the developers were clearly vigilant in what they were doing. It had test code etc. > But even when this is not the case, it makes sense for Sage to > do some serious testing before assuming the library is bug free. This > is particularly easy to do in Python, and much harder to do at the > level of the C library itself, by the way. > I have been saying this for a very long time, to many people. *ALL* > mathematical libraries are broken and contain bugs. Of course. Any non-trivial piece of code will contain bugs. > If you don't test > the code you are using, it *is* broken. Even if you do test, if the code is non-trivial, it will contain bugs. There is no such thing as a non-trivial bug-free program. > The right ratio of test code > to code is really pretty close to 50/50. And if you think I don't do > this myself when I write code (even Sage code), well you'd be wrong. Where do you get this 50:50 figure from Bill? Did you can rand(), or is it based on any hard facts? What appears to be to be a rather extreme example, is the SQlite database which is in Sage, where the test code is 647 times bigger than the database http://www.sqlite.org/testing.html If you pick up any decent book on software engineering you will find that the most expensive part of developing commercial software is the maintenance costs. So it does not surprise me one bit that a lot of time is spent in Sage in resolving bug problems - that seems pretty normal in the computer industry. > One solution would be for everyone to test more widely. If you write > code that depends on feature Y of module X and module X doesn't > properly test feature Y, assume it is broken and write doctests for > that code as well as the code you are writing yourself. Unfortunately that will often get impractical, and I think would easily consume more than the 50:50 mix you mention above. > To give an example, Andy Novocin and I have been working on new > polynomial factoring code in FLINT for a couple of years now. Around 6 > months ago we had a long test of some 100,000 or so polynomials > factoring correctly. We also had a long test of some 20 odd very > difficult polynomials factoring correctly. Thus there was no reason at > all to suppose there were *ANY* bugs in the polynomial factoring code > or any of the functions it made use of. That's just rubbish. If this code is non-trivial, then you must expect bugs. > By Sage standards I think this > is an insane level of testing. It's certainly higher than normal in Sage I would agree. There are a wide mix of Sage developers, so who pay very scant attention to code quality, others who do take it seriously. > But I insisted that every function we have written have its own test > code. This has meant 6 months more work (there was something like > 40,000 lines of new code to test). But I cannot tell you how many new > serious bugs (and also performance problems too) that we turned up. > There must be dozens of serious bugs we've fixed, many of which would > have led to incorrect factorisations of whole classes of polynomials. I'm really puzzled you are surprised at this. > The lesson for me was: just because my very extensive 5 or 6 doctests > passed for the very complex new functionality I added, does not mean > there aren't incredibly serious bugs in the underlying modules I used, > nor does it mean that my new code is worth printing out and using as > toilet paper. You do not need 6 months worth to prove this - you can pick up any decent book on software engineering and these issues are discussed. > Detecting bugs in Sage won't make Sage a viable alternative to the > MA*'s (that a whole nuther thread). Agreed. But failing to detect too many bugs will make Sage non-viable for a large number of users. > After all, testing standards in > those other packages are quite possibly much worse. I do believe Wolfram Research take testing quite seriously. Whilst I'm aware of some of which is written at http://reference.wolfram.com/mathematica/tutorial/TestingAndVerification.html is total rubbish, I doubt they actually lie about some of the testing methods they use. They certainly claim to use a lot of techniques which Sage does not. > But testing more > thoroughly will mean less time is spent wasted trying to track down > bugs in an ad hoc manner, and eventually, much more time available for > addressing those issues that are relevant to becoming a viable > alternative. Agreed. But one has to balance the conflicting demands of testing code better, and adding more features. Finding the right balance point is not easy. Different developers each have their own ideas. > Bill. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
