It concerns me that in many cases the "doctests" in Sage are not really testing the validity of the implementation in Sage, but testing the reproducibility of that implementation. Whether the "Expected" result is actually correct is not always tested.
I propose that when a doctest is written, there should be a comment in the code, substantiating why the "Expected" result is correct. That might take the form of at least all of the following, with hopefully not too many of #7. 1) It is very obviously correct - cases like exp(1.0). 2) As shown in a journal or book, with a citation to the journal/book added as a comment in the source code. 3) Personally been verified by pen & paper. If practical, state roughly the steps taken (integration by parts etc). 4) Computed by other open-source software, that is not part of Sage. Cut/paste the input & output from that software. 5) As computed by Wolfram|Alpha - based on Mathematica. This is nice, in that anyone can check it easily, without a license for Mathematica license. Add the URL to the test in the doctest. 6) Computed by other closed-source Software, which is developed independently of Sage - e.g. MATLAB, Maple, Mathematica, Macsyma. Cut/paste the input & output from that software 7) Computed in Sage in a way which would use primarily different bits of Sage. Add the code used. 8) Computed by Sage, but the exact result is largely irrelevant. E.g. if the size of a bounding box of a plot was measured. 9) This is the only code to compute this, so there's no independent verification. State what software you have checked to determine if there is another way of doing this. (Doing this would enable a reviewer who might have access to other software to try to verify the result ) I'd propose that the above, or some modification of it, is added to the Sage Developers Guide, so people state how the results are verified. I believe this would encourage people to write better doctests. It will also be a useful education tool for younger developers, who might not appreciate the benefits of better verification. If a standard was introduced, for example one added the text "VerificationMethod#3" if #3 was used, then it would be possible to grep for cases of #9, and hopefully someone try to remove them over time. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
