[sage-devel] Proposal - Add notes of "verification" as comments to doctests
It concerns me that in many cases the "doctests" in Sage are not really testing the validity of the implementation in Sage, but testing the reproducibility of that implementation. Whether the "Expected" result is actually correct is not always tested. I propose that when a doctest is written, there should be a comment in the code, substantiating why the "Expected" result is correct. That might take the form of at least all of the following, with hopefully not too many of #7. 1) It is very obviously correct - cases like exp(1.0). 2) As shown in a journal or book, with a citation to the journal/book added as a comment in the source code. 3) Personally been verified by pen & paper. If practical, state roughly the steps taken (integration by parts etc). 4) Computed by other open-source software, that is not part of Sage. Cut/paste the input & output from that software. 5) As computed by Wolfram|Alpha - based on Mathematica. This is nice, in that anyone can check it easily, without a license for Mathematica license. Add the URL to the test in the doctest. 6) Computed by other closed-source Software, which is developed independently of Sage - e.g. MATLAB, Maple, Mathematica, Macsyma. Cut/paste the input & output from that software 7) Computed in Sage in a way which would use primarily different bits of Sage. Add the code used. 8) Computed by Sage, but the exact result is largely irrelevant. E.g. if the size of a bounding box of a plot was measured. 9) This is the only code to compute this, so there's no independent verification. State what software you have checked to determine if there is another way of doing this. (Doing this would enable a reviewer who might have access to other software to try to verify the result ) I'd propose that the above, or some modification of it, is added to the Sage Developers Guide, so people state how the results are verified. I believe this would encourage people to write better doctests. It will also be a useful education tool for younger developers, who might not appreciate the benefits of better verification. If a standard was introduced, for example one added the text "VerificationMethod#3" if #3 was used, then it would be possible to grep for cases of #9, and hopefully someone try to remove them over time. Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Proposal - Add notes of "verification" as comments to doctests
On 12 March 2011 08:56, David Kirkby wrote: > I propose that when a doctest is written, there should be a comment in > the code, substantiating why the "Expected" result is correct. That > might take the form of at least all of the following, with hopefully > not too many of #7. Oops, I mean not too many of #9. Number 7 is not too bad, though far from optimal. I'd base that list in rough order of merit. > Dave > Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Proposal - Add notes of "verification" as comments to doctests
On 2011-03-12 09:56, David Kirkby wrote: > I propose that when a doctest is written, there should be a comment in > the code, substantiating why the "Expected" result is correct. That > might take the form of at least all of the following, with hopefully > not too many of #7. I agree that your idea makes sense but I'm afraid that adding all this stuff will be a lot of effort and also greatly reduce readability of the source code. So if you want to do this, do it in an "out-of-band" way, i.e. not in the doctests themselves. Jeroen. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: FLINT 2.1 released
Cool, thanks! -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Proposal - Add notes of "verification" as comments to doctests
On 03/12/11 09:56 AM, Jeroen Demeyer wrote: On 2011-03-12 09:56, David Kirkby wrote: I propose that when a doctest is written, there should be a comment in the code, substantiating why the "Expected" result is correct. That might take the form of at least all of the following, with hopefully not too many of #7. I agree that your idea makes sense but I'm afraid that adding all this stuff will be a lot of effort I can't see why it should add much effort, as really people should be making sure their doctests are actually test the implementation of their software, and *not* just the reproducibility of the result. There have been several instances of where doctests have been found to be wrong, and I suspect there are others, but we simply don't know about them. and also greatly reduce readability of the source code. Point taken. So if you want to do this, do it in an "out-of-band" way, i.e. not in the doctests themselves. So how about stating such information must be put on the trac ticket before it can be given a positive review? The only thing I'd suggest needs adding to the source code would be the type of test, and a ticket reference number where the justification for the test is verified. Something like: "VerificationMethod#3 - see trac #23455" or whatever the case may be. I listed 9 categories for tests - I don't know if there's any I missed. Jeroen. Dave -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: evaluation of polynomials in several variables
I agree that thats the only valid question here: is f(x=2,y=3) substitution or evaluation? The call syntax suggests evaluation, but keywords allow you to only substitute some variables. I'm tempted to say that since its not obvious we shouldn't have the function at all, that is, don't allow keyword arguments in _call_. -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Proposal - Add notes of "verification" as comments to doctests
On Saturday, March 12, 2011 1:49:40 PM UTC, Dr. David Kirkby wrote: > > There have been several instances of where doctests have been found to be > wrong, > There have been several instances where stated results in published papers have been found to be wrong :-) Just because you quote some paper or claim that its obvious to experts in the field doesn't guarantee that its correct. There is also the issue that some equation's validity might depend on sign conventions, choices of branch points, etc. While I of course agree that doctests should be as correct as possible, I don't think that some bureaucratic act where you demand that people sign of on doctests is going to help much. If anything, the referee should use his personal judgement to determine if a doctest is good or not. In the long run I feel that it is more productive to have code released to expose it to a larger audience so that they can use it with examples they are familiar. Release early, and release often :-) -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Proposal - Add notes of "verification" as comments to doctests
On 03/12/11 02:23 PM, Volker Braun wrote: On Saturday, March 12, 2011 1:49:40 PM UTC, Dr. David Kirkby wrote: There have been several instances of where doctests have been found to be wrong, There have been several instances where stated results in published papers have been found to be wrong :-) True. Wolfram Research claims to have found many, numerous one from one particualr book in one of their library files. Just because you quote some paper or claim that its obvious to experts in the field doesn't guarantee that its correct. True, though if a piece of software, and a published paper agree, then it is much more likely to be true. You can never be 100% sure. There is also the issue that some equation's validity might depend on sign conventions That should be possible to substantiate though. choices of branch points, etc. Agreed, though it should be possible to state why the expected result is believe to be true. IMHO, just because Sage gives a result, does not make it true. While I of course agree that doctests should be as correct as possible, I don't think that some bureaucratic act where you demand that people sign of on doctests is going to help much. I think it would force people to think more clearly about the tests. If anything, the referee should use his personal judgement to determine if a doctest is good or not. From what I can see of tickets, theres a lot of cases where this just does appear to be happening. In the long run I feel that it is more productive to have code released to expose it to a larger audience so that they can use it with examples they are familiar. A lot of the stuff in Sage is very specialised, and I would not assume that people will find bugs too easily. > Release early, and release often :-) That very much has its downsides. If Sage was a word processor, drawing package, or any other of many other sorts of software, bugs would tend to be found, and if discovered would not cause a major issue. In contrast, bugs in software like Sage are much more difficult to find, and are going to be more troublesome to people. Hence I feel the need to try to push the quality up, rather than the quantity of code released. To me at least, people being able to substantiate the results of their doctests is not asking too much. -- A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? Dave -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Proposal - Add notes of "verification" as comments to doctests
On Mar 12, 6:23 am, Volker Braun wrote:
> In the long run I feel that it is more productive to have code released to
> expose it to a larger audience so that they can use it with examples they
> are familiar. Release early, and release often :-)
+1
I wrote the doctest below last night for a commutator() method for
permutation groups. It will give the user confidence that the
commutator, the quotient, and is_abelian are all correct, but is not a
proof of correctness. And if anybody breaks those methods, there is a
good chance this test will fail. If you know group theory, you know
the referenced theorem. If you do not know group theory, it is easy
enough to find in almost any text. I'd rather not provide a reference
when the result is ubiquitous.
So I agree with the sentiment, and I think we can write careful
doctests which achieve that aim, along with informative comments.
Referees should expect, request and/or contribute such tests. But I
do not feel consulting a numbered list, always digging up references,
or figuring out the equivalent commands in another language will add
much value, and will discourage contributions. As Volker said, we
should encourage use, and *welcome* reports of errors which can be
fixed promptly (which I think we do well currently).
Rob
{{{
The quotient of a group by its commutator is always abelian. ::
sage: G = DihedralGroup(20)
sage: C = G.commutator()
sage: Q = G.quotient(C)
sage: Q.is_abelian()
True
}}}
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
[sage-devel] News from the broken ARM
Hi, the title is a little pun : the ARM support isn't really broken since it was never there anyway in the first place. I took sage-4.6.2's sources and tried to build it on my ARM netbook ; it tooks two days as usual, and three spkg are still problematic (by alphabetical order) : - atlas (http://trac.sagemath.org/sage_trac/ticket/10808) - singular (http://trac.sagemath.org/sage_trac/ticket/10903) - tachyon (http://trac.sagemath.org/sage_trac/ticket/10820) Once those will be fixed, sage will compile successfully on ARM. The next step will of course be making the tests pass... but that will be another story (http://trac.sagemath.org/sage_trac/ticket/10285 has some details, but I think I'll try to close that report and open different ones on more specific issues -- or discuss them on this list directly). Help, comments and ideas are welcome. Snark on #sage-devel -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
Re: [sage-devel] Proposal - Add notes of "verification" as comments to doctests
As I've mentioned on a related thread, I think that increasing the quality of doctests is a valuable goal, but that we shouldn't add more impediments to getting code accepted into Sage. In particular, I'm strongly against requiring such justifications for each doctest before code can get a positive review, whether or not that justification is done on the ticket or in the source. And personally, I'd be quite happy if there are more doctests that fall under your category #9, where Sage is the only tool that can currently compute something of interest. David On Sat, Mar 12, 2011 at 03:56, David Kirkby wrote: > It concerns me that in many cases the "doctests" in Sage are not > really testing the validity of the implementation in Sage, but testing > the reproducibility of that implementation. Whether the "Expected" > result is actually correct is not always tested. > > I propose that when a doctest is written, there should be a comment in > the code, substantiating why the "Expected" result is correct. That > might take the form of at least all of the following, with hopefully > not too many of #7. > > 1) It is very obviously correct - cases like exp(1.0). > > 2) As shown in a journal or book, with a citation to the journal/book > added as a comment in the source code. > > 3) Personally been verified by pen & paper. If practical, state > roughly the steps taken (integration by parts etc). > > 4) Computed by other open-source software, that is not part of Sage. > Cut/paste the input & output from that software. > > 5) As computed by Wolfram|Alpha - based on Mathematica. This is nice, > in that anyone can check it easily, without a license for Mathematica > license. Add the URL to the test in the doctest. > > 6) Computed by other closed-source Software, which is developed > independently of Sage - e.g. MATLAB, Maple, Mathematica, Macsyma. > Cut/paste the input & output from that software > > 7) Computed in Sage in a way which would use primarily different bits > of Sage. Add the code used. > > 8) Computed by Sage, but the exact result is largely irrelevant. E.g. > if the size of a bounding box of a plot was measured. > > 9) This is the only code to compute this, so there's no independent > verification. State what software you have checked to determine if > there is another way of doing this. (Doing this would enable a > reviewer who might have access to other software to try to verify the > result ) > > > I'd propose that the above, or some modification of it, is added to > the Sage Developers Guide, so people state how the results are > verified. I believe this would encourage people to write better > doctests. It will also be a useful education tool for younger > developers, who might not appreciate the benefits of better > verification. > > If a standard was introduced, for example one added the text > "VerificationMethod#3" if #3 was used, then it would be possible to > grep for cases of #9, and hopefully someone try to remove them over > time. > > Dave > > -- > To post to this group, send an email to sage-devel@googlegroups.com > To unsubscribe from this group, send an email to > sage-devel+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/sage-devel > URL: http://www.sagemath.org > -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: Proposal - Add notes of "verification" as comments to doctests
On Mar 12, 7:23 am, Volker Braun wrote: > On Saturday, March 12, 2011 1:49:40 PM UTC, Dr. David Kirkby wrote: > > > There have been several instances of where doctests have been found to be > > wrong, > > There have been several instances where stated results in published papers > have been found to be wrong :-) > > Just because you quote some paper or claim that its obvious to experts in > the field doesn't guarantee that its correct. There is also the issue that > some equation's validity might depend on sign conventions, choices of branch > points, etc. While I of course agree that doctests should be as correct as > possible, I don't think that some bureaucratic act where you demand that > people sign of on doctests is going to help much. If anything, the referee > should use his personal judgement to determine if a doctest is good or not. > In the long run I feel that it is more productive to have code released to > expose it to a larger audience so that they can use it with examples they > are familiar. Release early, and release often :-) In general, I tend to agree with David's points on improving the quality of Sage and testing more thoroughly, e.g. I like that the transition alpha-rc-release has slowed down a bit recently and does not happen in a single day anymore. But in this case I completely agree with Volker's comment. The author of the patch and the referee share the responsibility for sensible doctests and their results. If someone consistently neglects this responsibility and ignores requests to take it more serious, well, maybe there should be some blacklist of authors/referees... But as far as I know it was not yet necessary. Requiring some formal justification for every single doctest puts too much burden on the author and there are many cases where it would be completely unnecessary (e.g. _add_ or _repr_ methods). Trying to require it "only when necessary" already fits into current model, I think. I do ask as a referee for references/explanations/documentation improvements when I am not comfortable with a presented patch and I am sure that many people do the same, that's the point of refereeing. I saw some tickets where authors give up, e.g. this one is hanging in my list of "open tickets I participated in" for a year: http://trac.sagemath.org/sage_trac/ticket/8616 Well - that's the point of refereeing. Requiring description of parameters and at least one doctest for absolutely every method is valuable - it shows that the method at least "more or less works" and helps to isolate problem places when something gets broken. Formalizing something beyond this does not seem like a very good idea to me... Thank you, Andrey -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
[sage-devel] Re: News from the broken ARM
On Mar 12, 11:36 am, Julien PUYDT wrote: > Help, comments and ideas are welcome. Are the changes at "Rewrite ATLAS spkg-install" http://trac.sagemath.org/sage_trac/ticket/10226 helpful or relevant with the ATLAS install/build? Rob -- To post to this group, send an email to sage-devel@googlegroups.com To unsubscribe from this group, send an email to sage-devel+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/sage-devel URL: http://www.sagemath.org
