On 06/11/10 12:38 AM, Simon King wrote:
Hi David,
On 11 Jun., 00:32, "Dr. David Kirkby"<david.kir...@onetel.net> wrote:
At least from my very little understanding of this, Having 89% coverage would be
better than 90% coverage, if those 89% were well targeted.
It is not clear to me why one module should be considered being more
important than another. I would not support if you suggest targeting
the effort by the *topic* of the modules being doc tested. Arguing
like "many people do calculus and graphics, so, concentrate on this"
will only lead to a meta-discussion.
I would not say there is no logic to that, but it was not what I was suggesting.
Or do you just say that adding a single doc test to a module with 0%
coverage will have a better impact than adding a single test for a
module with 70% coverage? This might indeed be a good way to find a
starting point.
That was exactly what I meant. In the case of the interface to tachyon, it would
appear there is absolutely no testing of this whatsoever
# interfaces/tachyon.py: 0% (0 of 4)
so adding just one test would be useful. It would at least show the interface is
not completely broken.
In contrast, getting graphs/generic_graph.py up to 100%, by adding one test when
there are already 200, would seem less useful to me.
# graphs/generic_graph.py: 99% (200 of 201)
So I would suggest that doctests are targeted, rather than randomly chosen. (The
obvious way to get the coverage up quickly is to write simple tests.)
Anyway, I am +1 to trying and getting a 90% overall doc test coverage;
it is a valuable aim.
Yes, but IMHO, it is a bit of a simplistic metric.
I personally do not feel Sage is sufficiently tested, and I doubt my opinion
will change if the doctest coverage is increased to 100%.
IMO it is *always* worth it to write doc tests since it is very likely
to uncover flaws (in particular if the person who writes the test is
not the same as the one who wrote the code).
Really, that should be the case.
http://www.sqlite.org/testing.html
is worth a read. Particularly section 7.1 about how there are various ways of
defining test coverage. (I stuck section 7.1 below my name). What is clear is
that the method of calculating coverage in Sage is even weaker than what that
sqlite document consider to be a weak method (statement coverage). We don't even
ensure that every statement of code gets executed at least once.
Dave
Section 7.1 from http://www.sqlite.org/testing.html
=============================================================
There are many ways to measure test coverage. The most popular metric is
"statement coverage". When you hear someone say that their program as "XX% test
coverage" without further explanation, they usually mean statement coverage.
Statement coverage measures what percentage of lines of code are executed at
least once by the test suite.
Branch coverage is more rigorous than statement coverage. Branch coverage
measure the number of machine-code branch instructions that are evaluated at
least once on both directions.
To illustrate the difference between statement coverage and branch coverage,
consider the following hypothetical line of C code:
if( a>b && c!=25 ){ d++; }
Such a line of C code might generate a dozen separate machine code instructions.
If any one of those instructions is ever evaluated, then we say that the
statement has been tested. So, for example, it might be the case that the
conditional expression is always false and the "d" variable is never
incremented. Even so, statement coverage counts this line of code as having been
tested.
Branch coverage is more strict. With branch coverage, each test and each
subblock within the statement is considered separately. In order to achieve 100%
branch coverage in the example above, there must be at least three test cases:
* a<=b
* a>b && c==25
* a>b && c!=25
Any one of the above test cases would provide 100% statement coverage but all
three are required for 100% branch coverage. Generally speaking, 100% branch
coverage implies 100% statement coverage, but the converse is not true. To
reemphasize, the TH3 test harness for SQLite provides the stronger form of test
coverage - 100% branch test coverage.
====================================================
My 0.02€
Simon
Dave
--
To post to this group, send an email to sage-devel@googlegroups.com
To unsubscribe from this group, send an email to
sage-devel+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/sage-devel
URL: http://www.sagemath.org
