Tels,
I believe you have misunderstood my intentions. I was not advocating
that any algorithmic tests be non-public. The only tests that should
be private are ones that satisfy one or more of the following
restrictions:
1. require special additional software that’s difficult or
expensive to acquire,
2. require special configuration to run properly,
3. don’t affect the quality of the final software, or
4. take too long to run.
That is, tests like spell-checking, pod-checking, Perl::Critic and
ones that take tens of minutes to run. Naturally, any test whose
results may vary from machine to machine should be public wherever
feasible.
After reading some of the insightful comments posted on my blog, I've
been convinced that the private tests should be included in the CPAN
distribution, but disabled in some way (perhaps via a file extension
other than .t?). That way, resourceful or interested users can run
the tests but average users don't need to.
The example I included at the end of the article stating a failed
test under Windows was only tangentially related and perhaps was a
distraction. I was saying that perhaps it would be useful if authors
could assert things about their personal testing experience that
would be machine readable and useful to others. In that example I
was suggesting that the author could announce that the hypothetical
module was known to fail under Windows and, as a TODO test, was an
implicit request for help from other developers.
Chris
On Nov 15, 2005, at 4:12 AM, Tels wrote:
-----BEGIN PGP SIGNED MESSAGE-----
Moin,
On Monday 14 November 2005 18:21, Chris Dolan wrote:
Hello all,
I've just published an article about public vs. private regression
tests. I've defined private tests as t/*.t files that are for the
author only and don't go in MANIFEST. Naturally, those don't get as
much publicity as tests included in CPAN distributions.
In the article I advocate that some tests should be private. In
particular,
1) those that test non-critical aspects of a module (like
documentation and coding style)
2) those that are too expensive to run often
3) those that require special software or customization
In my conclusion I describe a possible system where authors publish
the results of private tests with their distributions as a trust-
based kwalitee system. That is, authors assert kwalitee rather than
be judged for it.
http://www.chrisdolan.net/talk/index.php/2005/11/14/private-
regression-tests/
Both positive and negative feedback is very welcome!
Private tests will only be run by the author, meaning they will be
only
run on a very small subset of all systems the modules can be used on.
This limits their usefullness quite a bit.
Case ein point: I can test my modules on linux, 32 bit, unthreaded,
under
unicode, and under perl 5.8.x. Thats about it, everything else gets
really really complicated for me to set up and maintain/test.
So, no win32, no mac, now solaris, no irix, no perl 5.6.x, no
iso-something, no EBDIC (or however it is spelled), no threading,
no 64
bit, no SMP system.
As for 1), these things should matter (the "broken window analogy")
and
you would be surprised to know how these tests can pass on your
system,
and still fail on other systesm (forget to include the .pod file in
MANIFEST is the most obvious one).
As for 2), random testing should be employed (Math::BigInt does
this, it
runs 256 or so tests with random number patterns (and thus known
results
like "2 * A - A == A"). The tests are quite fast, but they cover
only a
small subset of potential values. However, since each system and user
runs a new, different random set, you end up with a really huge
testing
number being run. (Yes, this has found some bugs)
And for 3), this might be the only point I can think that private
tests
are usefull (I have a private testset for Graph::Easy that I use from
time to time, it is not public mainly because it fails/hangs/takes
forever and is work-in-progress).
However, I have to actually read your article to find out what your
proposal solves (compared to me just running thetest once in a
while :)
Hope that was usefull :)
Best wishes,
Tels
- --
Signed on Tue Nov 15 11:04:21 2005 with key 0x93B84C15.
Visit my photo gallery at http://bloodgate.com/photos/
PGP key on http://bloodgate.com/tels.asc or per email.
"Now, _you_ behave!"
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iQEVAwUBQ3m0pHcLPEOTuEwVAQF6lQf8CtubDMQjLdCBcEGNczxZj2Y1kTVhOU7Z
bvweeJe4RWFfmd2JMw7dwiu3Sjb57hNlVkl4LwN+7vx3tm3DsnhRUoMHvkDtCddC
8bfxpLcOi8WMlySAud+unKnpZVwlwn2rZ/enu2Dd01QKOgOQkBr1HWTJUguPv4No
eWT3UiEZhV4hU764gtF7a8raHjbvxpTJcNk22KHnRmyTKX+SugCyI0qkmIQrFntl
cQWXyA9GV7V+5bK5/Sp2eWv2MXX3fhNDxtZywkmqun+6/YhPgSDJQp3FcKThZFYy
WxPXsrXVIXFJbbtkSs+PGe18VdXqEFCHOI29H1+9gyiC3FW3N6AARw==
=GA3y
-----END PGP SIGNATURE-----
--
Chris Dolan, Software Developer, Clotho Advanced Media Inc.
608-294-7900, fax 294-7025, 1435 E Main St, Madison WI 53703
Clotho Advanced Media, Inc. - Creators of MediaLandscape Software
(http://www.media-landscape.com/) and partners in the revolutionary
Croquet project (http://www.opencroquet.org/)
