Andrew Pinski wrote:
My guess is that most or all of those are factors, but some are more
important than others. My favorite tactic to decrease the number of
bugs is to set up a unit test framework for your code base (so you can
test changes to individual functions without having to run the whole
compiler), and to strongly encourage patches to be accompanied by unit
tests.
This does not catch corner cases which is where most of the issues.
This gets us back to the original problem, where you can think of inputs
that should be valid but no, there are other inputs which are valid.
Andrew, you are both correct and incorrect. Certainly, deterministic
unit testing
is not very useful. However, random unit testing is priceless. I have
been doing
pseudo-random unit tests for years, believe me it makes your code
practically
bug free.
By employing random tests, you free yourself from most of the burden of
"there are other inputs which are valid". Also, the tests are not fragile
w.r.t. minor changes that shift corner cases.
Deterministic unit-tests are almost useless in long lived projects,
random-tests are priceless. Writing a random test is much more difficult,
but it is also more interesting. You have to figure out how to
automatically
check the results, and how to have a reasonable probability for all sorts
of potential corner cases.
I have been doing this for years, and as a result people rarely find
bugs in my code (and I have a complex algorithmic code with complex
data structures and transformations). I write code with lots of bugs, but
almost all of them are caught by my random unit tests.
A simple google search for: random unit test
gave me the following papers that look promising:
http://www.csd.uwo.ca/faculty/andrews/papers/sourceforge-tr.ps
http://netlib.bell-labs.com/who/god/public_psfiles/pldi2005.pdf
I am willing to help here, as long as no copyright assignment is involved.
(Time constraints, and legal issues I don't want to iron out). I would have
written a paper on the subject if it were not that difficult to make it
scientific (e.g. how to have a reliable measure of quality of testing).
--
Michael Veksler
http:///tx.technion.ac.il/~mveksler
