On 1/22/16 12:14 PM, Andres Freund wrote:
On 2016-01-22 08:40:28 -0600, Jim Nasby wrote:
Ideally reviewers shouldn't be doing any testing, because the tests
that are part of the patch should answer every question they would
have, but I don't see that happening until we have a separate
automation-only target that we don't care how long it takes to run.
I think that's completely wrong.
Yes, more tests are good, and we need a place for longer running
tests. But assuming that every patch author will create a testsuite that
covers every angle is just about akin to assuming every submitter will
deliver perfect, bug free code. And we know how well that turns out.
I think actively trying to break a feature, and postgres in general, is
one of the most important tasks of reviewers and testers. And with that
I don't mean trying to run "make check". Look e.g. at the tests Jeff
Janes has performed, what the recent plug tests of Tomas Vondra brought
to light, or at what the full page write checker tool of Heikki's
showed.
IIRC Jeff's tests are scripted and obviously the page write checker is
as well. I don't recall the exact methodology Tomas was using but I
suspect it could also be scripted if you had a way to pull the plug via
software (via a power management unit or maybe kill -9 of a VM). All of
that is stuff that can and should be automated. Presumably it won't ever
be part of the Makefile tests, but that's fine. Heck, the test scripts
could stay in completely separate repos.
Where the code lives isn't the issue; it's getting stuff like this
automated so humans can go back do doing things that can't be automated.
--
Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX
Experts in Analytics, Data Architecture and PostgreSQL
Data in Trouble? Get it in Treble! http://BlueTreble.com
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
