Peter Geoghegan <pe...@2ndquadrant.com> writes: > On 19 March 2012 00:10, Andrew Dunstan <and...@dunslane.net> wrote: >> Why exactly does this feature need particularly to have script-driven >> regression test generation when others don't?
> It's not that it needs it, so much as that it is possible to provide > coverage for much of the code with black-box testing. In the case of > most of the hundreds of tests, I can point to a particular piece of > code that is being tested, that was written *after* the test was. Well, basically what you're saying is that you did test-driven development, which is fine. However, that does not mean that those same tests are ideal for ongoing regression testing. What we want from a regression test these days is primarily (a) portability testing, ie does the feature work on platforms other than yours?, and (b) early warning if someone breaks it down the road. In most cases, fairly coarse testing is enough to catch drive-by breakage; and when it's not enough, like as not the breakage is due to something you never thought about originally and thus never tested for, so you'd not have caught it anyway. I am *not* a fan of regression tests that try to microscopically test every feature in the system. Sure you should do that when initially developing a feature, but it serves little purpose to do it over again every time any other developer runs the regression tests for the foreseeable future. That road leads to a regression suite that's so voluminous that it takes too long to run and developers start to avoid running it, which is counterproductive. For an example in our own problem space look at mysql, whose regression tests take well over an hour to run on a fast box. So they must be damn near bug-free right? Uh, not so much, and I think the fact that developers can't easily run their test suite is not unrelated to that. So what I'd rather see is a small set of tests that are designed to do a smoke-test of functionality and then exercise any interfaces to the rest of the system that seem likely to break. Over time we might augment that, when we find particular soft spots as a result of previously undetected bugs. But sheer volume of tests is not a positive IMO. As for the scripted vs raw-SQL-in-pg_regress question, I'm making the same point as Andrew: only the pg_regress method is likely to get run nearly everywhere, which means that the scripted approach is a FAIL so far as the portability-testing aspect is concerned. Lastly, even given that we were willing to accept a scripted set of tests, I'd want to see it in perl not python. Perl is the project standard; I see no reason to expect developers to learn two different scripting languages to work on PG. (There might be a case for accepting python-scripted infrastructure for pl/python, say, but not for components that are 100% unrelated to python.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers