On 19 March 2012 00:10, Andrew Dunstan <and...@dunslane.net> wrote: > If your tests are that voluminous then maybe they are not what we're looking > for anyway. As Tom noted: > > > IMO the objective of a regression test is not to memorialize every single > case the code author thought about during development. ISTM it would not > take very many cases to have reasonable code coverage.
Fair enough. > Why exactly does this feature need particularly to have script-driven > regression test generation when others don't? It's not that it needs it, so much as that it is possible to provide coverage for much of the code with black-box testing. In the case of most of the hundreds of tests, I can point to a particular piece of code that is being tested, that was written *after* the test was. Doing this with pg_regress the old-fashioned way is going to be incredibly verbose. I'm all for doing script-generation of pg_regress tests in a well-principled way, and I'm happy to take direction from others as to what that should look like. I know that for the most part the tests provide coverage for discrete units of functionality, and so add value. If they add value, why not include them? Tests are supposed to be comprehensive. If that inconveniences you, by slowing down the buildfarm for questionable benefits, maybe it would be okay to have some tests not run automatically, even if that did make them "next door to useless" in Tom's estimation. There could be a more limited set of conventional pg_regress tests that are run automatically, plus more comprehensive tests that are run less frequently, typically only as it becomes necessary to alter pg_stat_statements to take account of those infrequent changes (typically additions) to the query tree. We have tests that ensure that header files don't contain C++ keywords, and nominally promise to not do so, and they are not run automatically. I don't see the sense in requiring that tests should be easy to run, while also aspiring to have tests that are as useful and comprehensive as possible. It seems like the code should dictate the testing infrastructure, and not the other way around. Part of the reason why I'm resistant to reducing the number of tests is that it seems to me that excluding some tests but not others would be quite arbitrary. It is not the case that some tests are clearly more useful than others (except for the fuzz testing stuff, which probably isn't all that useful). -- Peter Geoghegan http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
