On 5/9/16, Stephen Frost <sfr...@snowman.net> wrote: >> And what if the tests are buggy? New test suites should go through a >> CF first I think for proper review. And this is clearly one. > > They still won't result in data loss, corruption, or other direct impact > on our users, even if they are buggy. They also won't destablize the > code base or introduce new bugs into the code.
Yes, but they could make things worse long term. For example if introduce intermittent failures or if they're hard to understand or if they test internals too deeply and don't let those internals evolve because the tests break. All of them reasons why it's good that they're reviewed. > There is pretty clearly a lack of consensus on the question about adding > new tests. > What *should* we be doing right now? Reviewing code and testing the > system is what I had thought but perhaps I'm wrong. To the extent that > we're testing the system, isn't it better to write repeatable tests, > particularly ones which add code coverage, than one-off tests that only > check that the current snapshot of the code is operating correctly? I also think that testing *and* keeping those tests for the future is more valuable than just testing. But committers can just push their tests while non committers need to wait for the next CF to get their new tests committed. And committers pushing tests means they bypass the review process which, as far as I know, doesn't happen for regular patches: committers usually only directly commit small fixes not lots of new (test) code. So, crazy idea: what about test only CommitFests in between now and the release? The rules would be: only new tests and existing test improvements are allowed. For the rest, the CF would be the same as usual: have an end date, committers agree to go through each patch and commit or return it with feedback etc. Stephen would have added his tests to the upcoming June test CF, Michael would have reviewed them and maybe add his own and so on. This does create work for committers (go through the submitted patches) but acts as an incentive for test writers. Want your test code committed? Well, there will be a CF very soon where it will get committer attention so better write that test. It also gives a clear date after which test churn also stops in order to not destabilize the buildfarm right before a release. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
