On Wed, Jan 24, 2018 at 11:02 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: > I may be wasting my breath here, but in one more attempt to convince > you that "time make check" on your laptop is not the only number that > anyone should be interested in, ...
Now that is not what I said, or at least not what I intended to say. I'm taking the position that what happens on a developer laptop is more relevant than what happens on an ancient buildfarm critter. I am NOT taking the position that my particular laptop or even developer laptops generally are the *only* thing that matters. I gave the numbers from my laptop because it's the one I can test. I cannot easily test yours. > What I take away here is that there's been a pretty steep cost increase > for the regression tests since v10, and that is *not* in line with the > historical average. In fact, in most years we've bought enough speedup > through performance improvements to pay for the test cases we added. > This is masked if you just eyeball "make check" compared to several years > ago. But to do that, you have to ignore the fact that we made substantial > improvements in the runtime of initdb as well as the regression tests > proper circa v10, and we've now thrown that away and more. OK, I can see some increase there. It's definitely more for you than it is for me, since you see something like a 10% slowdown between 10 and master and I see basically no difference. I don't know why that should be, but I'm not doubting you. > So I remain dissatisfied with these results, particularly because in > my own work habits, the time for "make installcheck-parallel" is way > more interesting than "make check". I avoid redoing installs and > initdbs if I don't need them. I'm not that efficient, but noted. >> ... Even if that meant that you had >> to wait 1 extra second every time you run 'make check', I would judge >> that worthwhile. > > I think this is a bad way of looking at it. Sure, in terms of > one developer doing one test run, a second or two means nothing. > But for instance, if you want to do 100 test runs in hope of catching > a seldom-reproduced bug, it adds up. It also adds up when you consider > the aggregate effort expended by the buildfarm, or the time you have > to wait to see buildfarm results. Sure, but as Andres said, you also have to consider how much developer time it takes to recoup the savings. If it takes a day of development time to save a second of regression test time, that might be worth it; if it takes a month, color me doubtful, especially if the result is a more fragile test that happens to pass on all of the buildfarm critters we have now but might fail spuriously when somebody adds a new one. > Yeah. What I thought this argument was about was convincing *you* > that that would be a reasonable patch to apply. It seems from my > experiment on gaur that that patch makes the results unstable, so > if we can do it at all it will need more work. But I do think > it's worth putting in some more sweat here. In the long run the > time savings will add up. Why me? Thomas wrote the patch, Andres committed it, and you complained about it. I'm just along for the ride. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company
