On Tue, Apr 9, 2024 at 7:47 AM Robert Haas <robertmh...@gmail.com> wrote: > - The streaming read API stuff was all committed very last minute. I > think this should have been committed much sooner. It's probably not > going to break the world; it's more likely to have performance > consequences. But if it had gone in sooner, we'd have had more time to > figure that out.
OK, let me give an update on this work stream (pun intended). One reason for the delay in committing was precisely that we were fretting about regressions risks. We tried pretty hard to identify and grind down every regression we could find, and cases with outstanding not-fully-understood or examined problems in that area have been booted into the next cycle for more work: streaming bitmap heapscan, several streaming vacuum patches, and more, basically things that seem to have more complex interactions with other machinery. The only three places using streaming I/O that went in were: 041b9680: Use streaming I/O in ANALYZE. b7b0f3f2: Use streaming I/O in sequential scans. 3a352df0: Use streaming I/O in pg_prewarm. The first is a good first exercise in streaming random blocks; hopefully no one would be too upset about an unexpected small regression in ANALYZE, but as it happens it goes faster hot and cold according to all reports. The second is a good first exercise in streaming sequential blocks, and it ranges from faster to no regression, according to testing and reports. The third is less important, but it also goes faster. Of those, streaming seq scan is clearly the most relevant to real workloads that someone might be upset about, and I made a couple of choices that you might say had damage control in mind: * A conservative choice not to get into the business of the issuing new hints to the kernel for random jumps in cold scans, even though we think we probably should for better performance: more research needed precisely to avoid unexpected interactions (cf the booted bitmap heapscan where that sort of thing seems to be afoot). * A GUC to turn off I/O combining if it somehow upsets your storage in ways we didn't foresee (io_combine_limit=1). For fully cached hot scans, it does seem to be quite sensitive to tiny changes in a hot code path that I and others spent a lot of time optimising and testing during the CF. Perhaps it is possible that someone else's microarchitecture or compiler could show a regression that I don't see, and I will certainly look into it with vim and vigour if so. In that case we could consider a tiny micro-optimisation that I've shared already (it seemed a little novel so I'd rather propose it in the new cycle if I can), or, if it comes to it based on evidence and inability to address a problem quickly, reverting just b7b0f3f2 which itself is a very small patch. An aspect you didn't mention is correctness. I don't actually know how to prove that buffer manager protocols are correct beyond thinking and torture testing, ie what kind of new test harness machinery could be used to cross-check more things about buffer pool state explicitly, and that is a weakness I'm planning to look into. I realise that "these are the good ones, you should see all the stuff we decided not to commit!" is not an argument, I'm just laying out how I see the patches that went in and why I thought they were good. It's almost an architectural change, but done in tiny footsteps. I appreciate that people would have liked to see those particular tiny footsteps in some of the other fine months available for patching the tree, and some of the earlier underpinning patches that were part of the same patch series did go in around New Year, but clearly my "commit spreading" didn't go as well as planned after that (not helped by Jan/Feb summer vacation season down here). Mr Paquier this year announced his personal code freeze a few weeks back on social media, which seemed like an interesting idea I might adopt. Perhaps that is what some other people are doing without saying so, and perhaps the time they are using for that is the end of the calendar year. I might still be naturally inclined to crunch-like behaviour, but it wouldn't be at the same time as everyone else, except all the people who follow the same advice.
