On Sat, Sep 7, 2024 at 11:27 AM Tomas Vondra <to...@vondra.me> wrote: > I started looking at this patch today.
Thanks for taking a look! > The first thing I usually do for > new patches is a stress test, so I did a simple script that generates > random table and runs a random query with IN() clause with various > configs (parallel query, index-only scans, ...). And it got stuck on a > parallel query pretty quick. I can reproduce this locally, without too much difficulty. Unfortunately, this is a bug on master/Postgres 17. Some kind of issue in my commit 5bf748b8. The timing of this is slightly unfortunate. There's only a few weeks until the release of 17, plus I have to travel for work over the next week. I won't be back until the 16th, and will have limited availability between then and now. I think that I'll have ample time to debug and fix the issue ahead of the release of 17, though. Looks like the problem is a parallel index scan with SAOP array keys can find itself in a state where every parallel worker waits for the leader to finish off a scheduled primitive index scan, while the leader itself waits for the scan's tuple queue to return more tuples. Obviously, the query will effectively go to sleep indefinitely when that happens (unless and until the DBA cancels the query). This is only possible with just the right/wrong combination of array keys and index cardinality. I cannot recreate the problem with parallel_leader_participation=off, which strongly suggests that leader participation is a factor. I'll find time to study this in detail as soon as I can. Further background: I was always aware of the leader's tendency to go away forever shortly after the scan begins. That was supposed to be safe, since we account for it by serializing the scan's current array keys in shared memory, at the point a primitive index scan is scheduled -- any backend should be able to pick up where any other backend left off, no matter how primitive scans are scheduled. That now doesn't seem to be completely robust, likely due to restrictions on when and how other backends can pick up the scheduled work from within _bt_first, at the point that it calls _bt_parallel_seize. In short, one or two details of how backends call _bt_parallel_seize to pick up BTPARALLEL_NEED_PRIMSCAN work likely need to be rethought. -- Peter Geoghegan
