On Wed, Apr 16, 2025 at 12:31 AM Thomas Munro <thomas.mu...@gmail.com> wrote: > More or less, yeah, just put the whole ReadStream object in shared > memory, pin an LWLock on it and call it a parallel-aware or shared > ReadStream. But how do you make the locking not terrible? > > My "work stealing" brain dump was imagining a way to achieve the same > net effect, except NOT have to acquire an exclusive lock for every > buffer you pull out of the stream. I was speculating that we could > achieve zero locking for most of the stream without any cache line > ping pong, but a cunning read barrier scheme could detect when you've > been flipped into a slower coordination mode by another backend and > need to turn on some locking and fight over the last handful of > buffers. And I was also observing that if you can figure out to make > it general and reusable enough, we have more unsolved problems like > this in unrelated parallel query code not even involving streams. > It's a tiny more approachable subset of the old "data buffered in > other workers" problem, as I think you called it once.
Maybe the work stealing stuff can hide inside the ReadStream? e.g. a ParallelReadStream is really one ReadStream per participant, each with a separate lock. Mostly you only touch your own, but if necessary you can poke your fingers into other people's ReadStreams. -- Robert Haas EDB: http://www.enterprisedb.com
