Hi, On 2025-03-23 01:45:35 -0700, Masahiko Sawada wrote: > Another idea is that parallel workers don't exit phase 1 until it > consumes all pinned buffers in the queue, even if the memory usage of > TidStore exceeds the limit.
Yes, that seems a quite reasonable approach to me. > It would need to add new functionality to the read stream to disable the > look-ahead reading. Couldn't your next block callback simply return InvalidBlockNumber once close to the memory limit? > Since we could use much memory while processing these buffers, exceeding the > memory limit, we can trigger this mode when the memory usage of TidStore > reaches 70% of the limit or so. It wouldn't be that much memory, would it? A few 10s-100s of buffers don't increase the size of a TidStore that much? Using 10 parallel vacuum with a m_w_m of 1MB doesn't make sense, we'd constantly start/stop workers. > On the other hand, it means that we would not use the streaming read for the > blocks in this mode, which is not efficient. I don't follow - why wouldn't you be using streaming read? Greetings, Andres Freund
