Re: Some read stream improvements

2025-04-06 Thread Thomas Munro
On Mon, Apr 7, 2025 at 1:29 PM Andres Freund wrote: > On 2025-04-03 14:43:40 +1300, Thomas Munro wrote: > > On Thu, Apr 3, 2025 at 11:17 AM Andres Freund wrote: > > > I don't actually see any reason for buffer_limit to be a 16bit quantity? > > > It's > > > just to clamp things down, right? > > >

Re: Some read stream improvements

2025-04-06 Thread Andres Freund
Hi, On 2025-04-03 14:43:40 +1300, Thomas Munro wrote: > On Thu, Apr 3, 2025 at 11:17 AM Andres Freund wrote: > > I don't actually see any reason for buffer_limit to be a 16bit quantity? > > It's > > just to clamp things down, right? > > Ugh. It might be worth just flipping this whole thing ove

Re: Some read stream improvements

2025-04-02 Thread Thomas Munro
On Thu, Apr 3, 2025 at 11:17 AM Andres Freund wrote: > I don't actually see any reason for buffer_limit to be a 16bit quantity? It's > just to clamp things down, right? Ugh. It might be worth just flipping this whole thing over to ints, let me look into that...

Re: Some read stream improvements

2025-04-02 Thread Andres Freund
Hi, On 2025-03-14 22:03:15 +1300, Thomas Munro wrote: > I have pushed the new pin limit patches, after some more testing and > copy editing. I dropped an unnecessary hunk (in read_stream_reset(), a > change I'd like to make but it didn't belong in this commit) and > dropped the word "Soft" from Ge

Re: Some read stream improvements

2025-03-30 Thread Thomas Munro
On Tue, Mar 18, 2025 at 5:56 AM Andres Freund wrote: > So one thing is that the pin count differs by 1 at the start of the scan. No > idea why. > > I still don't know what drives the difference between freebsd and the rest, > but IIUC the reason this fails is just that we are holding too many buff

Re: Some read stream improvements

2025-03-17 Thread Andres Freund
Hi, On 2025-03-14 22:03:15 +1300, Thomas Munro wrote: > I have pushed the new pin limit patches, after some more testing and > copy editing. I dropped an unnecessary hunk (in read_stream_reset(), a > change I'd like to make but it didn't belong in this commit) and > dropped the word "Soft" from Ge

Re: Some read stream improvements

2025-03-14 Thread Thomas Munro
I have pushed the new pin limit patches, after some more testing and copy editing. I dropped an unnecessary hunk (in read_stream_reset(), a change I'd like to make but it didn't belong in this commit) and dropped the word "Soft" from GetSoftPinLimit() as it wasn't helping comprehension and isn't ev

Re: Some read stream improvements

2025-03-13 Thread Thomas Munro
On Wed, Mar 12, 2025 at 8:29 AM Andres Freund wrote: > On 2025-03-12 07:35:46 +1300, Thomas Munro wrote: > > On Thu, Feb 27, 2025 at 11:20 PM Andres Freund wrote: > > > On 2025-02-27 11:19:55 +1300, Thomas Munro wrote: > > I wonder if we should use temp_buffers - 100? Then leave the minimum GUC

Re: Some read stream improvements

2025-03-11 Thread Andres Freund
Hi, On 2025-03-12 07:35:46 +1300, Thomas Munro wrote: > On Thu, Feb 27, 2025 at 11:20 PM Andres Freund wrote: > > On 2025-02-27 11:19:55 +1300, Thomas Munro wrote: > > > On Wed, Feb 26, 2025 at 10:55 PM Andres Freund wrote: > > > > I was working on expanding tests for AIO and as part of that wro

Re: Some read stream improvements

2025-03-11 Thread Thomas Munro
On Thu, Feb 27, 2025 at 11:20 PM Andres Freund wrote: > On 2025-02-27 11:19:55 +1300, Thomas Munro wrote: > > On Wed, Feb 26, 2025 at 10:55 PM Andres Freund wrote: > > > I was working on expanding tests for AIO and as part of that wrote a test > > > for > > > temp tables -- our coverage is fairl

Re: Some read stream improvements

2025-02-27 Thread Andres Freund
On 2025-02-27 11:19:55 +1300, Thomas Munro wrote: > On Wed, Feb 26, 2025 at 10:55 PM Andres Freund wrote: > > I was working on expanding tests for AIO and as part of that wrote a test > > for > > temp tables -- our coverage is fairly awful, there were many times during > > AIO > > development wh

Re: Some read stream improvements

2025-02-27 Thread Thomas Munro
On Thu, Feb 27, 2025 at 11:19 AM Thomas Munro wrote: > On Wed, Feb 26, 2025 at 10:55 PM Andres Freund wrote: > > I was working on expanding tests for AIO and as part of that wrote a test > > for > > temp tables -- our coverage is fairly awful, there were many times during > > AIO > > developmen

Re: Some read stream improvements

2025-02-26 Thread Thomas Munro
On Wed, Feb 26, 2025 at 10:55 PM Andres Freund wrote: > I was working on expanding tests for AIO and as part of that wrote a test for > temp tables -- our coverage is fairly awful, there were many times during AIO > development where I knew I had trivially reachable temp table specific bugs > but

Re: Some read stream improvements

2025-02-26 Thread Andres Freund
Hi, On 2025-02-17 17:55:09 +1300, Thomas Munro wrote: > 0004-Respect-pin-limits-accurately-in-read_stream.c.patch > > The current coding only computes the remaining "fair share" of the > buffer pool for this backend at stream initialisation. It's hard, but > not impossible, to get one backend to

Re: Some read stream improvements

2025-02-16 Thread Thomas Munro
On Mon, Feb 17, 2025 at 6:55 PM Kirill Reshke wrote: > Just out of curiosity, should we `Assert(*index + n < > stream->queue_size);` in `read_stream_index_advance_n`? No: it is allowed to be >= queue_size temporarily, but if so we subtract queue_size. The result should be equal to (index + n) %

Re: Some read stream improvements

2025-02-16 Thread Thomas Munro
On Mon, Feb 17, 2025 at 5:55 PM Thomas Munro wrote: > The solution we agreed on is to introduce a way for StartReadBuffers() > to communicate with future calls, and "forward" pinned buffers between > calls. The function arguments don't change, but its "buffers" > argument becomes an in/out array:

Re: Some read stream improvements

2025-02-16 Thread Kirill Reshke
On Mon, 17 Feb 2025 at 09:55, Thomas Munro wrote: > > Hi, > > Here are some patches that address some of Andres's feedback since the > AIO v2 rebase[1], anticipate out-of-order streams, and make some other > minor improvements. They are independent of the main AIO patch set > and apply to master,

Some read stream improvements

2025-02-16 Thread Thomas Munro
Hi, Here are some patches that address some of Andres's feedback since the AIO v2 rebase[1], anticipate out-of-order streams, and make some other minor improvements. They are independent of the main AIO patch set and apply to master, hence separate thread. 0001-Refactor-read_stream.c-s-circular-