Re: Proposal: Limitations of palloc inside checkpointer

Hi, I've moved this entry to the July CommitFest. The CI reported an unused variable warning in patch v5, so I've addressed it by removing the unused one. Sorry for not catching the issue locally. Xuneng Zhou 于2025年4月10日周四 23:43写道： > Hi, > > I’ve updated and rebased Maxim's patch version 4 with

Re: Proposal: Limitations of palloc inside checkpointer

Hi, I’ve updated and rebased Maxim's patch version 4 with optimizations like ring buffer and capped max_requests proposed by Heikki. These are the summary of Changes in v5: • Replaced the linear array model in AbsorbSyncRequests() with a bounded ring buffer to avoid large memmove() operations wh

Re: Proposal: Limitations of palloc inside checkpointer

On 12/03/2025 13:00, Maxim Orlov wrote: On Wed, 12 Mar 2025 at 10:27, Xuneng Zhou > wrote: The patch itself looks ok to me. I'm curious about the trade-offs between this incremental approach and the alternative of using palloc_extended() with the MCXT_ALLOC_HUGE fla

Re: Proposal: Limitations of palloc inside checkpointer

On Wed, 12 Mar 2025 at 10:27, Xuneng Zhou wrote: > Hi, > The patch itself looks ok to me. I'm curious about the trade-offs between > this incremental approach and the alternative of > using palloc_extended() with the MCXT_ALLOC_HUGE flag. The approach of > splitting the requests into fixed-size s

Re: Proposal: Limitations of palloc inside checkpointer

Hi, The patch itself looks ok to me. I'm curious about the trade-offs between this incremental approach and the alternative of using palloc_extended() with the MCXT_ALLOC_HUGE flag. The approach of splitting the requests into fixed-size slices avoids OOM failures or process termination by the OOM

Re: Proposal: Limitations of palloc inside checkpointer

I think I figured it out. Here is v4. If the number of requests is less than 1 GB, the algorithm stays the same as before. If we need to process more, we will do it incrementally with slices of 1 GB. Best regards, Maxim Orlov. v4-0001-Process-sync-requests-incrementally-in-AbsorbSync.patch Desc

Re: Proposal: Limitations of palloc inside checkpointer

Here is an alternative solution. We can limit the number of processed requests to fit in a 1Gb memory chunk for each pass. Obviously, we left some requests in the queue to be processed in the next call. We can overcome this by using multi-step processing: estimating the number of steps in the begin

Re: Proposal: Limitations of palloc inside checkpointer

After done some testing, I found a bug in the patch. If more requests were pushed while we release the lock, num_requests could not be set to zero. Here is a fixed version. -- Best regards, Maxim Orlov. v2-0001-AbsorbSyncRequests-incrementally-instead-of-doing.patch Description: Binary data

Re: Proposal: Limitations of palloc inside checkpointer

I tried to implement the idea (4). This is the patch. But, there is a problem. See, when we release lock and call RememberSyncRequest() and acquire it again, CompactCheckpointerRequestQueue() may be called and the state of the request array may be changed. Of course, we can recheck num_requests af

Re: Proposal: Limitations of palloc inside checkpointer

On Wed, 26 Feb 2025 at 11:54, Andres Freund wrote: > > 4) Do compaction incrementally, instead of doing it for all requests at > once. Yeah, good idea! I completely forgot about that. Thanks! -- Best regards, Maxim Orlov.

Re: Proposal: Limitations of palloc inside checkpointer

Hi, On 2025-02-26 11:46:45 +0300, Maxim Orlov wrote: > On Tue, 25 Feb 2025 at 22:44, Ekaterina Sokolova > wrote: > > > Hi, hackers! > > > > Historically, the checkpointer process use palloc() into > > AbsorbSyncRequests() function. Therefore, the checkpointer does not > > expect to receive a req

Re: Proposal: Limitations of palloc inside checkpointer

On Tue, 25 Feb 2025 at 22:44, Ekaterina Sokolova wrote: > Hi, hackers! > > Historically, the checkpointer process use palloc() into > AbsorbSyncRequests() function. Therefore, the checkpointer does not > expect to receive a request larger than 1 GB. Yeah. And the most unpleasant thing is it won

Proposal: Limitations of palloc inside checkpointer

Hi, hackers! Historically, the checkpointer process use palloc() into AbsorbSyncRequests() function. Therefore, the checkpointer does not expect to receive a request larger than 1 GB. We encountered a case where the database went into recovery state, after applying all wal, the checkpointer