On Tue, Mar 7, 2023 at 1:01 AM John Naylor <john.nay...@enterprisedb.com> wrote: > > On Mon, Mar 6, 2023 at 1:28 PM Masahiko Sawada <sawada.m...@gmail.com> wrote: > > > > Since the block-level measurement is likely overestimating quite a bit, I > > > propose to simply reverse the order of the actions here, effectively > > > reporting progress for the *last page* and not the current one: First > > > update progress with the current memory usage, then add tids for this > > > page. If this allocated a new block, only a small bit of that will be > > > written to. If this block pushes it over the limit, we will detect that > > > up at the top of the loop. It's kind of like our earlier attempts at a > > > "fudge factor", but simpler and less brittle. And, as far as OS pages we > > > have actually written to, I think it'll effectively respect the memory > > > limit, at least in the local mem case. And the numbers will make sense. > > > > > > Thoughts? > > > > It looks to work but it still doesn't work in a case where a shared > > tidstore is created with a 64kB memory limit, right? > > TidStoreMemoryUsage() returns 1MB and TidStoreIsFull() returns true > > from the beginning. > > I have two ideas: > > 1. Make it optional to track chunk memory space by a template parameter. It > might be tiny compared to everything else that vacuum does. That would allow > other users to avoid that overhead. > 2. When context block usage exceeds the limit (rare), make the additional > effort to get the precise usage -- I'm not sure such a top-down facility > exists, and I'm not feeling well enough today to study this further.
I prefer option (1) as it's straight forward. I mentioned a similar idea before[1]. RT_MEMORY_USAGE() is defined only when the macro is defined. It might be worth checking if there is visible overhead of tracking chunk memory space. IIRC we've not evaluated it yet. [1] https://www.postgresql.org/message-id/CAD21AoDK3gbX-jVxT6Pfso1Na0Krzr8Q15498Aj6tmXgzMFksA%40mail.gmail.com Regards, -- Masahiko Sawada Amazon Web Services: https://aws.amazon.com
