Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-11 Thread Peter Geoghegan
On Wed, Jan 11, 2023 at 4:44 PM Andres Freund wrote: > > Attached patch fixes up these issues. It's almost totally mechanical. > > Looks better, thanks! Pushed that just now. Thanks -- Peter Geoghegan

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-11 Thread Andres Freund
Hi, On 2023-01-11 16:06:31 -0800, Peter Geoghegan wrote: > On Mon, Jan 9, 2023 at 2:18 PM Peter Geoghegan wrote: > > I'll try to get back to it this week. > > Attached patch fixes up these issues. It's almost totally mechanical. Looks better, thanks! > (Ended up using "git diff --color-moved=

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-11 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 2:18 PM Peter Geoghegan wrote: > I'll try to get back to it this week. Attached patch fixes up these issues. It's almost totally mechanical. (Ended up using "git diff --color-moved=dimmed-zebra --color-moved-ws=ignore-all-space" with this, per your recent tip, which did he

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 1:43 PM Andres Freund wrote: > ISTM that some of the page level freezing functions are misnamed. In heapam.c > the heap_xlog* routines are for replay, afaict. However > heap_xlog_freeze_plan() is used to WAL log the freeze > plan. heap_xlog_freeze_page() is used to replay th

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2023-01-09 Thread Andres Freund
Hi, On 2022-11-15 10:26:05 -0800, Peter Geoghegan wrote: > Pushed something like this earlier today, though without any changes > to VISIBLE records. While updating a patch to log various offsets in pg_waldump, I noticed a few minor issues in this patch: ISTM that some of the page level freezing

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-11-15 Thread Peter Geoghegan
On Fri, Nov 11, 2022 at 10:38 AM Peter Geoghegan wrote: > Attached is v4, which removes the old comments you pointed out were > now out of place (they weren't adding much anyway). Also fixed bitrot > against HEAD from today's visibility map commit from Jeff Davis. Pushed something like this earli

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-11-11 Thread Peter Geoghegan
On Thu, Nov 10, 2022 at 7:00 PM Justin Pryzby wrote: > Note that this comment is dangling in your patch: Attached is v4, which removes the old comments you pointed out were now out of place (they weren't adding much anyway). Also fixed bitrot against HEAD from today's visibility map commit from J

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-11-10 Thread Justin Pryzby
On Thu, Nov 10, 2022 at 04:48:17PM -0800, Peter Geoghegan wrote: > On Tue, Sep 20, 2022 at 3:12 PM Peter Geoghegan wrote: > > Attached is v2, which I'm just posting to keep CFTester happy. No real > > changes here. > > Attached is v3. I'd like to move forward with commit soon. I'll do so > in the

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-11-10 Thread Peter Geoghegan
On Tue, Sep 20, 2022 at 3:12 PM Peter Geoghegan wrote: > Attached is v2, which I'm just posting to keep CFTester happy. No real > changes here. Attached is v3. I'd like to move forward with commit soon. I'll do so in the next few days, barring objections. v3 has vacuumlazy.c pass NewRelfrozenXid

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-22 Thread Peter Geoghegan
On Wed, Sep 21, 2022 at 9:21 PM Nathan Bossart wrote: > I wouldn't mind giving this a try. Definitely seems worth experimenting with. Many WAL records generated during VACUUM (and during opportunistic pruning/index tuple deletion) have offset number arrays that can be assumed to be both sorted an

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-21 Thread Nathan Bossart
On Wed, Sep 21, 2022 at 02:11:36PM -0700, Peter Geoghegan wrote: > On Wed, Sep 21, 2022 at 1:14 PM Nathan Bossart > wrote: >> Presumably a >> generic WAL record compression mechanism could be reused for other large >> records, too. That could be much easier than devising a deduplication >> strat

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-21 Thread Nathan Bossart
On Wed, Sep 21, 2022 at 02:41:28PM -0700, Peter Geoghegan wrote: > On Wed, Sep 21, 2022 at 2:11 PM Peter Geoghegan wrote: >> > Presumably a >> > generic WAL record compression mechanism could be reused for other large >> > records, too. That could be much easier than devising a deduplication >> >

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-21 Thread Peter Geoghegan
On Wed, Sep 21, 2022 at 2:11 PM Peter Geoghegan wrote: > > Presumably a > > generic WAL record compression mechanism could be reused for other large > > records, too. That could be much easier than devising a deduplication > > strategy for every record type. > > It's quite possible that that's a

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-21 Thread Peter Geoghegan
On Wed, Sep 21, 2022 at 1:14 PM Nathan Bossart wrote: > This idea seems promising. I see that you called this patch a > work-in-progress, so I'm curious what else you are planning to do with it. I really just meant that the patch wasn't completely finished at that point. I hadn't yet convinced m

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-21 Thread Nathan Bossart
On Tue, Sep 20, 2022 at 03:12:00PM -0700, Peter Geoghegan wrote: > On Mon, Sep 12, 2022 at 2:01 PM Peter Geoghegan wrote: >> I'd like to talk about one such technique on this thread. The attached >> WIP patch reduces the size of xl_heap_freeze_page records by applying >> a simple deduplication pro

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-20 Thread Peter Geoghegan
On Mon, Sep 12, 2022 at 2:01 PM Peter Geoghegan wrote: > I'd like to talk about one such technique on this thread. The attached > WIP patch reduces the size of xl_heap_freeze_page records by applying > a simple deduplication process. Attached is v2, which I'm just posting to keep CFTester happy.

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-16 Thread Peter Geoghegan
On Fri, Sep 16, 2022 at 12:30 AM Masahiko Sawada wrote: > After a quick benchmark, I've confirmed that the amount of WAL records > for freezing 1 million tuples reduced to about one-fifth (1.2GB vs > 250MB). Great. I think that the really interesting thing about the patch is how this changes the

Re: Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-16 Thread Masahiko Sawada
On Tue, Sep 13, 2022 at 6:02 AM Peter Geoghegan wrote: > > My ongoing project to make VACUUM more predictable over time by > proactive freezing [1] will increase the overall number of tuples > frozen by VACUUM significantly (at least in larger tables). It's > important that we avoid any new user-v

Reducing the WAL overhead of freezing in VACUUM by deduplicating per-tuple freeze plans

2022-09-12 Thread Peter Geoghegan
My ongoing project to make VACUUM more predictable over time by proactive freezing [1] will increase the overall number of tuples frozen by VACUUM significantly (at least in larger tables). It's important that we avoid any new user-visible impact from extra freezing, though. I recently spent a lot