Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-25 Thread Peter Geoghegan
On Wed, Jan 25, 2023 at 1:19 AM Andres Freund wrote: > Here's a draft of what I am thinking of. Not perfect yet, but I think it looks > better. I'm afraid that I will be unable to do any more work on this project. I have withdrawn it from the CF app. If you would like to complete some or all of

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-25 Thread Robert Haas
On Tue, Jan 24, 2023 at 3:33 PM Peter Geoghegan wrote: > Sure, it's possible that such a cancellable aggressive autovacuum was > indeed cancelled, and that that factor made the crucial difference. > But I find it far easier to believe that there simply was no such > aggressive autovacuum in the fi

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-25 Thread Andres Freund
Hi, On 2023-01-24 20:59:04 -0800, Andres Freund wrote: > I find this to be awkward code. The booleans are kinda pointless, and the > tableagevac case is hard to follow because trigger is set elsewhere. > > I can give reformulating it a go. Need to make some food first. Here's a draft of what I am

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-24 Thread Andres Freund
Hi, On 2023-01-23 19:22:18 -0800, Peter Geoghegan wrote: > On Mon, Jan 23, 2023 at 6:56 PM Andres Freund wrote: > > Why is there TABLE_ in AUTOVACUUM_TABLE_XID_AGE but not > > AUTOVACUUM_DEAD_TUPLES? Both are on tables. > > Why does vacuum_freeze_table_age contain the word "table", while > autov

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-24 Thread Peter Geoghegan
On Tue, Jan 24, 2023 at 11:21 AM Robert Haas wrote: > > The whole article was about how this DROP TRIGGER pattern worked just > > fine most of the time, because most of the time autovacuum was just > > autocancelled. They say this at one point: > > > > "The normal autovacuum mechanism is skipped w

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-24 Thread Robert Haas
On Fri, Jan 20, 2023 at 4:24 PM Peter Geoghegan wrote: > > It sounds like they used DROP TRIGGER pretty regularly. So I think this > > sounds like exactly the kind of case I was talking about, where > > autovacuums keep getting cancelled until we decide to stop cancelling > > them. > > I don't kno

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-23 Thread Peter Geoghegan
On Mon, Jan 23, 2023 at 6:56 PM Andres Freund wrote: > Why is there TABLE_ in AUTOVACUUM_TABLE_XID_AGE but not > AUTOVACUUM_DEAD_TUPLES? Both are on tables. Why does vacuum_freeze_table_age contain the word "table", while autovacuum_vacuum_scale_factor does not? To me, "table XID age" is a less

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-23 Thread Andres Freund
Hi, On 2023-01-18 17:54:27 -0800, Peter Geoghegan wrote: > From 0afaf310255a068d3c1ca9d2ce6f00118cbff890 Mon Sep 17 00:00:00 2001 > From: Peter Geoghegan > Date: Fri, 25 Nov 2022 11:23:20 -0800 > Subject: [PATCH v5 1/2] Add autovacuum trigger instrumentation. > > Add new instrumentation that lis

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-21 Thread Peter Geoghegan
On Sat, Jan 21, 2023 at 6:54 PM Andres Freund wrote: > Is > https://www.postgresql.org/message-id/CAH2-WzmytCuSpaMEhv8H-jt8x_9whTi0T5bjNbH2gvaR0an2Pw%40mail.gmail.com > the last / relevant version of the patch to look at? Yes. I'm mostly just asking about v5-0001-* right now. -- Peter Geoghega

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-21 Thread Andres Freund
Hi, On 2023-01-21 18:37:59 -0800, Peter Geoghegan wrote: > On Tue, Jan 17, 2023 at 10:02 AM Andres Freund wrote: > > I think with a bit of polish "Add autovacuum trigger instrumentation." ought > > to be quickly mergeable. > > Any thoughts on v5-0001-*? > > It would be nice to get the uncontent

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-21 Thread Peter Geoghegan
On Tue, Jan 17, 2023 at 10:02 AM Andres Freund wrote: > I think with a bit of polish "Add autovacuum trigger instrumentation." ought > to be quickly mergeable. Any thoughts on v5-0001-*? It would be nice to get the uncontentious part of all this (which is the instrumentation patch) out of the wa

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-20 Thread Peter Geoghegan
On Fri, Jan 20, 2023 at 12:57 PM Robert Haas wrote: > It doesn't seem that way to me. What am I missing? In that case, the > problem was a DROP TRIGGER command waiting behind autovacuum's lock > and thus causing all new table locks to wait behind DROP TRIGGER's > lock request. But it does not soun

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-20 Thread Robert Haas
On Fri, Jan 20, 2023 at 3:43 PM Peter Geoghegan wrote: > The only reason why I'm using table age at all is because that's how > it works already, rightly or wrongly. If nothing else, t's pretty > clear that there is no theoretical or practical reason why it has to > be exactly the same table age a

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-20 Thread Peter Geoghegan
On Fri, Jan 20, 2023 at 5:47 AM Robert Haas wrote: > Yeah, this is a major reason why I'm very leery about changes in this > area. A lot of autovacuum behavior is emergent, in the sense that it > wasn't directly intended by whoever wrote the code. It's just a > consequence of other decisions that

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-20 Thread Robert Haas
On Thu, Jan 19, 2023 at 5:51 PM Andres Freund wrote: > I don't agree. But mainly my issue is that the devil you know (how this has > worked for a while) is preferrable to introducing an unknown quantity (your > patch that hasn't yet seen real world exposure). Yeah, this is a major reason why I'm

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Peter Geoghegan
On Thu, Jan 19, 2023 at 3:38 PM Andres Freund wrote: > Another version of this could be to integrate analyze.c's scan more closely > with vacuum all the time. It's a bit bonkers that we often sequentially read > blocks, evict them from shared buffers if we read them, just to then > afterwards do r

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Andres Freund
Hi, On 2023-01-19 15:10:38 -0800, Peter Geoghegan wrote: > On Thu, Jan 19, 2023 at 2:54 PM Andres Freund wrote: > > Yea. Hence my musing about potentially addressing this by choosing to visit > > additional blocks during the heap scan using vacuum's block sampling logic. > > I'd rather just inve

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Peter Geoghegan
On Thu, Jan 19, 2023 at 2:54 PM Andres Freund wrote: > Yea. Hence my musing about potentially addressing this by choosing to visit > additional blocks during the heap scan using vacuum's block sampling logic. I'd rather just invent a way for vacuumlazy.c to tell the top-level vacuum.c caller "I d

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Andres Freund
Hi, On 2023-01-19 13:22:28 -0800, Peter Geoghegan wrote: > On Thu, Jan 19, 2023 at 12:56 PM Andres Freund wrote: > > But in contrast to dead_tuples, where I think we can just stop analyze from > > updating it unless we crashed recently, I do think we need to update > > reltuples > > in vacuum. S

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Andres Freund
Hi, On 2023-01-19 13:36:41 -0800, Peter Geoghegan wrote: > On Thu, Jan 19, 2023 at 12:58 PM Andres Freund wrote: > > There's absolutely no guarantee that autoanalyze is triggered > > there. Particularly with repeated vacuums triggered due to an relfrozenxid > > age > > that can't be advanced tha

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Peter Geoghegan
On Thu, Jan 19, 2023 at 12:58 PM Andres Freund wrote: > There's absolutely no guarantee that autoanalyze is triggered > there. Particularly with repeated vacuums triggered due to an relfrozenxid age > that can't be advanced that very well might not happen within days on a large > relation. Argume

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Peter Geoghegan
On Thu, Jan 19, 2023 at 12:56 PM Andres Freund wrote: > > In other words, ANALYZE sometimes (but not always) produces wrong answers. > > For dead tuples, but not live tuples. > > In other words, VACUUM sometimes (but not always) produces wrong answers. > > For live tuples, but not badly so for de

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Andres Freund
Hi On 2023-01-18 19:26:22 -0800, Peter Geoghegan wrote: > On Wed, Jan 18, 2023 at 7:04 PM Andres Freund wrote: > > > You seem to be saying that it's a problem if we don't update reltuples > > > -- an estimate -- when less than 2% of the table is scanned by VACUUM. > > > But why? Why can't we just

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Andres Freund
Hi, On 2023-01-19 15:12:12 -0500, Robert Haas wrote: > On Wed, Jan 18, 2023 at 1:31 PM Peter Geoghegan wrote: > > pgstat_report_analyze() will totally override the > > tabentry->dead_tuples information that drives autovacuum.c, based on > > an estimate derived from a random sample -- which seems

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-19 Thread Robert Haas
On Wed, Jan 18, 2023 at 1:31 PM Peter Geoghegan wrote: > pgstat_report_analyze() will totally override the > tabentry->dead_tuples information that drives autovacuum.c, based on > an estimate derived from a random sample -- which seems to me to be an > approach that just doesn't have any sound the

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 7:04 PM Andres Freund wrote: > > You seem to be saying that it's a problem if we don't update reltuples > > -- an estimate -- when less than 2% of the table is scanned by VACUUM. > > But why? Why can't we just do nothing sometimes? I mean in general, > > leaving aside the h

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 18:21:33 -0800, Peter Geoghegan wrote: > On Wed, Jan 18, 2023 at 5:49 PM Andres Freund wrote: > > On 2023-01-18 15:28:19 -0800, Peter Geoghegan wrote: > > > Perhaps we should make vac_estimate_reltuples focus on the pages that > > > VACUUM newly set all-visible each time (not in

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 6:10 PM Andres Freund wrote: > > This creates an awkward but logical question, though: what if > > dead_tuples doesn't go down at all? What if VACUUM actually has to > > increase it, because VACUUM runs so slowly relative to the workload? > > Sure, that can happen - but it'

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 5:49 PM Andres Freund wrote: > On 2023-01-18 15:28:19 -0800, Peter Geoghegan wrote: > > Perhaps we should make vac_estimate_reltuples focus on the pages that > > VACUUM newly set all-visible each time (not including all-visible > > pages that got scanned despite being all-v

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 17:00:48 -0800, Peter Geoghegan wrote: > On Wed, Jan 18, 2023 at 4:37 PM Andres Freund wrote: > > I can, it should be just about trivial code-wise. A bit queasy about trying > > to > > forsee the potential consequences. > > That's always going to be true, though. > > > A some

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Fri, Jan 13, 2023 at 9:55 PM Andres Freund wrote: > How about a float autovacuum_no_auto_cancel_age where positive values are > treated as absolute values, and negative values are a multiple of > autovacuum_freeze_max_age? And where the "computed" age is capped at > vacuum_failsafe_age? A "fail

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 16:19:02 -0800, Peter Geoghegan wrote: > On Wed, Jan 18, 2023 at 4:02 PM Andres Freund wrote: > > vacuum-no reltuples/n_live_tupn_dead_tup > > 1 476 500 > > 2 2500077 500 > > 3 125018

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 4:37 PM Andres Freund wrote: > I can, it should be just about trivial code-wise. A bit queasy about trying to > forsee the potential consequences. That's always going to be true, though. > A somewhat related issue is that pgstat_report_vacuum() sets dead_tuples to > what

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 13:45:19 -0800, Peter Geoghegan wrote: > On Wed, Jan 18, 2023 at 1:08 PM Andres Freund wrote: > > I suggested nearby to only have ANALYZE dead_tuples it if there's been no > > [auto]vacuum since the stats entry was created. That allows recovering from > > stats resets, be it via

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 4:02 PM Andres Freund wrote: > vacuum-no reltuples/n_live_tupn_dead_tup > 1 476 500 > 2 2500077 500 > 3 1250184 500 > 4625266

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 14:37:20 -0800, Peter Geoghegan wrote: > On Wed, Jan 18, 2023 at 2:22 PM Andres Freund wrote: > > The problem with the change is here: > > > > /* > > * Okay, we've covered the corner cases. The normal calculation is > > to > > * convert the old measur

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 3:28 PM Peter Geoghegan wrote: > The problems in this area tend to be that vac_estimate_reltuples() > behaves as if it sees a random sample, when in fact it's far from > random -- it's the same scanned_pages as last time, and the ten other > times before that. That's a feat

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 2:37 PM Peter Geoghegan wrote: > Maybe you're right to be concerned to the degree that you're concerned > -- I'm not sure. I'm just adding what I see as important context. The problems in this area tend to be that vac_estimate_reltuples() behaves as if it sees a random sam

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 2:22 PM Andres Freund wrote: > The problem with the change is here: > > /* > * Okay, we've covered the corner cases. The normal calculation is to > * convert the old measurement to a density (tuples per page), then > * estimate the number

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 13:42:40 -0800, Andres Freund wrote: > The real point of change appears to be 10->11. > > There's a relevant looking difference in the vac_estimate_reltuples call: > 10: > /* now we can compute the new value for pg_class.reltuples */ > vacrelstats->new_rel_tuples = va

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 1:08 PM Andres Freund wrote: > I suggested nearby to only have ANALYZE dead_tuples it if there's been no > [auto]vacuum since the stats entry was created. That allows recovering from > stats resets, be it via crashes or explicitly. What do you think? I like that idea. It's

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 13:08:44 -0800, Andres Freund wrote: > One complicating factor is that VACUUM sometimes computes an incrementally > more bogus n_live_tup when it skips pages due to the VM, whereas ANALYZE > computes something sane. I unintentionally encountered one when I was trying > something

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 1:02 PM Peter Geoghegan wrote: > Some of what I'm proposing arguably amounts to deliberately adding a > bias. But that's not an unreasonable thing in itself. I think of it as > related to the bias-variance tradeoff, which is a concept that comes > up a lot in machine learni

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-18 12:15:17 -0800, Peter Geoghegan wrote: > On Wed, Jan 18, 2023 at 11:02 AM Robert Haas wrote: > > On Wed, Jan 18, 2023 at 1:31 PM Peter Geoghegan wrote: > > > pgstat_report_analyze() will totally override the > > > tabentry->dead_tuples information that drives autovacuum.c, base

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 12:44 PM Robert Haas wrote: > I don't know enough about the specifics of how this works to have an > intelligent opinion about how likely these particular ideas are to > work out. However, I think it's risky to look at estimates and try to > infer whether they are reliable.

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Robert Haas
On Wed, Jan 18, 2023 at 3:15 PM Peter Geoghegan wrote: > Suppose that we notice that its new > estimate for live_tuples approximately matches what the stats > subsystem already thought about live_tuples, while dead_tuples is far > far lower. We shouldn't be so credulous as to believe the new > dea

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Andres Freund
Hi, On 2023-01-17 12:08:01 -0800, Peter Geoghegan wrote: > > I think that's not the fault of relfrozenxid as a trigger, but that we > > simply > > don't keep enough other stats. We should imo at least keep track of: > > If you assume that there is chronic undercounting of dead tuples > (which I

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
On Wed, Jan 18, 2023 at 11:02 AM Robert Haas wrote: > On Wed, Jan 18, 2023 at 1:31 PM Peter Geoghegan wrote: > > pgstat_report_analyze() will totally override the > > tabentry->dead_tuples information that drives autovacuum.c, based on > > an estimate derived from a random sample -- which seems t

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Robert Haas
On Wed, Jan 18, 2023 at 1:31 PM Peter Geoghegan wrote: > pgstat_report_analyze() will totally override the > tabentry->dead_tuples information that drives autovacuum.c, based on > an estimate derived from a random sample -- which seems to me to be an > approach that just doesn't have any sound the

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Peter Geoghegan
. On Wed, Jan 18, 2023 at 7:54 AM Robert Haas wrote: > > It just fits: the dead tuples approach can sometimes be so > > completely wrong that even an alternative triggering condition based > > on something that is virtually unrelated to the thing we actually care > > about can do much better in p

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-18 Thread Robert Haas
On Tue, Jan 17, 2023 at 5:56 PM Peter Geoghegan wrote: > > Why do you think that? > > For the reasons I gave about statistics, random sampling, the central > limit theorem. All that stuff. This matches the experience of Andres. > And is obviously the only explanation behind the reliance on > antiw

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Peter Geoghegan
On Tue, Jan 17, 2023 at 2:11 PM Robert Haas wrote: > On Tue, Jan 17, 2023 at 3:08 PM Peter Geoghegan wrote: > > If you assume that there is chronic undercounting of dead tuples > > (which I think is very common), ... > > Why do you think that? For the reasons I gave about statistics, random samp

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Robert Haas
On Tue, Jan 17, 2023 at 3:08 PM Peter Geoghegan wrote: > If you assume that there is chronic undercounting of dead tuples > (which I think is very common), ... Why do you think that? > How many dead heap-only tuples are equivalent to one LP_DEAD item? > What about page-level concentrations, and

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Andres Freund
Hi, On 2023-01-17 14:57:27 -0500, Robert Haas wrote: > > In pg_class: > > > > - The number of all frozen pages, like we do for all-visible > > > > That'd give us a decent upper bound for the amount of work we need to do > > to > > increase relfrozenxid. It's important that this is crash safe

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Peter Geoghegan
On Tue, Jan 17, 2023 at 10:02 AM Andres Freund wrote: > Both you and Robert said this, and I have seen it be true, but typically not > for large high-throughput OLTP databases, where I found increasing > relfrozenxid to be important. Sure, there's probably some up/down through the > day / week, bu

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Robert Haas
On Tue, Jan 17, 2023 at 1:02 PM Andres Freund wrote: > On 2023-01-16 13:58:21 -0800, Peter Geoghegan wrote: > > On Fri, Jan 13, 2023 at 9:55 PM Andres Freund wrote: > > When I express skepticism of very high autovacuum_freeze_max_age > > settings, it's mostly just that I don't think that age(relf

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Andres Freund
Hi, On 2023-01-17 10:26:52 -0500, Robert Haas wrote: > On Mon, Jan 16, 2023 at 11:11 PM Peter Geoghegan wrote: > > On Mon, Jan 16, 2023 at 8:25 AM Robert Haas wrote: > > > I really dislike formulas like Min(freeze_max_age * 2, 1 billion). > > > That looks completely magical from a user perspecti

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Andres Freund
Hi, On 2023-01-16 13:58:21 -0800, Peter Geoghegan wrote: > On Fri, Jan 13, 2023 at 9:55 PM Andres Freund wrote: > When I express skepticism of very high autovacuum_freeze_max_age > settings, it's mostly just that I don't think that age(relfrozenxid) > is at all informative in the way that somethi

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-17 Thread Robert Haas
On Mon, Jan 16, 2023 at 11:11 PM Peter Geoghegan wrote: > On Mon, Jan 16, 2023 at 8:25 AM Robert Haas wrote: > > I really dislike formulas like Min(freeze_max_age * 2, 1 billion). > > That looks completely magical from a user perspective. Some users > > aren't going to understand autovacuum behav

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-16 Thread Peter Geoghegan
On Mon, Jan 16, 2023 at 8:25 AM Robert Haas wrote: > I really dislike formulas like Min(freeze_max_age * 2, 1 billion). > That looks completely magical from a user perspective. Some users > aren't going to understand autovacuum behavior at all. Some will, and > will be able to compare age(relfroze

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-16 Thread Peter Geoghegan
On Fri, Jan 13, 2023 at 9:55 PM Andres Freund wrote: > I think part of our difference around a high autovacuum_freeze_max_age are due > to things you're trying to address here - if no-auto-cancel is a separate > threshold from autovacuum_freeze_max_age, it's less problematic to set > autovacuum_fr

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-16 Thread Robert Haas
On Fri, Jan 13, 2023 at 9:09 PM Andres Freund wrote: > > > If I understand the patch correctly, we now have the following age based > > > thresholds for av: > > > > > > - force-enable autovacuum: > > > oldest_datfrozenxid + autovacuum_freeze_max_age < nextXid > > > - autovacuum based on age: > >

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-13 Thread Andres Freund
Hi, On 2023-01-13 19:39:41 -0800, Peter Geoghegan wrote: > On Fri, Jan 13, 2023 at 6:09 PM Andres Freund wrote: > > I don't think the split is right. There's too much in 0001 - it's basically > > introducing the terminology of 0002 already. Could you make it a much more > > minimal change? > > Ok

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-13 Thread Peter Geoghegan
On Fri, Jan 13, 2023 at 6:09 PM Andres Freund wrote: > I don't think the split is right. There's too much in 0001 - it's basically > introducing the terminology of 0002 already. Could you make it a much more > minimal change? Okay. I thought that you might say that. I really just wanted to show

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-13 Thread Andres Freund
Hi, On 2023-01-13 16:13:45 -0800, Peter Geoghegan wrote: > On Fri, Jan 13, 2023 at 2:00 PM Andres Freund wrote: > > I think it'd be a good idea to split off the part of the patch that > > introduces > > AutoVacType / adds logging for what triggered. That's independently useful, > > likely uncont

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-13 Thread Peter Geoghegan
On Fri, Jan 13, 2023 at 2:00 PM Andres Freund wrote: > On 2023-01-12 16:08:06 -0500, Robert Haas wrote: > > Normally, the XID age of a table never reaches autovacuum_freeze_max_age in > > the first place. > > That's not at all my experience. I often see it being the primary reason for > autovacuum

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-13 Thread Andres Freund
Hi, On 2023-01-12 16:08:06 -0500, Robert Haas wrote: > Normally, the XID age of a table never reaches autovacuum_freeze_max_age in > the first place. That's not at all my experience. I often see it being the primary reason for autovacuum vacuuming large tables on busy OLTP systems. Even without a

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-12 Thread Peter Geoghegan
On Thu, Jan 12, 2023 at 1:08 PM Robert Haas wrote: > I doubt it. Wiggle room that's based on the XID threshold being > different for one behavior vs. another can easily fail to produce any > benefit, because there's no guarantee that the autovacuum launcher > will ever try to launch a worker again

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-12 Thread Robert Haas
On Thu, Jan 12, 2023 at 2:22 PM Peter Geoghegan wrote: > All that I really want to do here is give an autovacuum that *can* be > auto cancelled *some* non-zero chance to succeed with these kinds of > tables. TRUNCATE completes immediately, so the AEL is no big deal. > Except when it's blocked behi

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-12 Thread Peter Geoghegan
On Thu, Jan 12, 2023 at 9:12 AM Robert Haas wrote: > I do agree that it's good to slowly increase the aggressiveness of > VACUUM as we get further behind, rather than having big behavior > changes all at once, but I think that should happen by smoothly > varying various parameters rather than by m

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-12 Thread Robert Haas
On Mon, Jan 9, 2023 at 8:40 PM Peter Geoghegan wrote: > That's not what the patch does. It doubles the time that the anti-wrap > no-autocancellation behaviors kick in, up to a maximum of 1 billion > XIDs/MXIDs. So it goes from autovacuum_freeze_max_age to > autovacuum_freeze_max_age x 2, without c

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-09 Thread Peter Geoghegan
On Mon, Jan 9, 2023 at 5:22 PM Andres Freund wrote: > I've also seen the inverse, with recent versions of postgres: Autovacuum can > only ever make progress if it's an anti-wraparound vacuum, because it'll > always get cancelled otherwise. I'm worried that substantially increasing the > time unti

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-09 Thread Andres Freund
Hi, On 2023-01-08 17:49:20 -0800, Peter Geoghegan wrote: > Teach autovacuum.c to launch "table age" autovacuums at the same point > that it previously triggered antiwraparound autovacuums. Antiwraparound > autovacuums are retained, but are only used as a true option of last > resort, when regular

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2023-01-08 Thread Peter Geoghegan
On Thu, Dec 29, 2022 at 7:01 PM Peter Geoghegan wrote: > Attached is v2, which is just to fix bitrot. Attached is v3. We no longer apply vacuum_failsafe_age when determining the cutoff for antiwraparound autovacuuming -- the new approach is a bit simpler. This is a fairly small change overall. N

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-12-29 Thread Peter Geoghegan
On Fri, Nov 25, 2022 at 2:47 PM Peter Geoghegan wrote: > Attached WIP patch invents the idea of a regular autovacuum that is > tasked with advancing relfrozenxid -- which is really just another > trigger criteria, reported on in the server log in its autovacuum > reports. Attached is v2, which is

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-11-27 Thread Peter Geoghegan
On Sun, Nov 27, 2022 at 8:54 AM Laurenz Albe wrote: > That's exactly what I was trying to debate. Wouldn't it make sense to > trigger VACUUM earlier so that it has a chance of being less heavy? > On the other hand, if there are not sufficiently many modifications > on the table to trigger autovac

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-11-27 Thread Laurenz Albe
On Sat, 2022-11-26 at 11:00 -0800, Peter Geoghegan wrote: > > I think that is a good idea. > > Wouldn't it make sense to trigger that at *half* > > "autovacuum_freeze_max_age"? > > That would be equivalent to what I've done here, provided you also > double the autovacuum_freeze_max_age setting.

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-11-26 Thread Peter Geoghegan
On Sat, Nov 26, 2022 at 9:58 AM Laurenz Albe wrote: > The target is a table that receives no DML at all, right? Sort of, but not really. The target is a table that doesn't get vacuumed for any other reason -- I don't make any claims beyond that one. It seems a little too optimistic to suppose th

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-11-26 Thread Laurenz Albe
On Fri, 2022-11-25 at 14:47 -0800, Peter Geoghegan wrote: > Attached WIP patch invents the idea of a regular autovacuum that is > tasked with advancing relfrozenxid -- which is really just another > trigger criteria, reported on in the server log in its autovacuum > reports. Of course we retain the

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-11-25 Thread Peter Geoghegan
On Mon, Oct 24, 2022 at 9:00 AM Peter Geoghegan wrote: > Maybe it could be broken out into a separate "autovacuum triggered by: > " line, seen only in the autovacuum log instrumentation (and not in > the similar report output by a manual VACUUM VERBOSE). When we still > end up "escalating" to an a

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-10-24 Thread Peter Geoghegan
On Mon, Oct 24, 2022 at 8:43 AM Jeff Davis wrote: > > Even if regular/early autovacuum had just one > > opportunity to run to completion, we'd already be much better off. > > By "opportunity", you mean that, regardless of configuration, the > cancellable autovacuum would at least start; though it

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-10-24 Thread Jeff Davis
On Mon, 2022-10-24 at 07:25 -0700, Peter Geoghegan wrote: > The really important thing is giving a regular/early autocancellable > autovacuum triggered by age(relfrozenxid) *some* opportunity to run. +1. That principle seems both reasonable from a system standpoint and understandable to a user. >

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-10-24 Thread Peter Geoghegan
On Sun, Oct 23, 2022 at 9:32 PM Jeff Davis wrote: > It's possible this would be easier for users to understand: one process > that does cleanup work over time in a way that minimizes interference; > and another process that activates in more urgent situations (perhaps > due to misconfiguration of

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-10-23 Thread Jeff Davis
On Fri, 2022-10-21 at 17:39 -0700, Peter Geoghegan wrote: > One small thought on the presentation/docs side of this: maybe it > would be better to invent a new kind of autovacuum It's possible this would be easier for users to understand: one process that does cleanup work over time in a way that

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-10-21 Thread Peter Geoghegan
On Thu, Oct 20, 2022 at 11:52 AM Peter Geoghegan wrote: > Leaving my patch series aside, I still don't think that it makes sense > to make it impossible to auto-cancel antiwraparound autovacuums, > across the board, regardless of the underlying table age. One small thought on the presentation/doc

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-10-20 Thread Peter Geoghegan
On Thu, Oct 20, 2022 at 11:09 AM Jeff Davis wrote: > The terminology is getting slightly confusing here: by > "antiwraparound", you mean that it's not skipping unfrozen pages, and > therefore is able to advance relfrozenxid. Whereas the > PROC_VACUUM_FOR_WRAPAROUND is the same thing, except done w

Re: Decoupling antiwraparound autovacuum from special rules around auto cancellation

2022-10-20 Thread Jeff Davis
On Wed, 2022-10-19 at 14:58 -0700, Peter Geoghegan wrote: > Why should the PROC_VACUUM_FOR_WRAPAROUND behavior happen on > *exactly* > the same timeline as the one used to launch an antiwraparound > autovacuum, though? The terminology is getting slightly confusing here: by "antiwraparound", you me