Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-05-14 Thread Alvaro Herrera
Thanks for these corrections -- I applied them and a few minor changes from myself, and pushed. Another set of eyes over the result would be most welcome. I hope we can close this now :-) -- Álvaro Herrera39°49'30"S 73°17'W "Those who use electric razors are infidels

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-05-13 Thread Justin Pryzby
On Thu, May 13, 2021 at 05:33:33PM -0400, Alvaro Herrera wrote: > +++ b/doc/src/sgml/maintenance.sgml > @@ -817,6 +817,11 @@ analyze threshold = analyze base threshold + analyze > scale factor * number of tu > > is compared to the total number of tuples inserted, updated, or deleted >

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-05-13 Thread Alvaro Herrera
With English fixes from Bruce. I think the note about autovacuum in the reference page for ANALYZE is a bit out of place, but not enough to make me move the whole paragraph elsewhere. Maybe we should try to do that sometime. -- Álvaro Herrera Valdivia, Chile Officer Krupke, what are we to

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-05-13 Thread Alvaro Herrera
New version, a bit more ambitious. I think it's better to describe behavior for partitioned tables ahead of inheritance. Also, in the ANALYZE reference page I split the topic in two: in one single paragraph we now describe what happens with manual analyze for partitioned tables and inheritance hi

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-05-11 Thread Alvaro Herrera
On 2021-Apr-23, Justin Pryzby wrote: > On Thu, Apr 22, 2021 at 12:43:46PM -0500, Justin Pryzby wrote: > > > > I think that should probably have been written down somewhere other than for > > the manual ANALYZE command, but in any case it seems to be outdated now. > > Starting with this Agreed,

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-05-11 Thread Michael Paquier
On Wed, Apr 21, 2021 at 07:06:49PM -0400, Alvaro Herrera wrote: > On 2021-Apr-09, Robert Haas wrote: >> Does this need to worry about new partitions getting attached to a >> partitioned table, or old ones getting detached? (Maybe it does >> already, not sure.) > > I was pinged because this is list

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-23 Thread Justin Pryzby
On Thu, Apr 22, 2021 at 12:43:46PM -0500, Justin Pryzby wrote: > Maybe the behavior should be documented, though. Actually, I thought the > pre-existing (non)behavior of autoanalyze would've been documented, and we'd > now update that. All I can find is this: > > https://www.postgresql.org/docs/

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-22 Thread Justin Pryzby
On Wed, Apr 21, 2021 at 07:06:49PM -0400, Alvaro Herrera wrote: > On 2021-Apr-09, Robert Haas wrote: > > Does this need to worry about new partitions getting attached to a > > partitioned table, or old ones getting detached? (Maybe it does > > already, not sure.) > > I was pinged because this is l

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-21 Thread yuzuko
Hi, Thank you for discussing this item. > I think we should treat ATTACH/ > DETACH/DROP handling as a further feature to be added in a future > release, not an open item to be fixed in the current one. > I agree with your opinion. > Now, if somebody sees a very trivial way to handle it, let's di

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-21 Thread Alvaro Herrera
On 2021-Apr-09, Robert Haas wrote: > Does this need to worry about new partitions getting attached to a > partitioned table, or old ones getting detached? (Maybe it does > already, not sure.) I was pinged because this is listed as an open item. I don't think it is one. Handling ATTACH/DETACH/DR

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-21 Thread Alvaro Herrera
On 2021-Apr-08, Tom Lane wrote: > BTW, another thing that looks like a race condition is the > extract_autovac_opts() call that is done a little bit earlier, > also without lock. I think this is actually safe, but it's ONLY > safe because we resisted the calls by certain people to add a > toast t

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-10 Thread Justin Pryzby
On Thu, Apr 08, 2021 at 04:11:49PM -0400, Alvaro Herrera wrote: > On 2021-Apr-08, Tom Lane wrote: > > > > So I tend to think that my initial instinct was the better direction: we > > > should not be doing any find_all_inheritors() here at all, but instead > > > rely on pg_class.reltuples to be set

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-10 Thread Tomas Vondra
On 4/10/21 12:29 AM, Justin Pryzby wrote: > On Fri, Apr 09, 2021 at 06:16:59PM -0400, Alvaro Herrera wrote: >> On 2021-Apr-09, Justin Pryzby wrote: >> >>> One data point: we do DETACH/ATTACH tables during normal operation, before >>> type-promoting ALTERs, to avoid worst-case disk use, and to avoid

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Tom Lane
Andres Freund writes: > On 2021-04-09 11:54:30 -0400, Alvaro Herrera wrote: >> Pushed now, thanks. > I assume this is also the likely explanation for / fix for: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2021-04-08%2016%3A03%3A03 > ==3500389== VALGRINDERROR-BEGIN > ==3500

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Alvaro Herrera
Hello On 2021-Apr-09, Andres Freund wrote: > I assume this is also the likely explanation for / fix for: > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2021-04-08%2016%3A03%3A03 > > ==3500389== VALGRINDERROR-BEGIN > ==3500389== Invalid read of size 8 > ==3500389==at 0x4EC

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Andres Freund
Hi, On 2021-04-09 11:54:30 -0400, Alvaro Herrera wrote: > On 2021-Apr-09, Tom Lane wrote: > > > Could we get this pushed sooner rather than later? The buildfarm > > is showing a wide variety of intermittent failures on HEAD, and it's > > hard to tell how many of them trace to this one bug. > > Pu

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Justin Pryzby
On Fri, Apr 09, 2021 at 06:16:59PM -0400, Alvaro Herrera wrote: > On 2021-Apr-09, Justin Pryzby wrote: > > > One data point: we do DETACH/ATTACH tables during normal operation, before > > type-promoting ALTERs, to avoid worst-case disk use, and to avoid locking > > the > > table for a long time.

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Alvaro Herrera
On 2021-Apr-09, Justin Pryzby wrote: > One data point: we do DETACH/ATTACH tables during normal operation, before > type-promoting ALTERs, to avoid worst-case disk use, and to avoid locking the > table for a long time. It'd be undesirable (but maybe of no great > consequence) > to trigger an ALT

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Tomas Vondra
On 4/9/21 11:45 PM, Justin Pryzby wrote: > On Fri, Apr 09, 2021 at 05:31:55PM -0400, Alvaro Herrera wrote: >> On 2021-Apr-09, Robert Haas wrote: >> >>> Does this need to worry about new partitions getting attached to a >>> partitioned table, or old ones getting detached? (Maybe it does >>> alrea

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Justin Pryzby
On Fri, Apr 09, 2021 at 05:31:55PM -0400, Alvaro Herrera wrote: > On 2021-Apr-09, Robert Haas wrote: > > > Does this need to worry about new partitions getting attached to a > > partitioned table, or old ones getting detached? (Maybe it does > > already, not sure.) > > Good question. It does not

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Alvaro Herrera
On 2021-Apr-09, Robert Haas wrote: > Does this need to worry about new partitions getting attached to a > partitioned table, or old ones getting detached? (Maybe it does > already, not sure.) Good question. It does not. I suppose you could just let that happen automatically -- I mean, next time

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Robert Haas
On Fri, Apr 9, 2021 at 11:54 AM Alvaro Herrera wrote: > On 2021-Apr-09, Tom Lane wrote: > > Could we get this pushed sooner rather than later? The buildfarm > > is showing a wide variety of intermittent failures on HEAD, and it's > > hard to tell how many of them trace to this one bug. > > Pushed

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Alvaro Herrera
On 2021-Apr-09, Tom Lane wrote: > Could we get this pushed sooner rather than later? The buildfarm > is showing a wide variety of intermittent failures on HEAD, and it's > hard to tell how many of them trace to this one bug. Pushed now, thanks. -- Álvaro Herrera Valdivia, Chile "Digital

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-09 Thread Tom Lane
Could we get this pushed sooner rather than later? The buildfarm is showing a wide variety of intermittent failures on HEAD, and it's hard to tell how many of them trace to this one bug. regards, tom lane

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-08 Thread Alvaro Herrera
On 2021-Apr-08, Zhihong Yu wrote: > Hi, > Within truncate_update_partedrel_stats(), dirty is declared within the loop. > + if (rd_rel->reltuples != 0) > + { > ... > + if (dirty) > > The two if blocks can be merged. The variable dirty can be dropped. Hi, thanks for reviewing. Y

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-08 Thread Zhihong Yu
On Thu, Apr 8, 2021 at 1:12 PM Alvaro Herrera wrote: > On 2021-Apr-08, Tom Lane wrote: > > > > So I tend to think that my initial instinct was the better direction: > we > > > should not be doing any find_all_inheritors() here at all, but instead > > > rely on pg_class.reltuples to be set for the

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-08 Thread Alvaro Herrera
On 2021-Apr-08, Tom Lane wrote: > > So I tend to think that my initial instinct was the better direction: we > > should not be doing any find_all_inheritors() here at all, but instead > > rely on pg_class.reltuples to be set for the partitioned table. > > +1 This patch does that. -- Álvaro Her

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-08 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Apr-08, Tom Lane wrote: >> Yeah. I hit this on another machine that isn't using EXEC_BACKEND, >> and I concur it looks more like a race condition. I think the problem >> is that autovacuum is calling find_all_inheritors() on a relation it >> has no lock on, contr

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-08 Thread Alvaro Herrera
On 2021-Apr-08, Tom Lane wrote: > Yeah. I hit this on another machine that isn't using EXEC_BACKEND, > and I concur it looks more like a race condition. I think the problem > is that autovacuum is calling find_all_inheritors() on a relation it > has no lock on, contrary to that function's API sp

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-08 Thread Tom Lane
Alvaro Herrera writes: > On 2021-Apr-08, Tom Lane wrote: >> Looks like this has issues under EXEC_BACKEND: >> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=culicidae&dt=2021-04-08%2005%3A50%3A08 > Hmm, I couldn't reproduce this under EXEC_BACKEND or otherwise, but I > think this is unre

Re: pgsql: autovacuum: handle analyze for partitioned tables

2021-04-08 Thread Alvaro Herrera
On 2021-Apr-08, Tom Lane wrote: > Alvaro Herrera writes: > > autovacuum: handle analyze for partitioned tables > > Looks like this has issues under EXEC_BACKEND: > > https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=culicidae&dt=2021-04-08%2005%3A50%3A08 Hmm, I couldn't reproduce this un