Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-25 16:34:21 +0900, Michael Paquier wrote: > diff --git a/src/include/access/xlogarchive.h > b/src/include/access/xlogarchive.h > index 299304703e..71c9b88165 100644 > --- a/src/include/access/xlogarchive.h > +++ b/src/include/access/xlogarchive.h > @@ -30,9 +30,45 @@ extern bool XL

Re: recovery modules

2023-01-27 Thread Michael Paquier
On Fri, Jan 27, 2023 at 04:09:39PM -0800, Andres Freund wrote: > I think it would be hard to write a good module that isn't just implementing > the existing commands without it. Needing to clean up archives and reacting to > the end of recovery seems a pretty core task. FWIW, recovery_end_command

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Tomas Vondra
On 1/27/23 22:33, Andres Freund wrote: > Hi, > > On 2023-01-27 21:45:16 +0100, Tomas Vondra wrote: >> On 1/27/23 08:18, Bharath Rupireddy wrote: I think my idea of only forcing to flush/wait an LSN some distance in the past would automatically achieve that? >>> >>> I'm sorry, I

Re: Syncrep and improving latency due to WAL throttling

2023-01-27 Thread Tomas Vondra
On 1/27/23 22:19, Andres Freund wrote: > Hi, > > On 2023-01-27 12:06:49 +0100, Jakub Wartak wrote: >> On Thu, Jan 26, 2023 at 4:49 PM Andres Freund wrote: >> >>> Huh? Why did you remove the GUC? >> >> After reading previous threads, my optimism level of getting it ever >> in shape of being widely

Re: Optimizing PostgreSQL with LLVM's PGO+LTO

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 18:28:16 -0500, Tom Lane wrote: > Andres Freund writes: > > I'm sure we have a few places that aren't that careful, but I would hope > > it's > > not a large number. Are you thinking of specific "patterns" we've repeated > > all > > over, or just a few cases you recall? > > I

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, >> + const char >> *lastRestartPointFileName); >> +typedef void (*RecoveryArchiveCleanupCB) (const

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
Hi, On 2023-01-18 14:51:32 -0500, Melanie Plageman wrote: > I only changed these few lines in type_sanity to be more correct; I > didn't change anything else in regress to actually exercise them (e.g. > ensuring a partitioned table is around when running type_sanity). It > might be worth moving ty

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Thomas Munro
The nearby thread about searching for uses of volatile reminded me: we can now drop a bunch of these in postmaster.c. The patch I originally wrote to do that as part of this series somehow morphed into an experimental patch to nuke all global variables[1], but of course we should at least drop the

Re: Small omission in type_sanity.sql

2023-01-27 Thread Tom Lane
Andres Freund writes: > Tom, is there a reason we run the various sanity tests early-ish in the > schedule? It does seem to reduce their effectiveness a bit... Originally, those tests were mainly needed to sanity-check the hand-maintained initial catalog data, so it made sense to run them early.

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Tom Lane
Thomas Munro writes: > The nearby thread about searching for uses of volatile reminded me: we > can now drop a bunch of these in postmaster.c. The patch I originally > wrote to do that as part of this series somehow morphed into an > experimental patch to nuke all global variables[1], but of cour

Re: recovery modules

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 04:23:19PM -0800, Andres Freund wrote: > >> +typedef bool (*RecoveryRestoreCB) (const char *file, const char *path, > >> + const char > >> *lastRestartPoin

Re: Using WaitEventSet in the postmaster

2023-01-27 Thread Andres Freund
Hi, On 2023-01-28 14:25:38 +1300, Thomas Munro wrote: > The nearby thread about searching for uses of volatile reminded me: we > can now drop a bunch of these in postmaster.c. The patch I originally > wrote to do that as part of this series somehow morphed into an > experimental patch to nuke all

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 3:55 PM Andres Freund wrote: > I wonder if it's quite detailed enough - we can be forced to do out-of-page > updates because we actually are out of space, or because we reach the max > number of line pointers we allow in a page. HOT pruning can't remove dead line > pointers

Re: Small omission in type_sanity.sql

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 20:39:04 -0500, Tom Lane wrote: > Andres Freund writes: > > Tom, is there a reason we run the various sanity tests early-ish in the > > schedule? It does seem to reduce their effectiveness a bit... > > Originally, those tests were mainly needed to sanity-check the > hand-mainta

Re: Something is wrong with wal_compression

2023-01-27 Thread Michael Paquier
On Fri, Jan 27, 2023 at 06:06:05AM +0100, Laurenz Albe wrote: > On Fri, 2023-01-27 at 16:15 +1300, Thomas Munro wrote: >> There is no >> doubt that the current situation is unacceptable, though, so maybe we >> really should just do it and make a faster one later.  Anyone else >> want to vote on thi

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 17:59:32 -0800, Peter Geoghegan wrote: > > I think this might cause some trouble for existing monitoring setups after > > an > > upgrade. Suddenly the number of updates will appear way lower than > > before... And if we end up eventually distinguishing between different > > rea

Re: Add n_tup_newpage_upd to pg_stat table views

2023-01-27 Thread Peter Geoghegan
On Fri, Jan 27, 2023 at 6:44 PM Andres Freund wrote: > I don't think that'd make it particularly easy to figure out how often > out-of-space causes non-HOT updates to go out of page, and how often it causes > potential HOT updates to go out of page. If you just have a single index, > it's not too

Re: bug: copy progress reporting of backends which run multiple COPYs

2023-01-27 Thread Michael Paquier
On Sat, Jan 21, 2023 at 02:45:40AM +0100, Matthias van de Meent wrote: > Let me think about it. I think it would work, but I'm not sure that's > a great option - it adds backend state that we would need to add > cleanup handles for. But then again, COPY ... TO could use TRIGGER to > trigger actual

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 16:15:08 +1300, Thomas Munro wrote: > It would be pg_current_xact_id() that would have to pay the cost of > the WAL flush, not pg_xact_status() itself, but yeah that's what the > patch does (with some optimisations). I guess one question is whether > there are any other reasonab

Re: Something is wrong with wal_compression

2023-01-27 Thread Maciek Sakrejda
On Fri, Jan 27, 2023, 18:58 Andres Freund wrote: > Hi, > > On 2023-01-27 16:15:08 +1300, Thomas Munro wrote: > > It would be pg_current_xact_id() that would have to pay the cost of > > the WAL flush, not pg_xact_status() itself, but yeah that's what the > > patch does (with some optimisations).

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: > On Fri, Jan 27, 2023 at 06:06:05AM +0100, Laurenz Albe wrote: > > On Fri, 2023-01-27 at 16:15 +1300, Thomas Munro wrote: > >> There is no > >> doubt that the current situation is unacceptable, though, so maybe we > >> really should just do

Re: Generating code for query jumbling through gen_node_support.pl

2023-01-27 Thread Michael Paquier
On Tue, Jan 24, 2023 at 03:57:56PM +0900, Michael Paquier wrote: > Still, my plan here is to enforce the loading of > pg_stat_statements with compute_query_id = regress and > utility_query_id = jumble (if needed) in a new buildfarm machine, Actually, about this specific point, I have been able to

Re: heapgettup refactoring

2023-01-27 Thread David Rowley
"On Wed, 25 Jan 2023 at 10:17, Melanie Plageman wrote: > I've added a comment but I didn't include the function name in it -- I > find it repetitive when the comments above functions do that -- however, > I'm not strongly attached to that. I think the general format for header comments is: /* *

Re: Something is wrong with wal_compression

2023-01-27 Thread Tom Lane
Andres Freund writes: > On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: >> FWIW, my vote goes for a more expensive but reliable function even in >> stable branches. > I very strenuously object. If we make txid_current() (by way of > pg_current_xact_id()) flush WAL, we'll cause outages. What

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 22:23:58 +1300, Thomas Munro wrote: > After 1000 make check loops, and 1000 make -C src/test/modules/test_shm_mq > check loops, on the same FBSD 13.1 machine as elver which has failed > like this once before, I haven't been able to reproduce this on > REL_12_STABLE. Did you use

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 22:39:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2023-01-28 11:38:50 +0900, Michael Paquier wrote: > >> FWIW, my vote goes for a more expensive but reliable function even in > >> stable branches. > > > I very strenuously object. If we make txid_current() (by way o

Re: Something is wrong with wal_compression

2023-01-27 Thread Andrey Borodin
On Fri, Jan 27, 2023 at 7:40 PM Tom Lane wrote: > > What are you using it for, that you don't care whether the answer > is trustworthy? > It's not trustworthy anyway. Xid wraparound might happen during reconnect. I suspect we can design a test that will show that it does not always show correct r

RE: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread houzj.f...@fujitsu.com
On Friday, January 27, 2023 8:16 PM Amit Kapila > > On Fri, Jan 27, 2023 at 3:45 PM vignesh C wrote: > > > > On Mon, 23 Jan 2023 at 10:52, Amit Kapila > wrote: > > > > > > IIRC, this is done to prevent concurrent drops of origin drop say by > > > exposed API pg_replication_origin_drop(). See th

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 19:49:17 -0800, Andres Freund wrote: > It's quite commonly used as part of trigger based replication tools (IIRC > that's its origin), monitoring, as part of client side logging, as part of > snapshot management. Forgot one: Queues. The way it's used for trigger based replicati

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 05:55:42PM -0800, Andres Freund wrote: > On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: >> I think it would be weird for the archive module and >> recovery module interfaces to look so different, but if that's okay, I can >> change it. > > I'm a bit sad about the archi

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Tom Lane
Andres Freund writes: > Except that you're saying that you hit this on elver (amd64), I think it'd be > interesting that we see the failure on an arm host, which has a less strict > memory order model than x86. I also saw it on florican, which is/was an i386 machine using clang and pretty standar

Re: Something is wrong with wal_compression

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 19:57:35 -0800, Andrey Borodin wrote: > On Fri, Jan 27, 2023 at 7:40 PM Tom Lane wrote: > > > > What are you using it for, that you don't care whether the answer > > is trustworthy? > > > > It's not trustworthy anyway. Xid wraparound might happen during > reconnect. I think th

RE: Time delayed LR (WAS Re: logical replication restrictions)

2023-01-27 Thread Takamichi Osumi (Fujitsu)
Hi, On Friday, January 27, 2023 8:00 PM Amit Kapila wrote: > On Fri, Jan 27, 2023 at 1:39 PM Takamichi Osumi (Fujitsu) > wrote: > > > > On Wednesday, January 25, 2023 11:24 PM I wrote: > > > Attached the updated v22. > > Hi, > > > > During self-review, I noticed some changes are required for som

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Andres Freund
Hi, On 2023-01-27 23:18:39 -0500, Tom Lane wrote: > I also saw it on florican, which is/was an i386 machine using clang and > pretty standard build options other than > 'CFLAGS' => '-msse2 -O2', > so I think this isn't too much about machine architecture or compiler > flags. Ah. Florican dr

Re: suppressing useless wakeups in logical/worker.c

2023-01-27 Thread Amit Kapila
On Fri, Jan 27, 2023 at 4:07 AM Tom Lane wrote: > > Thanks, pushed. > > Returning to the prior patch ... I don't much care for this: > > +/* Maybe there will be a free slot in a second... */ > +retry_time = TimestampTzPlusSeconds(now, 1); > +

Re: Something is wrong with wal_compression

2023-01-27 Thread Thomas Munro
On Sat, Jan 28, 2023 at 4:57 PM Andrey Borodin wrote: > It's not trustworthy anyway. Xid wraparound might happen during > reconnect. I suspect we can design a test that will show that it does > not always show correct results during xid->2pc conversion (there is a > point in time when xid is not i

Re: Something is wrong with wal_compression

2023-01-27 Thread Tom Lane
Thomas Munro writes: > Reading Andres's comments and realising how relatively young > txid_status() is compared to txid_current(), I'm now wondering if we > shouldn't just disclaim the whole thing in back branches. My thoughts were trending in that direction too. It's starting to sound like we a

Re: lockup in parallel hash join on dikkop (freebsd 14.0-current)

2023-01-27 Thread Thomas Munro
On Sat, Jan 28, 2023 at 4:42 PM Andres Freund wrote: > Did you use the same compiler / compilation flags as when elver hit it? > Clearly Tomas' case was with at least some optimizations enabled. I did use the same compiler version and optimisation level, clang llvmorg-13.0.0-0-gd7b669b3a303 at -O

Re: Deadlock between logrep apply worker and tablesync worker

2023-01-27 Thread Amit Kapila
On Sat, Jan 28, 2023 at 9:36 AM houzj.f...@fujitsu.com wrote: > > On Friday, January 27, 2023 8:16 PM Amit Kapila > > > > On Fri, Jan 27, 2023 at 3:45 PM vignesh C wrote: > > > > > > On Mon, 23 Jan 2023 at 10:52, Amit Kapila > > wrote: > > > > > > > > IIRC, this is done to prevent concurrent dr

Re: recovery modules

2023-01-27 Thread Nathan Bossart
On Fri, Jan 27, 2023 at 08:17:46PM -0800, Nathan Bossart wrote: > On Fri, Jan 27, 2023 at 05:55:42PM -0800, Andres Freund wrote: >> On 2023-01-27 16:59:10 -0800, Nathan Bossart wrote: >>> I think it would be weird for the archive module and >>> recovery module interfaces to look so different, but i

<    1   2