Re: adding 'zstd' as a compression algorithm

2022-02-15 Thread Andres Freund
Hi, On 2022-02-16 09:58:44 +0900, Michael Paquier wrote: > On Tue, Feb 15, 2022 at 01:42:55PM -0800, Andres Freund wrote: > > I think well before removing stuff we should default to decent compression > > algorithms. E.g. -Z/--compress should probably not continue to use gzip if > > better things

Small and unaffected typo in pg_logical_slot_get_changes_guts()

2022-02-15 Thread Kasahara Tatsuhito
Hi, I noticed the small typo in pg_logical_slot_get_changes_guts(). == --- a/src/backend/replication/logical/logicalfuncs.c +++ b/src/backend/replication/logical/logicalfuncs.c @@ -295,7 +295,7 @@ pg_logical_slot_get_changes_guts(FunctionCallInfo fcinfo, bool c

Re: adding 'zstd' as a compression algorithm

2022-02-15 Thread Michael Paquier
On Tue, Feb 15, 2022 at 02:33:32PM -0600, Justin Pryzby wrote: > What I mean is that any patches which *use* the zstd support should update > those for completeness. > > doc/src/sgml/config.sgml | 14 +- > doc/src/sgml/install-windows.sgml | 2 +- > d

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-02-15 Thread Thomas Munro
On Wed, Feb 16, 2022 at 3:09 PM Thomas Munro wrote: > I tried adding another log line so I could see the unlink() happening, > but with that it doesn't fail (though I'm still trying). We may be in > heisenbug territory. This time I was luckier: https://api.cirrus-ci.com/v1/artifact/task/4915815

Re: Small and unaffected typo in pg_logical_slot_get_changes_guts()

2022-02-15 Thread Michael Paquier
On Wed, Feb 16, 2022 at 11:27:58AM +0900, Kasahara Tatsuhito wrote: > Since commit dd04e958c8b03c0f0512497651678c7816af3198, > tuplestore_donestoring() seems to be left only for compatibility, so > it looks like it can be removed from the core code, except for the > header (tuplestore.h). FWIW, it

Re: adding 'zstd' as a compression algorithm

2022-02-15 Thread Andres Freund
Hi,m On 2022-02-15 20:56:15 -0500, Tom Lane wrote: > Maybe we have a bit more flexibility for TOAST, not sure. It'd be nice to at least add it as an option for initdb. Afaics there's no way to change the default at that point. initdb itself is measurably faster. Although sadly it's a bigger diffe

Re: [Proposal] Add foreign-server health checks infrastructure

2022-02-15 Thread Kyotaro Horiguchi
At Tue, 1 Feb 2022 23:51:54 +0900, Fujii Masao wrote in > This logic sounds complicated to me. I'm afraid that FDW developers > may a bit easily misunderstand the logic and make the bug in their > FDW. > Isn't it simpler to just disable the timeout in core whenever the > transaction ends whethe

Re: [PATCH] Fix out-of-bouds access (src/common/wchar.c)

2022-02-15 Thread Kyotaro Horiguchi
At Tue, 15 Feb 2022 09:17:34 -0300, Ranier Vilela wrote in > Per Coverity. Thanks for the source:) > Like the function pg_encoding_max_length_sql > (src/backend/utils/mb/mbutils.c) > Only assertion is insufficient to avoid accessing array out-of-bounds. > > This bug is live according Coverity

Re: initdb / bootstrap design

2022-02-15 Thread John Naylor
On Wed, Feb 16, 2022 at 9:12 AM Andres Freund wrote: > To me the division of labor between initdb and bootstrap doesn't make much > sense anymore: [...] > I don't plan to work on this immediately, but I thought it's worth bringing up > anyway. Added TODO item "Rationalize division of labor betwe

RE: logical replication empty transactions

2022-02-15 Thread osumi.takami...@fujitsu.com
Hi I'll quote one other remaining discussion of this thread again to invoke more attentions from the community. On Friday, August 13, 2021 8:01 PM Ajin Cherian wrote: > On Mon, Aug 2, 2021 at 7:20 PM Amit Kapila > wrote: > > Few other miscellaneous comments: > > 1. > > static void > > pgoutput

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-02-15 Thread Bharath Rupireddy
On Wed, Feb 16, 2022 at 1:57 AM Robert Haas wrote: > > On Tue, Feb 15, 2022 at 2:31 PM Bharath Rupireddy > wrote: > > > +/* > > > + * Verify the authenticity of the given raw WAL record. > > > + */ > > > +Datum > > > +pg_verify_raw_wal_record(PG_FUNCTION_ARGS) > > > +{ > > > > > > > > > Do we rea

Re: pg_walinspect - a new extension to get raw WAL data and WAL stats

2022-02-15 Thread Ashutosh Sharma
On Wed, Feb 16, 2022 at 1:01 AM Bharath Rupireddy wrote: > > On Mon, Feb 14, 2022 at 8:32 PM Ashutosh Sharma wrote: > > > > Here are few comments: > > Thanks for reviewing the patches. > > > +/* > > + * Verify the authenticity of the given raw WAL record. > > + */ > > +Datum > > +pg_verify_raw_wa

Re: Observability in Postgres

2022-02-15 Thread Stephan Doliov
I am curious what you mean by standard metrics format? I am all for standards-based but what are those in the case of DBs. For environments where O11y matters a lot, I think the challenge lies in mapping specific query executions back to system characteristics. I am just thinking aloud as a newbie

Re: pg_upgrade verbosity when redirecting output to log file

2022-02-15 Thread Thomas Munro
On Tue, Jan 11, 2022 at 4:42 AM Bruce Momjian wrote: > On Sun, Jan 9, 2022 at 10:39:58PM -0800, Andres Freund wrote: > > On 2022-01-10 01:14:32 -0500, Tom Lane wrote: > > > I think I'd vote for just nuking that output altogether. > > > It seems of very dubious value. > > > > It seems worthwhile i

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-02-15 Thread Andres Freund
Hi, On 2022-02-16 15:09:15 +1300, Thomas Munro wrote: > I pushed a hack to log the name of the file that's getting in the way: FWIW, I think we should just have that in core. Right now it's just pointlessly hard to debug. I think just reporting the first file would be good enough... Greetings,

Re: Small and unaffected typo in pg_logical_slot_get_changes_guts()

2022-02-15 Thread Kasahara Tatsuhito
Hi ! 2022年2月16日(水) 11:42 Michael Paquier : > > On Wed, Feb 16, 2022 at 11:27:58AM +0900, Kasahara Tatsuhito wrote: > > Since commit dd04e958c8b03c0f0512497651678c7816af3198, > > tuplestore_donestoring() seems to be left only for compatibility, so > > it looks like it can be removed from the core c

Re: Rewriting the test of pg_upgrade as a TAP test - take three - remastered set

2022-02-15 Thread Michael Paquier
On Tue, Feb 15, 2022 at 01:02:41PM +0900, Michael Paquier wrote: > Hmm. At the end of the day, I am wondering whether we should not give > up entirely on the concept of running the regression tests on older > branches in the TAP script of a newer branch. pg_regress needs to > come from the old so

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-02-15 Thread Nathan Bossart
On Wed, Feb 16, 2022 at 03:34:08PM +1300, Thomas Munro wrote: > So it's not getting unlinked until the *next* checkpoint cycle. I > don't know why. On my machine (5.11.0-43), it looks like the test starts failing after cc50080. That commit adjusted some regression tests, so I'm assuming it's not

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-02-15 Thread Andres Freund
Hi, On 2022-02-16 15:34:08 +1300, Thomas Munro wrote: > So it's not getting unlinked until the *next* checkpoint cycle. I > don't know why. It might be helpful to know what the relfilenode maps to, so we know the operations done to it. Perhaps logging in heap_create() and RelationSetNewRelfileno

Proposal for internal Numeric to Uint64 conversion function.

2022-02-15 Thread Amul Sul
Hi, Currently, numeric_pg_lsn is the only one that accepts the Numeric value and converts to uint64 and that is the reason all the type conversion code is embedded into it. I think it would be helpful if that numeric-to-uint64 conversion code is extracted as a separate function so that any other

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-02-15 Thread Andres Freund
Hi, On 2022-02-15 20:59:11 -0800, Nathan Bossart wrote: > On Wed, Feb 16, 2022 at 03:34:08PM +1300, Thomas Munro wrote: > > So it's not getting unlinked until the *next* checkpoint cycle. I > > don't know why. > > On my machine (5.11.0-43), it looks like the test starts failing after > cc50080.

Re: support for CREATE MODULE

2022-02-15 Thread Pavel Stehule
Hi > Yes, anything a user may want to do with modules is likely possible to > do already with schemas. But just because it is possible doesn't mean > it is not tedious and awkward because of the mechanisms available to do > them now. And I would advocate for more expressive constructs to enable >

Re: Small and unaffected typo in pg_logical_slot_get_changes_guts()

2022-02-15 Thread Justin Pryzby
On Wed, Feb 16, 2022 at 11:27:58AM +0900, Kasahara Tatsuhito wrote: > - tuplestore_donestoring(tupstore); > + tuplestore_donestoring(p->tupstore); Melanie's tuplestore patch also removes the bogus line. https://www.postgresql.org/message-id/20220106005748.GT14051%40tels

Re: Race conditions in 019_replslot_limit.pl

2022-02-15 Thread Kyotaro Horiguchi
At Tue, 15 Feb 2022 15:51:57 -0800, Andres Freund wrote in > I think what happened is that there was no WAL to receive between the start of > the primary and the $node_primary3->wait_for_catchup($node_standby3); > > Because the slot is created without reserving WAL that allows the primary to > r

Re: Small and unaffected typo in pg_logical_slot_get_changes_guts()

2022-02-15 Thread Michael Paquier
On Tue, Feb 15, 2022 at 11:23:29PM -0600, Justin Pryzby wrote: > On Wed, Feb 16, 2022 at 11:27:58AM +0900, Kasahara Tatsuhito wrote: >> - tuplestore_donestoring(tupstore); >> + tuplestore_donestoring(p->tupstore); > > Melanie's tuplestore patch also removes the bogus li

Re: make tuplestore helper function

2022-02-15 Thread Michael Paquier
On Tue, Jan 11, 2022 at 10:19:33AM -0500, Melanie Plageman wrote: > On Wed, Jan 5, 2022 at 7:57 PM Justin Pryzby wrote: >> I'd leave it for input from a committer about those: >> >> - remove tuplestore_donestoring() calls ? This part has been raised on a different thread, as of: https://www.post

Re: Observability in Postgres

2022-02-15 Thread Greg Stark
On Tue, 15 Feb 2022 at 22:48, Stephan Doliov wrote: > > I am curious what you mean by standard metrics format? I am all for > standards-based but what are those in the case of DBs. For environments where > O11y matters a lot, I think the challenge lies in mapping specific query > executions bac

Re: make tuplestore helper function

2022-02-15 Thread Justin Pryzby
On Tue, Jan 11, 2022 at 10:19:33AM -0500, Melanie Plageman wrote: > > If the expectedDesc NULL check were an 0001 patch, then 0002 (the main > > patch) > > would be even easier to review. Only foreign.c is different. > > I'll wait to do that if preferred by committer. > Are you imagining that pa

Re: Race conditions in 019_replslot_limit.pl

2022-02-15 Thread Masahiko Sawada
On Wed, Feb 16, 2022 at 2:26 PM Kyotaro Horiguchi wrote: > > At Tue, 15 Feb 2022 15:51:57 -0800, Andres Freund wrote > in > > I think what happened is that there was no WAL to receive between the start > > of > > the primary and the $node_primary3->wait_for_catchup($node_standby3); > > > > Beca

Re: Race conditions in 019_replslot_limit.pl

2022-02-15 Thread Kyotaro Horiguchi
At Wed, 16 Feb 2022 14:26:37 +0900 (JST), Kyotaro Horiguchi wrote in > Agreed. Doing this att all slot creation seems fine. Done in the attached. The first slot is deliberately created unreserved so I changed the code to re-create the slot with "reserved" before taking backup. > > Even though

Re: Race conditions in 019_replslot_limit.pl

2022-02-15 Thread Kyotaro Horiguchi
At Wed, 16 Feb 2022 14:58:23 +0900, Masahiko Sawada wrote in > On Wed, Feb 16, 2022 at 2:26 PM Kyotaro Horiguchi > wrote: > > > > At Tue, 15 Feb 2022 15:51:57 -0800, Andres Freund > > wrote in > > > I guess it's conceivable that the backend was still working through > > > process > > > shutd

Re: Adding CI to our tree

2022-02-15 Thread Justin Pryzby
On Sun, Feb 13, 2022 at 01:53:19PM -0800, Andres Freund wrote: > Hi, > > On 2022-02-13 15:31:20 -0600, Justin Pryzby wrote: > > Oh - I suppose you're right. That's an unfortunate consequence of running a > > single prove instance without chdir. > > I don't think it's chdir that's relevant (that

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-02-15 Thread Nathan Bossart
On Tue, Feb 15, 2022 at 09:06:45PM -0800, Andres Freund wrote: > On 2022-02-15 20:59:11 -0800, Nathan Bossart wrote: >> On Wed, Feb 16, 2022 at 03:34:08PM +1300, Thomas Munro wrote: >> > So it's not getting unlinked until the *next* checkpoint cycle. I >> > don't know why. >> >> On my machine (5.

Re: Race conditions in 019_replslot_limit.pl

2022-02-15 Thread Kyotaro Horiguchi
At Wed, 16 Feb 2022 14:58:23 +0900, Masahiko Sawada wrote in > Or it's possible that the process took a time to clean up the > temporary replication slot? Checkpointer may take ReplicationSlotControlLock. Dead lock between ReplicationSlotCleanup and InvalidateObsoleteReplicationSlots happened?

Re: Adding CI to our tree

2022-02-15 Thread Andres Freund
Hi, On February 15, 2022 10:12:36 PM PST, Justin Pryzby wrote: >On Sun, Feb 13, 2022 at 01:53:19PM -0800, Andres Freund wrote: >> Hi, >> >> On 2022-02-13 15:31:20 -0600, Justin Pryzby wrote: >> > Oh - I suppose you're right. That's an unfortunate consequence of running >> > a >> > single prov

Re: USE_BARRIER_SMGRRELEASE on Linux?

2022-02-15 Thread Nathan Bossart
On Tue, Feb 15, 2022 at 10:14:04PM -0800, Nathan Bossart wrote: > It looks like register_unlink_segment() is called prior to the checkpoint, > but the checkpointer is not calling RememberSyncRequest() until after > SyncPreCheckpoint(). This means that the requests are registered with the > next ch

Re: Observability in Postgres

2022-02-15 Thread Greg Stark
On Tue, 15 Feb 2022 at 17:37, Magnus Hagander wrote: > > On Tue, Feb 15, 2022 at 11:24 PM Greg Stark wrote: > > > > On Tue, 15 Feb 2022 at 16:43, Magnus Hagander wrote: > > I really don't see the problem with having the monitoring on a different port. > > I *do* see the problem with having a dif

Re: Race conditions in 019_replslot_limit.pl

2022-02-15 Thread Kyotaro Horiguchi
At Wed, 16 Feb 2022 15:22:27 +0900 (JST), Kyotaro Horiguchi wrote in > At Wed, 16 Feb 2022 14:58:23 +0900, Masahiko Sawada > wrote in > > Or it's possible that the process took a time to clean up the > > temporary replication slot? > > Checkpointer may take ReplicationSlotControlLock. Dead l

Re: Observability in Postgres

2022-02-15 Thread Julien Rouhaud
Hi, On Mon, Feb 14, 2022 at 03:15:14PM -0500, Greg Stark wrote: > > [...] > 2) SQL connections are tied to specific databases within a cluster. > Making it hard to get data for all your databases if you have more > than one. The exporter needs to reconnect to each database. > > 3) The exporter nee

Re: Trap errors from streaming child in pg_basebackup to exit early

2022-02-15 Thread Michael Paquier
On Wed, Sep 29, 2021 at 01:18:40PM +0200, Daniel Gustafsson wrote: > So there is one mention of a background WAL receiver already in there, but > it's > pretty inconsistent as to what we call it. For now I've changed the messaging > in this patch to say "background process", leaving making this a

some aspects of our qsort might not be ideal

2022-02-15 Thread John Naylor
While reviewing Thomas Munro's patchset to consider expanding the uses of specialized qsort [1], I wondered about some aspects of the current qsort implementation. For background, ours is based on Bentley & McIlroy "Engineering a Sort Function" [2], which is a classic paper worth studying. 1) the

Re: BufferAlloc: don't take two simultaneous locks

2022-02-15 Thread Yura Sokolov
В Вс, 06/02/2022 в 19:34 +0300, Michail Nikolaev пишет: > Hello, Yura. > > A one additional moment: > > > 1332: Assert((oldFlags & (BM_PIN_COUNT_WAITER | BM_IO_IN_PROGRESS)) == 0); > > 1333: CLEAR_BUFFERTAG(buf->tag); > > 1334: buf_state &= ~(BUF_FLAG_MASK | BUF_USAGECOUNT_MASK); > > 1335: Unlock

Re: some aspects of our qsort might not be ideal

2022-02-15 Thread John Naylor
> [3] > https://www.postgresql.org/message-id/flat/87F42982BF2B434F831FCEF4C45FC33E5BD369DF%40EXCHANGE.corporate.connx.com#e69718293c45d89555020bd0922ad055 The top of that thread is where I meant to point to: https://www.postgresql.org/message-id/flat/CAEYLb_Xn4-6f1ofsf2qduf24dDCVHbQidt7JPpdL_Ri

Re: BufferAlloc: don't take two simultaneous locks

2022-02-15 Thread Yura Sokolov
Hello, all. I thought about patch simplification, and tested version without BufTable and dynahash api change at all. It performs suprisingly well. It is just a bit worse than v1 since there is more contention around dynahash's freelist, but most of improvement remains. I'll finish benchmarking

Re: some aspects of our qsort might not be ideal

2022-02-15 Thread John Naylor
> This way, we *dynamically* > decide to optimistically start an insertion sort, in the hopes that it > will occasionally prevent a recursion, and worst case the input is > slightly more sorted for the next recursion. I should mention one detail -- we put a limit on how many iterations we attempt

Re: Split xlog.c

2022-02-15 Thread Heikki Linnakangas
On 14/02/2022 11:36, Heikki Linnakangas wrote: On 27/01/2022 08:34, Michael Paquier wrote: On Tue, Jan 25, 2022 at 12:12:40PM +0200, Heikki Linnakangas wrote: In last round of review, I spotted one bug: I had mixed up the meaning of EndOfLogTLI. It is the TLI in the *filename* of the WAL segmen

<    1   2