RE: Skipping logical replication transactions on subscriber side

2022-01-17 Thread osumi.takami...@fujitsu.com
On Monday, January 17, 2022 3:18 PM Masahiko Sawada wrote: > I've attached an updated patch. Please review it. Hi, thank you for sharing a new patch. Few comments on the v6. (1) doc/src/sgml/ref/alter_subscription.sgml + resort. This option has no effect on the transaction that is already

Re: Proposal: More structured logging

2022-01-17 Thread Ronan Dunklau
Le samedi 15 janvier 2022, 07:09:59 CET Julien Rouhaud a écrit : > Hi, > > On Tue, Jan 11, 2022 at 11:05:26AM +0100, Ronan Dunklau wrote: > > Done, and I added anoher commit per your suggestion to add this comment. > > The cfbot reports that the patchset doesn't apply anymore: > > http://cfbot.c

Re: standby recovery fails (tablespace related) (tentative patch and discussion)

2022-01-17 Thread Kyotaro Horiguchi
At Sun, 16 Jan 2022 12:43:03 +0800, Julien Rouhaud wrote in > Hi, > > On Fri, Dec 24, 2021 at 07:21:59PM +0900, Kyotaro Horiguchi wrote: > > Just a complaint.. > > > > At Fri, 12 Nov 2021 16:43:27 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > "" on Japanese (CP-932) environment. I didn'

Re: Partial aggregates pushdown

2022-01-17 Thread Zhihong Yu
On Sun, Jan 16, 2022 at 11:47 PM Alexander Pyhalov wrote: > Julien Rouhaud писал 2022-01-14 15:16: > > Hi, > > > > On Mon, Nov 15, 2021 at 04:01:51PM +0300, Alexander Pyhalov wrote: > >> > >> I've updated patch - removed catversion dump. > > > > This version of the patchset doesn't apply anymore:

Re: Extensible Rmgr for Table AMs

2022-01-17 Thread Jeff Davis
On Mon, 2021-11-08 at 15:36 -0800, Jeff Davis wrote: > The attached patch (against v14, so it's easier to test columnar) is > somewhat like a simplified version of [3] combined with refactoring > to > make decoding a part of the rmgr. New patches attached (v3). Essentially the content as v2, but s

Re: Logical insert/update/delete WAL records for custom table AMs

2022-01-17 Thread Jeff Davis
On Wed, 2022-01-05 at 20:19 +, Simon Riggs wrote: > I spoke with Jeff in detail about this patch in NYC Dec 21, and I now > think it is worth pursuing. It seems much more likely that this would > be acceptable than fully extensible rmgr. Thank you. I had some conversations with others who felt

Re: row filtering for logical replication

2022-01-17 Thread Amit Kapila
On Sat, Jan 15, 2022 at 12:00 PM houzj.f...@fujitsu.com wrote: > > On Friday, January 14, 2022 7:31 PM Amit Kapila > wrote: > > On Thu, Jan 13, 2022 at 6:46 PM houzj.f...@fujitsu.com > > > > 9. > > --- a/src/backend/replication/logical/proto.c > > +++ b/src/backend/replication/logical/proto.c >

Re: New developer papercut - Makefile references INSTALL

2022-01-17 Thread Daniel Gustafsson
> On 17 Jan 2022, at 02:11, Tim McNamara wrote: > > Hello, > > I encountered a minor road bump when checking out the pg source today. The > Makefile's all target includes the following help message if GNUmakefile > isn't available: > > echo "You need to run the 'configure' program first. Se

Re: SLRUs in the main buffer pool, redux

2022-01-17 Thread Heikki Linnakangas
On 13/01/2022 15:59, Thomas Munro wrote: Hi, I was re-reviewing the proposed batch of GUCs for controlling the SLRU cache sizes[1], and I couldn't resist sketching out $SUBJECT as an obvious alternative. This patch is highly experimental and full of unresolved bits and pieces (see below for som

Re: New developer papercut - Makefile references INSTALL

2022-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2022 at 11:17 AM Daniel Gustafsson wrote: > > > On 17 Jan 2022, at 02:11, Tim McNamara wrote: > > > > Hello, > > > > I encountered a minor road bump when checking out the pg source today. The > > Makefile's all target includes the following help message if GNUmakefile > > isn't

Re: pg_replslotdata - a tool for displaying replication slot information

2022-01-17 Thread Bharath Rupireddy
On Sat, Jan 15, 2022 at 2:20 PM Julien Rouhaud wrote: > > Bharath, perhaps you should maintain this outside of core PostgreSQL > > for now. If some compelling use-cases ever surface that make it seem > > worth the added maintenance burden, this thread could probably be > > revisited. > > Ironical

Re: Asymmetric partition-wise JOIN

2022-01-17 Thread Alexander Pyhalov
Andrey Lepikhov писал 2021-09-15 09:31: On 14/9/21 11:37, Andrey V. Lepikhov wrote: Thank you for this good catch! The problem was in the adjust_child_relids_multilevel routine. The tmp_result variable sometimes points to original required_outer. This patch adds new ways which optimizer can gen

Re: Per-table storage parameters for TableAM/IndexAM extensions

2022-01-17 Thread Jelte Fennema
Big +1, this is a great addition! I think it would be very useful if there were some tests for this new feature. Something similar to the tests for storage parameters for index AMs in src/test/modules/dummy_index_am. Apart from that I think the documentation for table storage parameters needs to

Re: row filtering for logical replication

2022-01-17 Thread Amit Kapila
On Mon, Jan 17, 2022 at 3:19 PM Amit Kapila wrote: > > Some other comments: > == > Few more comments: == 1. +pgoutput_row_filter_init_expr(Node *rfnode) +{ + ExprState *exprstate; + Expr*expr; + + /* + * This is the same code as ExecPrepareExpr() but that is n

Re: missing indexes in indexlist with partitioned tables

2022-01-17 Thread Arne Roland
Hi, thank you for the heads up! Those files ended up accidentally in the dump from me running pg_indent. The file count was truly excessive, I should have noticed this sooner. I attached the patch without the excessive modifications. That should be way easier to read. Regards Arne diff --git

Re: TAP test to cover "EndOfLogTLI != replayTLI" case

2022-01-17 Thread Amul Sul
On Sat, Jan 15, 2022 at 11:35 AM Julien Rouhaud wrote: > > Hi, > > On Mon, Jan 10, 2022 at 09:46:23AM +0530, Amul Sul wrote: > > > > Thanks for the note, I can see the same test is failing on my centos > > vm too with the latest master head(376ce3e404b). The failing reason is > > the "recovery_ta

Re: Partial aggregates pushdown

2022-01-17 Thread Alexander Pyhalov
Zhihong Yu писал 2022-01-17 11:43: Hi, + FdwScanPrivateConvertors + * Generate attinmeta if there are some converters: I think it would be better if converter is spelled the same way across the patch. For build_conv_list(): + if (IS_UPPER_REL(foreignrel)) You can return NIL for !IS_UPPER

Re: New developer papercut - Makefile references INSTALL

2022-01-17 Thread Daniel Gustafsson
> On 17 Jan 2022, at 11:25, Magnus Hagander wrote: > That said, I'm not sure we're actually gaining anything by *not* > referring to the website as well. TBH, I bet the majority of users > will actually prefer to read them there. So I'd suggest always > including the reference to the website as w

Re: Partial aggregates pushdown

2022-01-17 Thread Alexander Pyhalov
Alexander Pyhalov писал 2022-01-17 15:26: Zhihong Yu писал 2022-01-17 11:43: Hi, + FdwScanPrivateConvertors + * Generate attinmeta if there are some converters: I think it would be better if converter is spelled the same way across the patch. For build_conv_list(): + if (IS_UPPER_REL(for

Re: row filtering for logical replication

2022-01-17 Thread Amit Kapila
On Mon, Jan 17, 2022 at 3:19 PM Amit Kapila wrote: > > Apart from the above, I have made a few other cosmetic changes atop > v65-0001*.patch. Kindly review and merge into the main patch if you > are okay with these changes. > Sorry, forgot to attach a top-up patch, sending it now. -- With Regar

RE: Skipping logical replication transactions on subscriber side

2022-01-17 Thread osumi.takami...@fujitsu.com
On Monday, January 17, 2022 5:03 PM I wrote: > Hi, thank you for sharing a new patch. > Few comments on the v6. > > (1) doc/src/sgml/ref/alter_subscription.sgml > > + resort. This option has no effect on the transaction that is > + already > > One TAB exists between "resort" and "This". >

Re: Skipping logical replication transactions on subscriber side

2022-01-17 Thread Masahiko Sawada
On Mon, Jan 17, 2022 at 5:03 PM osumi.takami...@fujitsu.com wrote: > > On Monday, January 17, 2022 3:18 PM Masahiko Sawada > wrote: > > I've attached an updated patch. Please review it. > Hi, thank you for sharing a new patch. > Few comments on the v6. Thank you for the comments! > > (1) doc/s

Re: Skipping logical replication transactions on subscriber side

2022-01-17 Thread Masahiko Sawada
On Mon, Jan 17, 2022 at 9:35 PM osumi.takami...@fujitsu.com wrote: > > On Monday, January 17, 2022 5:03 PM I wrote: > > Hi, thank you for sharing a new patch. > > Few comments on the v6. > > > > (1) doc/src/sgml/ref/alter_subscription.sgml > > > > + resort. This option has no effect on the t

Re: pg_replslotdata - a tool for displaying replication slot information

2022-01-17 Thread Julien Rouhaud
Hi, On Mon, Jan 17, 2022 at 04:10:13PM +0530, Bharath Rupireddy wrote: > > Thanks Juilen. I'm okay if the patch gets dropped. Ok, I will take care of that soon. > But, I'm curious to > know why the above error occurred. Is it because I included the nls.mk > file in the patch which I'm not suppo

Re: Skipping logical replication transactions on subscriber side

2022-01-17 Thread Amit Kapila
On Mon, Jan 17, 2022 at 6:22 PM Masahiko Sawada wrote: > > > > > (5) > > > > I can miss something here but, in one of > > the past discussions, there seems a consensus that > > if the user specifies XID of a subtransaction, > > it would be better to skip only the subtransaction. > > > > This time,

Re: TAP test to cover "EndOfLogTLI != replayTLI" case

2022-01-17 Thread Julien Rouhaud
Hi, On Mon, Jan 17, 2022 at 05:07:48PM +0530, Amul Sul wrote: > > I am not sure what really went wrong but I think the 'standby_9' > server shutdown too early before it gets a chance to archive a > required WAL file. The attached patch tries to shutdown that server > after the required WAL are ar

Re: suboverflowed subtransactions concurrency performance optimize

2022-01-17 Thread Simon Riggs
On Tue, 30 Nov 2021 at 12:19, Simon Riggs wrote: > > On Mon, 30 Aug 2021 at 11:25, Andrey Borodin wrote: > > > > Hi Pengcheng! > > > > You are solving important problem, thank you! > > > > > 30 авг. 2021 г., в 13:43, Pengchengliu > > > написал(а): > > > > > > To resolve this performance problem

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2022 at 12:31 AM Andres Freund wrote: > > Hi, > > On 2022-01-16 15:28:00 -0800, Andres Freund wrote: > > I hacked that up last night. And a fix or two later, it seems to be > > working. What I'd missed at first is that the event needs to be reset in > > reached_end_position(), othe

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread Robert Haas
On Fri, Jan 14, 2022 at 9:54 PM Michael Paquier wrote: > Okay. So, based on this feedback, I guess that something like the > attached would be what we are looking for. I have maximized the > amount of code removed with the removal of -z/-Z, but I won't fight > hard if the consensus is to keep t

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread Robert Haas
On Sat, Jan 15, 2022 at 10:15 AM Magnus Hagander wrote: > It never makes sense to compress *both* in server and client, right? Yeah. > One argument in that case for using --compress would be that we could > have that one take options like --compress=gzip (use gzip in the > client) and --compress

Re: a misbehavior of partition row movement (?)

2022-01-17 Thread Alvaro Herrera
On 2022-Jan-17, Amit Langote wrote: > Note that the fix involves adding fields to ResultRelInfo -- v13 needs > 2 additional, while v14 and HEAD need 1. That combined with needing > new catalog entries for parent FK triggers, back-patching this does > make me a bit uncomfortable. Yeah, that's a g

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2022 at 3:18 PM Robert Haas wrote: > > On Sat, Jan 15, 2022 at 10:15 AM Magnus Hagander wrote: > > It never makes sense to compress *both* in server and client, right? > > Yeah. > > > One argument in that case for using --compress would be that we could > > have that one take opti

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread Magnus Hagander
On Mon, Jan 17, 2022 at 4:56 AM Michael Paquier wrote: > > On Sat, Jan 15, 2022 at 04:15:26PM +0100, Magnus Hagander wrote: > > I think having --client-compress and --server-compress separately but > > having --compression-level *not* being separate would be confusing and > > I *think* that's what

Blank archive_command

2022-01-17 Thread Thom Brown
Hi, Should archive_command being blank when archiving is enabled result in a fatal error? This doesn't even produce a warning when restarting, just an entry in the log when it goes to archive a WAL segment, and finds the archive_command is empty. Is there a valid scenario where someone would hav

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2022-01-17 Thread Shruthi Gowda
On Sat, Jan 15, 2022 at 11:17 AM Julien Rouhaud wrote: > > Hi, > > On Fri, Dec 17, 2021 at 01:03:06PM +0530, Shruthi Gowda wrote: > > > > I have updated the DBOID preserve patch to handle this case and > > generated the latest patch on top of your v7-001-preserve-relfilenode > > patch. > > The cfb

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-17 Thread Robert Haas
On Thu, Jan 13, 2022 at 4:27 PM Peter Geoghegan wrote: > 1. Cases where our inability to get a cleanup lock signifies nothing > at all about the page in question, or any page in the same table, with > the same workload. > > 2. Pathological cases. Cases where we're at least at the mercy of the > ap

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 9:27 AM Magnus Hagander wrote: > I mean that I think it would be confusing to have > --client-compression=x, --server-compression=y, and > compression-level=z as the options. Why, in that scenario, does the > "compression" part get two parameters, but the "compression level

Re: a misbehavior of partition row movement (?)

2022-01-17 Thread Zhihong Yu
On Mon, Jan 17, 2022 at 6:26 AM Alvaro Herrera wrote: > On 2022-Jan-17, Amit Langote wrote: > > > Note that the fix involves adding fields to ResultRelInfo -- v13 needs > > 2 additional, while v14 and HEAD need 1. That combined with needing > > new catalog entries for parent FK triggers, back-pa

Re: Adding CI to our tree

2022-01-17 Thread Andrew Dunstan
On 1/14/22 18:54, Justin Pryzby wrote: > On Fri, Jan 14, 2022 at 03:34:11PM -0800, Andres Freund wrote: >> Hi, >> >> On 2022-01-13 15:27:40 -0500, Andrew Dunstan wrote: >>> I can probably adjust to whatever we decide to do. But I think we're >>> really just tinkering at the edges here. What I thi

Re: Blank archive_command

2022-01-17 Thread Bharath Rupireddy
On Mon, Jan 17, 2022 at 8:14 PM Thom Brown wrote: > > Hi, > > Should archive_command being blank when archiving is enabled result in > a fatal error? This doesn't even produce a warning when restarting, > just an entry in the log when it goes to archive a WAL segment, and > finds the archive_comm

RE: row filtering for logical replication

2022-01-17 Thread houzj.f...@fujitsu.com
On Mon, Jan 17, 2022 12:34 PM Peter Smith wrote: > > Here are some review comments for v65-0001 (review of updates since > v64-0001) Thanks for the comments! > ~~~ > > 1. src/include/commands/publicationcmds.h - rename func > > +extern bool contain_invalid_rfcolumn(Oid pubid, Relation relatio

RE: row filtering for logical replication

2022-01-17 Thread houzj.f...@fujitsu.com
On Mon, Jan 17, 2022 12:35 PM Greg Nancarrow wrote: > > On Sat, Jan 15, 2022 at 5:30 PM houzj.f...@fujitsu.com > wrote: > > > > Attach the V65 patch set which addressed the above comments and Peter's > comments[1]. > > I also fixed some typos and removed some unused code. > > > > I have several

Re: Blank archive_command

2022-01-17 Thread Tom Lane
Bharath Rupireddy writes: > However, a reasonable thing to do is to > emit a WARNING or ERROR-out when archive_command is set to null in > it's check_archive_command when archive_mode is on? We have been burned badly in the past by attempts to do that sort of thing (ie, make behavior that's condi

Re: Blank archive_command

2022-01-17 Thread Bharath Rupireddy
On Mon, Jan 17, 2022 at 9:02 PM Tom Lane wrote: > > Bharath Rupireddy writes: > > However, a reasonable thing to do is to > > emit a WARNING or ERROR-out when archive_command is set to null in > > it's check_archive_command when archive_mode is on? > > We have been burned badly in the past by att

Re: 2022-01 Commitfest

2022-01-17 Thread Julien Rouhaud
Hi, This is the beginning of the 3rd week of this commit fest. Since my last email 5 days ago, 6 patches were committed and a few patches closed. There are still overall 229 active patches, most of them unsurprisingly waiting for review. The cfbot is doing a great job at early problem detection

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread Alvaro Herrera
On 2022-Jan-17, Robert Haas wrote: > Of the two > alternatives that you propose, I prefer --compress=["server-"]METHOD > and --compression-level=NUMBER to having both > --client-compression-level and --server-compression-level. To me, > that's still a bit more surprising than my proposal, because

Re: ExecRTCheckPerms() and many prunable partitions

2022-01-17 Thread Zhihong Yu
On Mon, Jan 17, 2022 at 3:51 AM Amit Langote wrote: > On Thu, Jan 13, 2022 at 3:39 PM Amit Langote > wrote: > > On Thu, Jan 13, 2022 at 12:10 PM Julien Rouhaud > wrote: > > > On Mon, Dec 20, 2021 at 04:13:04PM +0900, Amit Langote wrote: > > > > Patch 0002 needed a rebase, because a conflicting

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread David G. Johnston
On Mon, Jan 17, 2022 at 8:17 AM Robert Haas wrote: > On Mon, Jan 17, 2022 at 9:27 AM Magnus Hagander > wrote: > > I mean that I think it would be confusing to have > > --client-compression=x, --server-compression=y, and > > compression-level=z as the options. Why, in that scenario, does the > >

Re: Blank archive_command

2022-01-17 Thread Thom Brown
On Mon, 17 Jan 2022 at 15:25, Bharath Rupireddy wrote: > > On Mon, Jan 17, 2022 at 8:14 PM Thom Brown wrote: > > > > Hi, > > > > Should archive_command being blank when archiving is enabled result in > > a fatal error? This doesn't even produce a warning when restarting, > > just an entry in the

Re: Blank archive_command

2022-01-17 Thread Bharath Rupireddy
On Mon, Jan 17, 2022 at 9:05 PM Bharath Rupireddy wrote: > > +1 to not create any GUC setting dependencies. Let's leave the > responsibility of setting appropriate archive_command to the archiving > handlers outside postgres. FWIW, having a note in the archive_command > GUC definition in the docs

Re: Add Boolean node

2022-01-17 Thread Peter Eisentraut
On 13.01.22 10:48, Pavel Stehule wrote: There are not objection from me or from community I'll mark this patch as ready for committer This patch set has been committed.

Re: suboverflowed subtransactions concurrency performance optimize

2022-01-17 Thread Andrey Borodin
> 17 янв. 2022 г., в 18:44, Simon Riggs > написал(а): > > Re-attached, so that the CFapp isn't confused between the multiple > patches on this thread. FWIW I've looked into the patch and it looks good to me. Comments describing when the cache is useful seem valid. Thanks! Best regards, An

Re: a misbehavior of partition row movement (?)

2022-01-17 Thread Alvaro Herrera
On 2022-Jan-17, Zhihong Yu wrote: > On Mon, Jan 17, 2022 at 6:26 AM Alvaro Herrera > wrote: > > On 2022-Jan-17, Amit Langote wrote: > > The other is suggested by you: > > > > > Another thing to consider is that we haven't seen many reports of the > > > problem (UPDATEs of partitioned PK tables

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread David G. Johnston
On Mon, Jan 17, 2022 at 8:41 AM Alvaro Herrera wrote: > On 2022-Jan-17, Robert Haas wrote: > > > Of the two > > alternatives that you propose, I prefer --compress=["server-"]METHOD > > and --compression-level=NUMBER to having both > > --client-compression-level and --server-compression-level. To

Re: Avoiding smgrimmedsync() during nbtree index builds

2022-01-17 Thread Justin Pryzby
On Sun, Jan 16, 2022 at 02:25:59PM -0600, Justin Pryzby wrote: > On Thu, Jan 13, 2022 at 09:52:55AM -0600, Justin Pryzby wrote: > > This is failing on windows CI when I use initdb --data-checksums, as > > attached. > > > > https://cirrus-ci.com/task/5612464120266752 > > https://api.cirrus-ci.com/

Re: ICU for global collation

2022-01-17 Thread Finnerty, Jim
Re: >> After this patch goes in, the big next thing would be to support >> nondeterministic collations for LIKE, ILIKE and pattern matching operators in >> general. Is anyone interested in working on that? > As far as I know you're the last person that seemed to be working on that > topic > bac

Re: Refactoring of compression options in pg_basebackup

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 11:50 AM David G. Johnston wrote: >> I think having a single option where you specify everything is simpler. >> I propose we accept these forms: >> >> --compress=[{server,client}-]method[:level] new in 15 >> --compress=level(accepted by 14) >> -Z level

Re: Blank archive_command

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 10:53 AM Bharath Rupireddy wrote: > This parameter can only be set in the postgresql.conf file or on the > server command line. It is ignored unless archive_mode was enabled at > server start. If archive_command is an empty string (the default) > while archive_mode is enabl

Re: missing indexes in indexlist with partitioned tables

2022-01-17 Thread Arne Roland
Using some valuable feedback from Zhihong Yu, I fixed a flipped negation error and updated the comments. Regards Arne From: Arne Roland Sent: Monday, January 17, 2022 12:25 To: Julien Rouhaud Cc: pgsql-hackers Subject: Re: missing indexes in indexlist with parti

Re: pg14 psql broke \d datname.nspname.relname

2022-01-17 Thread Mark Dilger
> On Jan 15, 2022, at 12:28 AM, Julien Rouhaud wrote: > > Could you send a rebased version? Yes. Here it is: v4-0001-Reject-patterns-with-too-many-parts-or-wrong-db.patch Description: Binary data — Mark Dilger EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company

Re: pg_basebackup WAL streamer shutdown is bogus - leading to slow tests

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 14:50:27 +0100, Magnus Hagander wrote: > On Mon, Jan 17, 2022 at 12:31 AM Andres Freund wrote: > > > > Hi, > > > > On 2022-01-16 15:28:00 -0800, Andres Freund wrote: > > > I hacked that up last night. And a fix or two later, it seems to be > > > working. What I'd missed at first

Re: missing indexes in indexlist with partitioned tables

2022-01-17 Thread Alvaro Herrera
Hmm, can you show cases of queries for which having this new partIndexlist changes plans? -- Álvaro Herrera PostgreSQL Developer — https://www.EnterpriseDB.com/ "Saca el libro que tu religión considere como el indicado para encontrar la oración que traiga paz a tu alma. Luego rebootea e

Re: Adding CI to our tree

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 10:25:12 -0500, Andrew Dunstan wrote: > The buildfarm is moving in the opposite direction, to disaggregate > steps. I'm a bit confused as to where you want changes to vcregress.pl going. Upthread you argued against adding more granular targets to vcregress. But this seems to be

Re: slowest tap tests - split or accelerate?

2022-01-17 Thread Andres Freund
Hi, On 2021-12-31 11:25:28 -0800, Andres Freund wrote: > cfbot now runs most tests on windows, the windows task is by far the slowest, > and the task limitted most in concurrency [2]. Running tap tests is the > biggest part of that. This is a bigger issue on windows because we don't have > infrast

Re: Adding CI to our tree

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 1:19 PM Andres Freund wrote: > FWIW, to me this shouldn't require a lot of separate manual test > invocations. And continuing to have lots of granular test invocations from the > buildfarm client is *bad*, because it requires constantly syncing up the set > of test targets.

Re: slowest tap tests - split or accelerate?

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 1:41 PM Andres Freund wrote: > The reason these in particular are slow is that they do a lot of > pg_basebackups without either / one-of -cfast / --no-sync. The lack of -cfast > in particularly is responsible for a significant proportion of the test > time. The only reason

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2022-01-17 Thread Robert Haas
On Tue, Dec 14, 2021 at 1:21 PM Shruthi Gowda wrote: > Thanks, Robert for the updated version. I reviewed the changes and it > looks fine. > I also tested the patch. The patch works as expected. Thanks. > > - I adjusted the function header comment for heap_create. Your > > proposed comment seeme

Re: ICU for global collation

2022-01-17 Thread Finnerty, Jim
On 10.01.22 12:49, Daniel Verite wrote: > I think some users would want their db-wide ICU collation to be > case/accent-insensitive. ... > IIRC, that was the context for some questions where people were > enquiring about db-wide ICU collations. +1. There is the DEFAULT_COLLATION_OID, which

Re: Adding CI to our tree

2022-01-17 Thread Tom Lane
Robert Haas writes: > I have a lot of sympathy with Andrew here, actually. If you just do > 'make check-world' and assume that will cover everything, you get one > giant output file. That is not great at all. Yeah. I agree with Andrew that we want output that is more modular, not less so. But w

Re: Pluggable toaster

2022-01-17 Thread Tomas Vondra
On 1/14/22 19:41, Teodor Sigaev wrote: In my understanding, we want to be able to 1. Access data from a toasted object one slice at a time, by using knowledge of the structure 2. If toasted data is updated, then update a minimum number of slices(s), without rewriting the existing slices 3. I

Re: Adding CI to our tree

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 13:50:04 -0500, Robert Haas wrote: > On Mon, Jan 17, 2022 at 1:19 PM Andres Freund wrote: > > FWIW, to me this shouldn't require a lot of separate manual test > > invocations. And continuing to have lots of granular test invocations from > > the > > buildfarm client is *bad*, b

Re: Adding CI to our tree

2022-01-17 Thread Tom Lane
Andres Freund writes: > I think it's not actually that hard, with something like I described in the > email upthread, with each tests going into a prescribed location, and the > on-disk status being inspectable in an automated way. check-world could invoke > a command to summarize the tests at the

Re: Blank archive_command

2022-01-17 Thread Tom Lane
Robert Haas writes: > It might be nice to do something about the fact that you can't change > archive_mode without a server restart, though. I suspect we had a good > reason for that limitation from an engineering perspective, but from a > user perspective, it sucks pretty hard. Agreed. I don't

Re: slowest tap tests - split or accelerate?

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 14:05:17 -0500, Robert Haas wrote: > On Mon, Jan 17, 2022 at 1:41 PM Andres Freund wrote: > > The reason these in particular are slow is that they do a lot of > > pg_basebackups without either / one-of -cfast / --no-sync. The lack of > > -cfast > > in particularly is responsibl

Re: SLRUs in the main buffer pool, redux

2022-01-17 Thread Thomas Munro
On Mon, Jan 17, 2022 at 11:23 PM Heikki Linnakangas wrote: > IIRC one issue with this has been performance. When an SLRU is working > well, a cache hit in the SLRU is very cheap. Linearly scanning the SLRU > array is cheap, compared to computing the hash and looking up a buffer > in the buffer cac

removing datlastsysoid

2022-01-17 Thread Robert Haas
Hi, While reviewing another patch, I noticed that it slightly adjusted the treatment of datlastsysoid. That made me wonder what datlastsysoid is used for, so I started poking around and discovered that the answer, at least insofar as I can determine, is "nothing". The documentation claims that the

Re: slowest tap tests - split or accelerate?

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 2:57 PM Andres Freund wrote: > I wonder if there's something explaining why pg_verifybackup is greatly slowed > down by sha224 but not crc32c, but the server's runtime only differs by ~20ms? > It seems incongruous that pg_basebackup, with all the complexity of needing to >

Re: Adding CI to our tree

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 14:30:53 -0500, Tom Lane wrote: > Andres Freund writes: > > I think it's not actually that hard, with something like I described in the > > email upthread, with each tests going into a prescribed location, and the > > on-disk status being inspectable in an automated way. check-w

Re: missing indexes in indexlist with partitioned tables

2022-01-17 Thread Arne Roland
Hi! Afaiac the join pruning where the outer table is a partitioned table is the relevant case. I am not sure whether there are other cases. The join pruning, which works great for plain relations since 9.0, falls short for partitioned tables, since the optimizer fails to prove uniqueness there.

Re: [EXTERNAL] Re: PQcancel does not use tcp_user_timeout, connect_timeout and keepalive settings

2022-01-17 Thread Tom Lane
Jelte Fennema writes: > Thanks for all the cleanup and adding of windows support. To me it now looks > good to merge. I was about to commit this when I started to wonder if it actually does anything useful. In particular, I read in the Linux tcp(7) man page TCP_USER_TIMEOUT (since Linux

Re: removing datlastsysoid

2022-01-17 Thread Tom Lane
Robert Haas writes: > Since that doesn't seem like an especially good idea, PFA a patch to > remove it. Note that, even prior to that commit, it wasn't being used > for anything when dumping modern servers, so it would still have been > OK to remove it from the current system catalog structure. No

Re: slowest tap tests - split or accelerate?

2022-01-17 Thread Tom Lane
Andres Freund writes: > I've occasionally pondered caching initdb results and reusing them across > tests - just the locking around it seems a bit nasty, but perhaps that could > be done as part of the tmp_install step. Of course, it'd need to deal with > different options etc... I'd actually bui

Re: slowest tap tests - split or accelerate?

2022-01-17 Thread Andres Freund
Hi, On 2022-01-17 15:13:57 -0500, Robert Haas wrote: > I guess there must be something explaining it, but I don't know what > it could be. The client and the server are each running the checksum > algorithm over the same data. If that's not the same speed then I > don't get it. Unless, someho

Re: preserving db/ts/relfilenode OIDs across pg_upgrade (was Re: storing an explicit nonce)

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 9:57 AM Shruthi Gowda wrote: > I have rebased and generated the patches on top of PostgreSQL commit > ID cf925936ecc031355cd56fbd392ec3180517a110. > Kindly apply v8-0001-pg_upgrade-Preserve-relfilenodes-and-tablespace-O.patch > first and then v8-0002-Preserve-database-OIDs-

Re: Adding CI to our tree

2022-01-17 Thread Andrew Dunstan
On 1/17/22 13:19, Andres Freund wrote: > Hi, > > On 2022-01-17 10:25:12 -0500, Andrew Dunstan wrote: >> The buildfarm is moving in the opposite direction, to disaggregate >> steps. > I'm a bit confused as to where you want changes to vcregress.pl > going. Upthread you argued against adding more g

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread Robert Haas
On Fri, Jan 14, 2022 at 7:42 PM James Coleman wrote: > I've attached a simple patch (sans tests and documentation) to get > feedback early. After poking around this afternoon it seemed to me > that the simplest approach was to hook into the commit timestamps > infrastructure and store the commit's

Re: a misbehavior of partition row movement (?)

2022-01-17 Thread Alvaro Herrera
> @@ -3398,7 +3432,7 @@ typedef SetConstraintStateData *SetConstraintState; > */ > typedef uint32 TriggerFlags; > > -#define AFTER_TRIGGER_OFFSET 0x0FFF /* must be > low-order bits */ > +#define AFTER_TRIGGER_OFFSET 0x07FF /* must be > low-

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-17 Thread Peter Geoghegan
On Mon, Jan 17, 2022 at 7:12 AM Robert Haas wrote: > On Thu, Jan 13, 2022 at 4:27 PM Peter Geoghegan wrote: > > 1. Cases where our inability to get a cleanup lock signifies nothing > > at all about the page in question, or any page in the same table, with > > the same workload. > > > > 2. Patholo

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread Alvaro Herrera
On 2022-Jan-14, James Coleman wrote: > The logical slot can't flush past the > last commit, so even if there's 100s of megabytes of unflushed WAL on > the slot there may be zero lag (in terms of what's possible to > process). > > I've attached a simple patch (sans tests and documentation) to get >

Re: a misbehavior of partition row movement (?)

2022-01-17 Thread Tom Lane
Alvaro Herrera writes: > So this patch releases one bit from AFTER_TRIGGER_OFFSET and makes it > become AFTER_TRIGGER_CP_UPDATE. As far as I can tell there is no harm > in doing so. I agree that taking a bit away from AFTER_TRIGGER_OFFSET is okay (it could spare even a couple more, if we need th

Re: pg14 psql broke \d datname.nspname.relname

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 1:06 PM Mark Dilger wrote: > > On Jan 15, 2022, at 12:28 AM, Julien Rouhaud wrote: > > Could you send a rebased version? > Yes. Here it is: This is not a full review, but I just noticed that: + * dotcnt: how many separators were parsed from the pattern, by reference. +

Re: Add last commit LSN to pg_last_committed_xact()

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 4:34 PM Alvaro Herrera wrote: > On 2022-Jan-14, James Coleman wrote: > > The logical slot can't flush past the > > last commit, so even if there's 100s of megabytes of unflushed WAL on > > the slot there may be zero lag (in terms of what's possible to > > process). > > > >

Re: \d with triggers: more than one row returned by a subquery used as an expression

2022-01-17 Thread Tom Lane
Justin Pryzby writes: > On Fri, Dec 17, 2021 at 09:43:56AM -0600, Justin Pryzby wrote: >> I want to mention that the 2nd problem I mentioned here is still broken. >> https://www.postgresql.org/message-id/20210717010259.gu20...@telsasoft.com >> It happens if non-inheritted triggers on child and par

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-17 Thread Robert Haas
On Mon, Jan 17, 2022 at 4:28 PM Peter Geoghegan wrote: > Updating relfrozenxid should now be thought of as a continuous thing, > not a discrete thing. I think that's pretty nearly 100% wrong. The most simplistic way of expressing that is to say - clearly it can only happen when VACUUM runs, which

Re: a misbehavior of partition row movement (?)

2022-01-17 Thread Alvaro Herrera
On 2022-Jan-17, Tom Lane wrote: > But could we please do it in a way that is designed to keep the > code readable, rather than to minimize the number of lines of diff? > It makes zero sense to have the bits in AFTER_TRIGGER_TUP_BITS not > be adjacent. So what should happen here is to renumber the

Re: Pluggable toaster

2022-01-17 Thread Nikita Malakhov
Hi, >This sounds interesting, but very much like column compression, which >was proposed some time ago. If we haven't made much progrees with that >patch (AFAICS), what's the likelihood we'll succeed here, when it's >combined with yet more complexity? The main concern is that this patch provides o

Re: Removing more vacuumlazy.c special cases, relfrozenxid optimizations

2022-01-17 Thread Peter Geoghegan
On Mon, Jan 17, 2022 at 2:13 PM Robert Haas wrote: > On Mon, Jan 17, 2022 at 4:28 PM Peter Geoghegan wrote: > > Updating relfrozenxid should now be thought of as a continuous thing, > > not a discrete thing. > > I think that's pretty nearly 100% wrong. The most simplistic way of > expressing that

Re: [PATCH] reduce page overlap of GiST indexes built using sorted method

2022-01-17 Thread Björn Harrtell
Hi Aliaksandr, Nice work on this. I've been following it a bit since the regression when it was noted and it sparked renewed interest in R-tree structure and optimization for me. As for ideas. I'm not deep into details of postgresql and gist, but I've learned that the node size for gist indexes a

Re: Push down time-related SQLValue functions to foreign server

2022-01-17 Thread Tom Lane
Alexander Pyhalov writes: >> Perhaps in a MACRO? > Changed this check to a macro, also fixed condition in > is_foreign_param() and added test for it. > Also fixed comment in prepare_query_params(). I took a quick look at this. I'm unconvinced that you need the TIME_RELATED_SQLVALUE_FUNCTION ma

Re: a misbehavior of partition row movement (?)

2022-01-17 Thread Tom Lane
Alvaro Herrera writes: > On 2022-Jan-17, Tom Lane wrote: >> It makes zero sense to have the bits in AFTER_TRIGGER_TUP_BITS not >> be adjacent. So what should happen here is to renumber the symbols >> in between to move their bits over one place. > Is it typical to enumerate bits starting from th

  1   2   >