Re: add checkpoint stats of snapshot and mapping files of pg_logical dir

2021-12-30 Thread Bharath Rupireddy
On Tue, Nov 30, 2021 at 6:24 AM Cary Huang wrote: > > The following review has been posted through the commitfest application: > make installcheck-world: tested, passed > Implements feature: tested, passed > Spec compliant: tested, passed > Documentation:not tested > >

Re: skip replication slot snapshot/map file removal during end-of-recovery checkpoint

2021-12-30 Thread Bharath Rupireddy
On Thu, Dec 23, 2021 at 4:46 PM Bharath Rupireddy wrote: > > Hi, > > Currently the end-of-recovery checkpoint can be much slower, impacting > the server availability, if there are many replication slot files > .snap or map- to be enumerated and deleted. How about skipping > the .snap and m

RE: [PATCH]Comment improvement in publication.sql

2021-12-30 Thread tanghy.f...@fujitsu.com
On Monday, December 13, 2021 11:53 PM vignesh C wrote: > > On Wed, Dec 8, 2021 at 11:07 AM tanghy.f...@fujitsu.com > wrote: > > > > On Wednesday, December 8, 2021 1:49 PM, vignesh C > wrote: > > > > > The patch no longer applies, could you post a rebased patch. > > > > Thanks for your kindly re

Re: Patch to avoid orphaned dependencies

2021-12-30 Thread Andres Freund
Hi, On 2021-12-17 14:19:18 +0100, Drouvot, Bertrand wrote: > Please find enclosed v1-0003-orphaned-dependencies.patch, that contains: > > - a mandatory rebase > > - a few isolation tests added in src/test/modules/test_dependencies (but I'm > not sure at all that's the right place to add them, is

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
On 2021-12-30 17:46:52 -0800, Andres Freund wrote: > I plan to push this after another cycle of tests passing (and driving > over the bay bridge...) unless I hear protests. Pushed. Marked CF entry as committed.

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
Hi, On 2021-12-30 20:28:46 -0600, Justin Pryzby wrote: > [ language fixes] Thanks! > script uses a pseudo-tty, which do support locking. > => which does This didn't seem right either way - it's pseudo-ttys that don't support locking, so plural seemed appropriate? I changed it to "script uses ps

Re: Adding CI to our tree

2021-12-30 Thread Justin Pryzby
commit message: agithub the the the buildfarm. => the access too. => to # Due to that it using concurrency within prove is helpful. => Due to that, it's useful to run prove with multiple jobs. further details src/tools/ci/README => further details , see src/tools/ci/README script uses a pseudo

Re: generic plans and "initial" pruning

2021-12-30 Thread Amit Langote
On Tue, Dec 28, 2021 at 22:12 Ashutosh Bapat wrote: > On Sat, Dec 25, 2021 at 9:06 AM Amit Langote > wrote: > > > > Executing generic plans involving partitions is known to become slower > > as partition count grows due to a number of bottlenecks, with > > AcquireExecutorLocks() showing at the t

Re: Autovacuum and idle_session_timeout

2021-12-30 Thread Japin Li
On Fri, 31 Dec 2021 at 00:24, Tom Lane wrote: > Japin Li writes: >> On Thu, 30 Dec 2021 at 18:53, Guillaume Lelarge >> wrote: >>> pg_dump works in a single transaction, so it's already dealt with >>> idle_in_transaction_timeout. Though I guess setting both would work too. > >> Attached fix th

Re: Adding CI to our tree

2021-12-30 Thread Andres Freund
Hi, Attached is v5 of the CI patch. Not a lot of changes: - a bunch of copy-editing, wrote a commit message etc - use ccache for CXX/CLANG in the CompilerWarnings task, I had missed that when making the task use all --with-* flags - switch linux to use ossp-uuid. I tried to switch macos at first

RE: Failed transaction statistics to measure the logical replication progress

2021-12-30 Thread tanghy.f...@fujitsu.com
On Wednesday, December 22, 2021 6:14 PM osumi.takami...@fujitsu.com wrote: > > Attached the new patch v19. > Thanks for your patch. I think it's better if you could add this patch to the commitfest. Here are some comments: 1) + commit_count bigint + + + Number of tran

Re: Apple's ranlib warns about protocol_openssl.c

2021-12-30 Thread Tom Lane
Andres Freund writes: > I also see it on an m1 mini I got when building against openssl 3. Huh, I wonder why I'm not seeing it. > There is -no_warning_for_no_symbols in apple's ranlib. But perhaps > there's another way around this: > We have ssl_protocol_version_to_openssl() in both be-secure-op

Re: Apple's ranlib warns about protocol_openssl.c

2021-12-30 Thread Andres Freund
Hi, On 2021-12-17 14:26:53 +1300, Thomas Munro wrote: > On Fri, Dec 17, 2021 at 9:38 AM Tom Lane wrote: > > Could be. I tried it on Monterey, but not anything older. > > (longfin is still on Big Sur, because I've been lazy about > > updating it.) > > Hmm. Happened[1] with Andres's CI scripts,

Re: Column Filtering in Logical Replication

2021-12-30 Thread Alvaro Herrera
On 2021-Dec-30, Justin Pryzby wrote: Thank you! I've incorporated your proposed fixes. > > + /* > > +* Even if the user listed all columns in the column list, we > > cannot > > +* allow a column list to be specified when REPLICA IDENTITY is > > FULL; > > +

More pg_dump performance hacking

2021-12-30 Thread Tom Lane
Attached are a couple of patches for loose ends that I didn't get to when I was working on pg_dump before the last CF. 0001 removes all the "username_subquery" subqueries in favor of doing local username lookups. On the regression database with no extra roles, it seems to be more or less a wash .

Re: Column Filtering in Logical Replication

2021-12-30 Thread Justin Pryzby
> + boolam_partition = false; >... > Assert(!isnull); > lrel->relkind = DatumGetChar(slot_getattr(slot, 3, &isnull)); > Assert(!isnull); > + am_partition = DatumGetChar(slot_getattr(slot, 4, &isnull)); I think this needs to be GetBool. You should Assert(!isnul

Re: Why is src/test/modules/committs/t/002_standby.pl flaky?

2021-12-30 Thread Andrew Dunstan
On 12/30/21 15:01, Thomas Munro wrote: > Hi, > > There's a wait for replay that is open coded (instead of using the > wait_for_catchup() routine), and sometimes the second of two such > waits at line 51 (in master) times out after 3 minutes with "standby > never caught up". It's happening on thr

Re: pg_stat_bgwriter.buffers_backend is pretty meaningless (and more?)

2021-12-30 Thread Melanie Plageman
On Tue, Dec 21, 2021 at 8:32 PM Melanie Plageman wrote: > On Thu, Dec 16, 2021 at 3:18 PM Andres Freund wrote: > > > > > From 9f22da9041e1e1fbc0ef003f5f78f4e72274d438 Mon Sep 17 00:00:00 2001 > > > > > From: Melanie Plageman > > > > > Date: Wed, 24 Nov 2021 12:20:10 -0500 > > > > > Subject: [PAT

Re: Column Filtering in Logical Replication

2021-12-30 Thread Alvaro Herrera
On 2021-Dec-29, Alvaro Herrera wrote: > This new stuff is not yet finished. For example I didn't refactor > handling of REPLICA IDENTITY, so the new command does not correctly > check everything, such as the REPLICA IDENTITY FULL stuff. Also, no > tests have been added yet. In manual tests it s

Why is src/test/modules/committs/t/002_standby.pl flaky?

2021-12-30 Thread Thomas Munro
Hi, There's a wait for replay that is open coded (instead of using the wait_for_catchup() routine), and sometimes the second of two such waits at line 51 (in master) times out after 3 minutes with "standby never caught up". It's happening on three particular Windows boxes, but once also happened

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-30 Thread Andres Freund
Hi, On 2021-12-29 23:06:31 -0800, SATYANARAYANA NARLAPURAM wrote: > I am afraid there are problems with making the RPO check post releasing the > locks. By this time the transaction is committed and visible to the other > backends (ProcArrayEndTransaction is already called) though the intention >

Re: Autovacuum and idle_session_timeout

2021-12-30 Thread Guillaume Lelarge
Le jeu. 30 déc. 2021 à 17:25, Tom Lane a écrit : > Japin Li writes: > > On Thu, 30 Dec 2021 at 18:53, Guillaume Lelarge > wrote: > >> pg_dump works in a single transaction, so it's already dealt with > >> idle_in_transaction_timeout. Though I guess setting both would work too. > > > Attached fi

Re: Strange path from pgarch_readyXlog()

2021-12-30 Thread Bossart, Nathan
On 12/29/21, 3:11 PM, "Tom Lane" wrote: > "Bossart, Nathan" writes: >> This crossed my mind, too. I also think one of the arrays can be >> eliminated in favor of just using the heap (after rebuilding with a >> reversed comparator). Here is a minimally-tested patch that >> demonstrates what I'm

Pluggable toaster

2021-12-30 Thread Teodor Sigaev
Hi! We are working on custom toaster for JSONB [1], because current TOAST is universal for any data type and because of that it has some disadvantages:   - "one toast fits all"  may be not the best solution for particular     type or/and use cases   - it doesn't know the internal structure o

Re: Autovacuum and idle_session_timeout

2021-12-30 Thread Tom Lane
Japin Li writes: > On Thu, 30 Dec 2021 at 18:53, Guillaume Lelarge > wrote: >> pg_dump works in a single transaction, so it's already dealt with >> idle_in_transaction_timeout. Though I guess setting both would work too. > Attached fix this, please consider reveiew it. Thanks. This seems rath

Re: Autovacuum and idle_session_timeout

2021-12-30 Thread Guillaume Lelarge
Le jeu. 30 déc. 2021 à 12:01, Japin Li a écrit : > > On Thu, 30 Dec 2021 at 18:53, Guillaume Lelarge > wrote: > > Le jeu. 30 déc. 2021 à 11:44, Japin Li a écrit : > > > >> > > pg_dump works in a single transaction, so it's already dealt with > > idle_in_transaction_timeout. Though I guess setti

Re: Fix BUG #17335: Duplicate result rows in Gather node

2021-12-30 Thread Dilip Kumar
On Thu, Dec 30, 2021 at 4:44 PM Yura Sokolov wrote: > Good day, hackers. > > Problem: > - Append path is created with explicitely parallel_aware = true > - It has two child, one is trivial, other is parallel_aware = false . > Trivial child is dropped. > - Gather/GatherMerge path takes Append pa

Re: ICU for global collation

2021-12-30 Thread Peter Eisentraut
There were a few inquiries about this topic recently, so I dug up the old thread and patch. What we got stuck on last time was that we can't just swap out all locale support in a database for ICU. We still need to set the usual locale environment, otherwise some things that are not ICU awar

Re: Pre-allocating WAL files

2021-12-30 Thread Maxim Orlov
I did check the patch too and found it to be ok. Check and check-world are passed. Overall idea seems to be good in my opinion, but I'm not sure where is the optimal place to put the pre-allocation. On Thu, Dec 30, 2021 at 2:46 PM Pavel Borisov wrote: > > pre-allocating during checkpoints. I've

Re: Pre-allocating WAL files

2021-12-30 Thread Pavel Borisov
> > > pre-allocating during checkpoints. I've done a few pgbench runs, and > > it seems to work pretty well. Initialization is around 15% faster, > > and I'm seeing about a 5% increase in TPS with a simple-update > > workload with wal_recycle turned off. Of course, these improvements > > go away

Fix BUG #17335: Duplicate result rows in Gather node

2021-12-30 Thread Yura Sokolov
Good day, hackers. Problem: - Append path is created with explicitely parallel_aware = true - It has two child, one is trivial, other is parallel_aware = false . Trivial child is dropped. - Gather/GatherMerge path takes Append path as a child and thinks its child is parallel_aware = true. - Bu

Re: Autovacuum and idle_session_timeout

2021-12-30 Thread Japin Li
On Thu, 30 Dec 2021 at 18:53, Guillaume Lelarge wrote: > Le jeu. 30 déc. 2021 à 11:44, Japin Li a écrit : > >> > pg_dump works in a single transaction, so it's already dealt with > idle_in_transaction_timeout. Though I guess setting both would work too. Attached fix this, please consider reveie

Re: Foreign key joins revisited

2021-12-30 Thread Joel Jacobson
On Wed, Dec 29, 2021, at 16:28, Tom Lane wrote: >Peter Eisentraut writes: >> In the 1990s, there were some SQL drafts that included syntax like >> JOIN ... USING PRIMARY KEY | USING FOREIGN KEY | USING CONSTRAINT ... >> AFAICT, these ideas just faded away because of other priorities, so if >> som

Re: Autovacuum and idle_session_timeout

2021-12-30 Thread Guillaume Lelarge
Le jeu. 30 déc. 2021 à 11:44, Japin Li a écrit : > > On Thu, 30 Dec 2021 at 17:18, Guillaume Lelarge > wrote: > > Hello, > > > > I've been reading the autovacuum code (the launcher and the worker) on > the > > 14 branch. As previously, I've seen some configuration at the beginning, > > especiall

Re: Autovacuum and idle_session_timeout

2021-12-30 Thread Japin Li
On Thu, 30 Dec 2021 at 17:18, Guillaume Lelarge wrote: > Hello, > > I've been reading the autovacuum code (the launcher and the worker) on the > 14 branch. As previously, I've seen some configuration at the beginning, > especially for statement_timeout, lock_timeout and > idle_in_transaction_ses

Re: Non-decimal integer literals

2021-12-30 Thread Peter Eisentraut
There has been some other refactoring going on, which made this patch set out of date. So here is an update. The old pg_strtouint64() has been removed, so there is no longer a naming concern with patch 0001. That one should be good to go. I also found that yet another way to parse integers

Autovacuum and idle_session_timeout

2021-12-30 Thread Guillaume Lelarge
Hello, I've been reading the autovacuum code (the launcher and the worker) on the 14 branch. As previously, I've seen some configuration at the beginning, especially for statement_timeout, lock_timeout and idle_in_transaction_session_timeout, and I was surprised to discover there was no configurat

Re: Add Boolean node

2021-12-30 Thread Peter Eisentraut
On 29.12.21 21:32, Andres Freund wrote: On 2021-12-27 09:53:32 -0500, Tom Lane wrote: Didn't really read the patch in any detail, but I did have one idea: I think that the different things-that-used-to-be-Value-nodes ought to use different field names, say ival, rval, bval, sval not just "val".

RE: row filtering for logical replication

2021-12-30 Thread wangw.f...@fujitsu.com
On Mon, Dec 28, 2021 9:03 PM houzj.f...@fujitsu.com wrote: > Attach a top up patch 0004 which did the above changes. A few comments about v55-0001 and v55-0002. v55-0001 1. There is a typo at the last sentence of function(rowfilter_walker)'s comment. * (b) a user-defined function can be used

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-30 Thread SATYANARAYANA NARLAPURAM
On Thu, Dec 30, 2021 at 12:20 AM Dilip Kumar wrote: > On Thu, Dec 30, 2021 at 1:41 PM Bharath Rupireddy < > bharath.rupireddyforpostg...@gmail.com> wrote: > >> >> > >> > Yeah, that's true, but even if we are blocking the transactions from >> committing then also it is possible that a new connecti

Re: warn if GUC set to an invalid shared library

2021-12-30 Thread Bharath Rupireddy
On Tue, Dec 28, 2021 at 11:15 PM Justin Pryzby wrote: > > forking > > On Mon, Dec 13, 2021 at 09:01:57AM -0500, Robert Haas wrote: > > On Thu, Dec 9, 2021 at 2:32 AM Maciek Sakrejda wrote: > > > > Considering the vanishingly small number of actual complaints we've > > > > seen about this, that s

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-30 Thread Dilip Kumar
On Thu, Dec 30, 2021 at 1:41 PM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > > > > > Yeah, that's true, but even if we are blocking the transactions from > committing then also it is possible that a new connection can come and > generate more WAL, yeah but I agree with the

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-30 Thread Bharath Rupireddy
On Thu, Dec 30, 2021 at 1:21 PM Dilip Kumar wrote: > > On Thu, Dec 30, 2021 at 12:36 PM SATYANARAYANA NARLAPURAM > wrote: >>> >>> >>> Yeah, I think that would make sense, even though we will be allowing a new >>> backend to get connected insert WAL, and get committed but after that, it >>> wil