Re: sequences vs. synchronous replication

2021-12-23 Thread Tomas Vondra
On 12/24/21 06:37, Kyotaro Horiguchi wrote: At Thu, 23 Dec 2021 19:50:22 +0100, Tomas Vondra wrote in On 12/23/21 15:42, Fujii Masao wrote: On 2021/12/23 3:49, Tomas Vondra wrote: Attached is a patch tweaking WAL logging - in wal_level=minimal we do the same thing as now, in higher levels

Re: An obsolete comment of pg_stat_statements

2021-12-23 Thread Kyotaro Horiguchi
At Mon, 22 Nov 2021 22:50:04 +0800, Julien Rouhaud wrote in > On Mon, Nov 22, 2021 at 2:48 PM Kyotaro Horiguchi > wrote: > > > > At Mon, 22 Nov 2021 15:38:23 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > * queryId is supposed to be a valid value, otherwise this function > > > dosen't >

Re: Be clear about what log_checkpoints emits in the documentation

2021-12-23 Thread Kyotaro Horiguchi
At Thu, 23 Dec 2021 20:56:22 +0530, Bharath Rupireddy wrote in > Hi, > > Currently the documentation of the log_checkpoints GUC says the following: > > Some statistics are included in the log messages, including the number > of buffers written and the time spent writing them. >

Re: Add checkpoint and redo LSN to LogCheckpointEnd log message

2021-12-23 Thread Kyotaro Horiguchi
At Thu, 23 Dec 2021 20:35:54 +0530, Bharath Rupireddy wrote in > Hi, > > It is useful (for debugging purposes) if the checkpoint end message > has the checkpoint LSN and REDO LSN [1]. It gives more context while > analyzing checkpoint-related issues. The pg_controldata gives the last > checkpoi

Re: sequences vs. synchronous replication

2021-12-23 Thread Kyotaro Horiguchi
At Thu, 23 Dec 2021 19:50:22 +0100, Tomas Vondra wrote in > On 12/23/21 15:42, Fujii Masao wrote: > > On 2021/12/23 3:49, Tomas Vondra wrote: > >> Attached is a patch tweaking WAL logging - in wal_level=minimal we do > >> the same thing as now, in higher levels we log every sequence fetch. > > T

Re: row filtering for logical replication

2021-12-23 Thread Peter Smith
The current PG docs text for CREATE PUBLICATION (in the v54-0001 patch) has a part that says + A nullable column in the WHERE clause could cause the + expression to evaluate to false; avoid using columns without not-null + constraints in the WHERE clause. I felt that the caution to "avoid u

Re: Allow escape in application_name

2021-12-23 Thread Kyotaro Horiguchi
At Thu, 23 Dec 2021 23:10:38 +0900, Fujii Masao wrote in > > > On 2021/12/17 16:50, Kyotaro Horiguchi wrote: > > Thus rewriting the code we're focusing on like the following would > > make sense to me. > > > >>if (strcmp(keywords[i], "application_name") == 0) > >>{ > >>val

Re: more descriptive message for process termination due to max_slot_wal_keep_size

2021-12-23 Thread Kyotaro Horiguchi
At Thu, 23 Dec 2021 18:08:08 +0530, Ashutosh Bapat wrote in > On Wed, Dec 15, 2021 at 9:42 AM Kyotaro Horiguchi > wrote: > > > LOG: invalidating slot "s1" > > > DETAIL: The slot's restart_lsn 0/1D68 is behind the limit 0/1100 > > > defined by max_slot_wal_keep_size. > > > > The secon

Re: Emit a warning if the extension's GUC is set incorrectly

2021-12-23 Thread Shinya Kato
On 2021/12/22 3:33, Tom Lane wrote: I wrote: 0003 looks to me like it was superseded by 75d22069e. I do not particularly like that patch though; it seems both inefficient and ill-designed. 0003 is adding a check in an equally bizarre place. Why isn't add_placeholder_variable the place to deal

Re: parallel vacuum comments

2021-12-23 Thread Masahiko Sawada
On Fri, Dec 24, 2021 at 11:59 AM Amit Kapila wrote: > > On Thu, Dec 23, 2021 at 10:56 AM Masahiko Sawada > wrote: > > > > On Wed, Dec 22, 2021 at 10:55 PM Amit Kapila > > wrote: > > > > > > On Wed, Dec 22, 2021 at 6:22 PM Amit Kapila > > > wrote: > > > > > > > > On Wed, Dec 22, 2021 at 5:39

Re: parallel vacuum comments

2021-12-23 Thread Amit Kapila
On Thu, Dec 23, 2021 at 10:56 AM Masahiko Sawada wrote: > > On Wed, Dec 22, 2021 at 10:55 PM Amit Kapila wrote: > > > > On Wed, Dec 22, 2021 at 6:22 PM Amit Kapila wrote: > > > > > > On Wed, Dec 22, 2021 at 5:39 PM houzj.f...@fujitsu.com > > > wrote: > > > > > > > > > > > > 2) > > > > +#include

Re: Buildfarm support for older versions

2021-12-23 Thread Larry Rosenman
On 12/23/2021 11:58 am, Andrew Dunstan wrote: Oh, you need to be building with the buildfarm client's git tip, not the released code. Alternatively, apply this patch: https://github.com/PGBuildFarm/client-code/commit/75c762ba74fdec96ebf6c2433d61d3eeead825c3 with git tip: output attached

Inconsistent ellipsis in regression test error message?

2021-12-23 Thread Peter Smith
The most recent cfbot run for a patch I am interested in has failed a newly added regression test. Please see http://cfbot.cputube.org/ for 36/2906 ~ The failure logs [2] are very curious because the error message is what was expected but it has a different position of the ellipsis (...). But o

Re: Delay the variable initialization in get_rel_sync_entry

2021-12-23 Thread Michael Paquier
On Thu, Dec 23, 2021 at 12:54:41PM -0300, Euler Taveira wrote: > On Wed, Dec 22, 2021, at 10:11 AM, houzj.f...@fujitsu.com wrote: >> The extra cost could sometimes be noticeable because get_rel_sync_entry is a >> hot function which is executed for each change. And the 'am_partition' and >> 'relkind

Fwd: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-23 Thread SATYANARAYANA NARLAPURAM
Please find the attached draft patch. On Thu, Dec 23, 2021 at 2:47 AM Bharath Rupireddy < bharath.rupireddyforpostg...@gmail.com> wrote: > On Thu, Dec 23, 2021 at 5:53 AM SATYANARAYANA NARLAPURAM > wrote: > > > > Hi Hackers, > > > > I am considering implementing RPO (recovery point objective) en

Re: CREATEROLE and role ownership hierarchies

2021-12-23 Thread Joshua Brindle
On Tue, Dec 21, 2021 at 8:26 PM Mark Dilger wrote: > > > > > On Dec 21, 2021, at 5:11 PM, Shinya Kato > > wrote: > > > > I fixed the patches because they cannot be applied to HEAD. > > Thank you. I reviewed and tested these and they LGTM. FYI the rebased v3 patches upthread are raw diffs so git

Re: sequences vs. synchronous replication

2021-12-23 Thread Tomas Vondra
On 12/23/21 15:42, Fujii Masao wrote: On 2021/12/23 3:49, Tomas Vondra wrote: Attached is a patch tweaking WAL logging - in wal_level=minimal we do the same thing as now, in higher levels we log every sequence fetch. Thanks for the patch! With the patch, I found that the regression test for

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2021-12-23 Thread Максим Орлов
> > The pstrdup is unnecessary but harmless. > > > - use the existing table for uniqueness check in 005_opclass_damage.pl > > It appears you still create a new table, bttest_unique, rather than using > the existing table int4tbl. That's fine. > > > - added tests into 003_check.pl . It is only smok

Re: Buildfarm support for older versions

2021-12-23 Thread Andrew Dunstan
On 12/23/21 12:23, Andrew Dunstan wrote: > On 12/23/21 11:27, Larry Rosenman wrote: >>> For the 9.2 error, try setting this in the config_env stanza: >>> >>> >>>     CFLAGS => '-O2 -fPIC', >>> >>> >>> >> That got us further, but it dies on startdb: >> $ cat startdb-C-1.log >> waiting for server t

Re: Buildfarm support for older versions

2021-12-23 Thread Larry Rosenman
On 12/23/2021 11:23 am, Andrew Dunstan wrote: On 12/23/21 11:27, Larry Rosenman wrote: For the 9.2 error, try setting this in the config_env stanza:     CFLAGS => '-O2 -fPIC', That got us further, but it dies on startdb: $ cat startdb-C-1.log waiting for server to start stopped w

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2021-12-23 Thread Justin Pryzby
On Thu, Dec 23, 2021 at 09:14:18AM -0400, Fabien COELHO wrote: > It seems that the v31 patch does not apply anymore: > > postgresql> git apply > ~/v31-0001-Document-historic-behavior-of-links-to-directori.patch > error: patch failed: doc/src/sgml/func.sgml:27410 > error: doc/src/sgml/func.s

Re: Buildfarm support for older versions

2021-12-23 Thread Andrew Dunstan
On 12/23/21 11:27, Larry Rosenman wrote: > >>> >> >> For the 9.2 error, try setting this in the config_env stanza: >> >> >>     CFLAGS => '-O2 -fPIC', >> >> >> > > That got us further, but it dies on startdb: > $ cat startdb-C-1.log > waiting for server to start stopped waiting > pg_ctl: coul

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-23 Thread SATYANARAYANA NARLAPURAM
On Thu, Dec 23, 2021 at 5:18 AM Ashutosh Bapat wrote: > On Thu, Dec 23, 2021 at 5:53 AM SATYANARAYANA NARLAPURAM > wrote: > > > > Hi Hackers, > > > > I am considering implementing RPO (recovery point objective) enforcement > feature for Postgres where the WAL writes on the primary are stalled wh

Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2021-12-23 Thread Mark Dilger
> On Dec 22, 2021, at 12:01 AM, Pavel Borisov wrote: > > Thank you, Mark! > > In v6 (PFA) I've made the changes on your advice i.e. > > - pg_amcheck with --checkunique option will ignore uniqueness check (with a > warning) if amcheck version in a db is <1.4 and doesn't support the feature.

Re: Buildfarm support for older versions

2021-12-23 Thread Larry Rosenman
On 12/23/2021 10:13 am, Andrew Dunstan wrote: On 12/23/21 08:50, Andrew Dunstan wrote: On 12/22/21 23:20, Larry Rosenman wrote: On 12/22/2021 10:15 pm, Tom Lane wrote: Larry Rosenman writes: On 12/22/2021 9:59 pm, Tom Lane wrote: Does it work if you drop --enable-nls?  (It'd likely be worth

Re: Buildfarm support for older versions

2021-12-23 Thread Andrew Dunstan
On 12/23/21 08:50, Andrew Dunstan wrote: > On 12/22/21 23:20, Larry Rosenman wrote: >> On 12/22/2021 10:15 pm, Tom Lane wrote: >>> Larry Rosenman writes: On 12/22/2021 9:59 pm, Tom Lane wrote: > Does it work if you drop --enable-nls?  (It'd likely be worth fixing > if so, but I'm tr

Re: Delay the variable initialization in get_rel_sync_entry

2021-12-23 Thread Euler Taveira
On Wed, Dec 22, 2021, at 10:11 AM, houzj.f...@fujitsu.com wrote: > When reviewing some logical replication patches. I noticed that in > function get_rel_sync_entry() we always invoke get_rel_relispartition() > and get_rel_relkind() at the beginning which could cause unnecessary > cache access. >

Re: correct the sizes of values and nulls arrays in pg_control_checkpoint

2021-12-23 Thread Bharath Rupireddy
On Thu, Dec 23, 2021 at 9:13 PM Euler Taveira wrote: > > On Thu, Dec 23, 2021, at 8:39 AM, Bharath Rupireddy wrote: > > pg_control_checkpoint emits 18 columns whereas the values and nulls > arrays are defined to be of size 19. Although it's not critical, > attaching a tiny patch to fix this. > > G

Re: correct the sizes of values and nulls arrays in pg_control_checkpoint

2021-12-23 Thread Euler Taveira
On Thu, Dec 23, 2021, at 8:39 AM, Bharath Rupireddy wrote: > pg_control_checkpoint emits 18 columns whereas the values and nulls > arrays are defined to be of size 19. Although it's not critical, > attaching a tiny patch to fix this. Good catch! I'm wondering if a constant wouldn't be useful for su

Be clear about what log_checkpoints emits in the documentation

2021-12-23 Thread Bharath Rupireddy
Hi, Currently the documentation of the log_checkpoints GUC says the following: Some statistics are included in the log messages, including the number of buffers written and the time spent writing them. Usage of the word "Some" makes it a vague statement. Why can't we just be clea

Re: Are datcollate/datctype always libc even under --with-icu ?

2021-12-23 Thread Daniel Verite
Chapman Flack wrote: > Next question: the "currently" in that comment suggests that could change, > but is there any present intention to change it, or is this likely to just > be the way it is for the foreseeable future? Some related patches and discussions: * ICU as default collation p

Add checkpoint and redo LSN to LogCheckpointEnd log message

2021-12-23 Thread Bharath Rupireddy
Hi, It is useful (for debugging purposes) if the checkpoint end message has the checkpoint LSN and REDO LSN [1]. It gives more context while analyzing checkpoint-related issues. The pg_controldata gives the last checkpoint LSN and REDO LSN, but having this info alongside the log message helps anal

Add new function to convert 32-bit XID to 64-bit

2021-12-23 Thread Fujii Masao
Hi, When periodically collecting and accumulating statistics or status information like pg_locks, pg_stat_activity, pg_prepared_xacts, etc for future troubleshooting or some reasons, I'd like to store a transaction ID of such information as 64-bit version so that the information of specified t

Re: pg_archivecleanup - add the ability to detect, archive and delete the unneeded wal files on the primary

2021-12-23 Thread Euler Taveira
On Thu, Dec 23, 2021, at 9:58 AM, Bharath Rupireddy wrote: > pg_archivecleanup currently takes a WAL file name as input to delete > the WAL files prior to it [1]. As suggested by Satya (cc-ed) in > pg_replslotdata thread [2], can we enhance the pg_archivecleanup to > automatically detect the last c

Re: sequences vs. synchronous replication

2021-12-23 Thread Fujii Masao
On 2021/12/23 3:49, Tomas Vondra wrote: Attached is a patch tweaking WAL logging - in wal_level=minimal we do the same thing as now, in higher levels we log every sequence fetch. Thanks for the patch! With the patch, I found that the regression test for sequences failed. +

Re: Allow escape in application_name

2021-12-23 Thread Fujii Masao
On 2021/12/17 16:50, Kyotaro Horiguchi wrote: Thus rewriting the code we're focusing on like the following would make sense to me. if (strcmp(keywords[i], "application_name") == 0) { values[i] = process_pgfdw_appname(values[i]); /*

Re: Buildfarm support for older versions

2021-12-23 Thread Andrew Dunstan
On 12/22/21 23:20, Larry Rosenman wrote: > On 12/22/2021 10:15 pm, Tom Lane wrote: >> Larry Rosenman writes: >>> On 12/22/2021 9:59 pm, Tom Lane wrote: Does it work if you drop --enable-nls?  (It'd likely be worth fixing if so, but I'm trying to narrow the possible causes.) >> >>> Nope

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-23 Thread Ashutosh Bapat
On Thu, Dec 23, 2021 at 5:53 AM SATYANARAYANA NARLAPURAM wrote: > > Hi Hackers, > > I am considering implementing RPO (recovery point objective) enforcement > feature for Postgres where the WAL writes on the primary are stalled when the > WAL distance between the primary and standby exceeds the

Re: pg_ls_tmpdir to show directories and shared filesets (and pg_ls_*)

2021-12-23 Thread Fabien COELHO
Hello Justin, It seems that the v31 patch does not apply anymore: postgresql> git apply ~/v31-0001-Document-historic-behavior-of-links-to-directori.patch error: patch failed: doc/src/sgml/func.sgml:27410 error: doc/src/sgml/func.sgml: patch does not apply -- Fabien.

Re: Multi-Column List Partitioning

2021-12-23 Thread Amul Sul
On Tue, Dec 21, 2021 at 6:34 PM Nitin Jadhav wrote: > > --- > > > + if (isnulls && isnulls[i]) > > + cmpval = 0; /* NULL "=" NULL */ > > + else > > + cmpval = 1; /* NULL ">" not-NULL */ > > + } > > + else if (isnulls && isnulls[i]

pg_archivecleanup - add the ability to detect, archive and delete the unneeded wal files on the primary

2021-12-23 Thread Bharath Rupireddy
Hi, pg_archivecleanup currently takes a WAL file name as input to delete the WAL files prior to it [1]. As suggested by Satya (cc-ed) in pg_replslotdata thread [2], can we enhance the pg_archivecleanup to automatically detect the last checkpoint (from control file) LSN, calculate the lowest restar

Re: more descriptive message for process termination due to max_slot_wal_keep_size

2021-12-23 Thread Ashutosh Bapat
On Wed, Dec 15, 2021 at 9:42 AM Kyotaro Horiguchi wrote: > > At Tue, 14 Dec 2021 19:31:21 +0530, Ashutosh Bapat > wrote in > > On Tue, Dec 14, 2021 at 9:35 AM Kyotaro Horiguchi > > wrote: > > > > [17605] LOG: terminating process 17614 to release replication slot "s1" > > > + [17605] DETAIL: T

回复:回复:Re: Re: 回复:Re: Is it worth pushing conditions to sublink/subplan?

2021-12-23 Thread 曾文旌(义从)
Fixed a bug found during testing. Wenjing --原始邮件 -- 发件人:曾文旌(义从) 发送时间:Sun Dec 12 20:51:08 2021 收件人:Zhihong Yu 抄送:Tomas Vondra , wjzeng , PostgreSQL Hackers , shawn wang , ggys...@gmail.com 主题:回复:Re: Re: 回复:Re: Is it worth pushing conditions to sublink/subp

Re: psql - add SHOW_ALL_RESULTS option

2021-12-23 Thread Fabien COELHO
Hello Peter, I finally took some time to look at this. Attached v11 is a rebase. This patch still has a few of the problems reported earlier this year. In [0], it was reported that certain replication commands result in infinite loops because of faulty error handling. This still happens.

correct the sizes of values and nulls arrays in pg_control_checkpoint

2021-12-23 Thread Bharath Rupireddy
Hi, pg_control_checkpoint emits 18 columns whereas the values and nulls arrays are defined to be of size 19. Although it's not critical, attaching a tiny patch to fix this. diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 209a20a882..b1db9a8d07

skip replication slot snapshot/map file removal during end-of-recovery checkpoint

2021-12-23 Thread Bharath Rupireddy
Hi, Currently the end-of-recovery checkpoint can be much slower, impacting the server availability, if there are many replication slot files .snap or map- to be enumerated and deleted. How about skipping the .snap and map- file handling during the end-of-recovery checkpoint? It makes the s

Re: Logical replication timeout problem

2021-12-23 Thread Amit Kapila
On Wed, Dec 22, 2021 at 8:50 PM Fabrice Chapuis wrote: > > Hello Amit, > > I was able to reproduce the timeout problem in the lab. > After loading more than 20 millions of rows in a table which is not > replicated (insert command ends without error), errors related to logical > replication proce

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-23 Thread Andrey Lepikhov
On 21/12/2021 00:05, Peter Geoghegan wrote: * Some index AMs don't work like nbtree and GiST in that they cannot do their scan sequentially -- they have to do something like a logical/keyspace order scan instead, which is *totally* different to heapam (not just a bit different). There is no telli

Re: Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-23 Thread Bharath Rupireddy
On Thu, Dec 23, 2021 at 5:53 AM SATYANARAYANA NARLAPURAM wrote: > > Hi Hackers, > > I am considering implementing RPO (recovery point objective) enforcement > feature for Postgres where the WAL writes on the primary are stalled when the > WAL distance between the primary and standby exceeds the

RE: Failed transaction statistics to measure the logical replication progress

2021-12-23 Thread wangw.f...@fujitsu.com
On Wednesday, December 22, 2021 10:30 PM osumi.takami...@fujitsu.com wrote: > On Wednesday, December 22, 2021 8:38 PM I wrote: > > Do we expect these commit counts which come from empty transactions ? > This is another issue discussed in [1] > where the patch in the thread is a work in progress, I

Re: Add index scan progress to pg_stat_progress_vacuum

2021-12-23 Thread Masahiko Sawada
On Tue, Dec 21, 2021 at 3:37 AM Peter Geoghegan wrote: > > On Wed, Dec 15, 2021 at 2:10 PM Bossart, Nathan wrote: > > nitpick: Shouldn't index_blks_scanned be index_blks_vacuumed? IMO it > > is more analogous to heap_blks_vacuumed. > > +1. > > > This will tell us which indexes are currently bein

Re: row filtering for logical replication

2021-12-23 Thread Peter Smith
On Thu, Dec 23, 2021 at 7:23 PM Peter Smith wrote: > > Here is the v54* patch set: > > Main changes from v53* are > 1. All files of all three patches have been pgindented. > 2. Another review comment is addressed > > ~~ > > Details > === > > v51-0001 (main) > - pgindent for all source files if