Re: [PATCH] Improve amcheck to also check UNIQUE constraint in btree index.

2021-12-22 Thread Pavel Borisov
> > The tests in check_btree.sql no longer create a bttest_unique table, so > the DROP TABLE is surplusage: > > +DROP TABLE bttest_unique; > +ERROR: table "bttest_unique" does not exist > > > The changes in pg_amcheck.c to pass the new checkunique parameter will > likely need to be based on a amch

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-22 Thread Ashutosh Sharma
Hi Dilip, On Tue, Dec 21, 2021 at 11:10 AM Ashutosh Sharma wrote: > I am getting the below error when running the same test-case that Neha > shared in her previous email. > > ERROR: 55000: some relations of database "test1" are already in > tablespace "tab1" > HINT: You must move them back to

fix crash with Python 3.11

2021-12-22 Thread Peter Eisentraut
This patch needs another close pass and possibly some refactoring to avoid copy-and-paste, but I'm putting this out here, since people are already testing with Python 3.11 and will surely run into this problem. The way plpy.commit() and plpy.rollback() handle errors is not ideal. They end up

Re: Checkpointer crashes with "PANIC: could not fsync file "pg_tblspc/.."

2021-12-22 Thread Ashutosh Sharma
On Wed, Dec 22, 2021 at 7:20 AM Dilip Kumar wrote: > On Wed, 22 Dec 2021 at 12:28 AM, Ashutosh Sharma > wrote: > >> >> Is it okay to share the same tablespace (infact relfile) between the >> primary and standby server? Perhaps NO. >> > >> Oops, yeah absolutely they can never share the tablespace

RE: In-placre persistance change of a relation

2021-12-22 Thread Jakub Wartak
Hi Kyotaro, > At Tue, 21 Dec 2021 13:07:28 +, Jakub Wartak > wrote in > > So what's suspicious is that 122880 -> 0 file size truncation. I've > > investigated WAL and it seems to contain TRUNCATE records after logged > FPI images, so when the crash recovery would kick in it probably clears th

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-22 Thread Dilip Kumar
On Tue, Dec 21, 2021 at 11:10 AM Ashutosh Sharma wrote: > > I am getting the below error when running the same test-case that Neha shared > in her previous email. > > ERROR: 55000: some relations of database "test1" are already in tablespace > "tab1" > HINT: You must move them back to the data

RE: Failed transaction statistics to measure the logical replication progress

2021-12-22 Thread osumi.takami...@fujitsu.com
On Tuesday, December 21, 2021 6:00 PM Greg Nancarrow wrote: > Some review comments on the v18 patches: Thank you for your review ! > v18-0002 > > doc/src/sgml/monitoring.sgml > (1) tablesync worker stats? > > Shouldn't the comment below only mention the apply worker? (since we're no > longer r

RE: Optionally automatically disable logical replication subscriptions on error

2021-12-22 Thread osumi.takami...@fujitsu.com
On Tuesday, December 21, 2021 11:18 PM I wrote: > On Thursday, December 16, 2021 9:51 PM I wrote: > > Attached the updated patch v14. > FYI, I've conducted a test of disable_on_error flag using pg_upgrade. I > prepared PG14 and HEAD applied with disable_on_error patch. > Then, I setup a logical re

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-22 Thread Ashutosh Sharma
On Wed, Dec 22, 2021 at 2:44 PM Dilip Kumar wrote: > On Tue, Dec 21, 2021 at 11:10 AM Ashutosh Sharma > wrote: > > > > I am getting the below error when running the same test-case that Neha > shared in her previous email. > > > > ERROR: 55000: some relations of database "test1" are already in >

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-22 Thread Dilip Kumar
On Wed, Dec 22, 2021 at 4:26 PM Ashutosh Sharma wrote: >> Basically, ALTER TABLE SET TABLESPACE, will register the >> SYNC_UNLINK_REQUEST for the table files w.r.t the old tablespace, but >> those will get unlinked during the next checkpoint. Although the >> files must be truncated during commit

RE: parallel vacuum comments

2021-12-22 Thread houzj.f...@fujitsu.com
On Wed, Dec 22, 2021 11:36 AM Masahiko Sawada wrote: > On Tue, Dec 21, 2021 at 10:24 PM Amit Kapila > wrote: > > > > On Tue, Dec 21, 2021 at 11:24 AM Masahiko Sawada > wrote: > > > > > > On Tue, Dec 21, 2021 at 2:04 PM Amit Kapila > wrote: > > > > > > > > > > Thank you for the comment. Agreed.

Re: sequences vs. synchronous replication

2021-12-22 Thread Tomas Vondra
On 12/22/21 05:56, Fujii Masao wrote: On 2021/12/22 10:57, Tomas Vondra wrote: On 12/19/21 04:03, Amit Kapila wrote: On Sat, Dec 18, 2021 at 7:24 AM Tomas Vondra wrote: while working on logical decoding of sequences, I ran into an issue with nextval() in a transaction that rolls bac

RE: Failed transaction statistics to measure the logical replication progress

2021-12-22 Thread wangw.f...@fujitsu.com
On Mon, Dec 22, 2021 at 6:14 PM osumi.takami...@fujitsu.com wrote: > > Attached the new patch v19. > I have a question on the v19-0002 patch: When I tested for this patch, I found pg_stat_subscription_workers has some unexpected data. For example: [Publisher] create table replica_test1(a int,

Re: parallel vacuum comments

2021-12-22 Thread Amit Kapila
On Wed, Dec 22, 2021 at 5:39 PM houzj.f...@fujitsu.com wrote: > > On Wed, Dec 22, 2021 11:36 AM Masahiko Sawada wrote: > > On Tue, Dec 21, 2021 at 10:24 PM Amit Kapila > > wrote: > > The patch looks mostly good to me. > I only have few comments. > > 1) > +/* > + * Do parallel index bulk-deletion

Delay the variable initialization in get_rel_sync_entry

2021-12-22 Thread houzj.f...@fujitsu.com
Hi, When reviewing some logical replication patches. I noticed that in function get_rel_sync_entry() we always invoke get_rel_relispartition() and get_rel_relkind() at the beginning which could cause unnecessary cache access. --- get_rel_sync_entry(PGOutputData *data, Oid relid) { Relati

Re: Clarifying/rationalizing Vars' varno/varattno/varnoold/varoattno

2021-12-22 Thread Andrey Lepikhov
On 5/2/2020 01:24, Tom Lane wrote: I've not written any actual code, but am close to being ready to. This thread gives us hope to get started on solving some of the basic planner problems. But there is no activity for a long time, as I see. Have You tried to implement this idea? Is it actual no

Re: Buildfarm support for older versions

2021-12-22 Thread Andrew Dunstan
On 12/21/21 15:06, Larry Rosenman wrote: > I filled out that form on the 16th, and haven't gotten a new animal > assignment.  Is there > a problem with my data? It's a manual process, done when your friendly admins have time. I have approved it now. cheers andrew -- Andrew Dunstan EDB: ht

Re: Replication slot drop message is sent after pgstats shutdown.

2021-12-22 Thread Masahiko Sawada
On Mon, Dec 13, 2021 at 12:11 PM Kyotaro Horiguchi wrote: > > At Fri, 10 Dec 2021 18:13:31 +0900, Masahiko Sawada > wrote in > > I agreed with Andres and Horiguchi-san and attached an updated patch. > > Thanks for the new version. > > It seems fine, but I have some comments from a cosmetic viewp

[Proposal][WIP] Add option to log auto_explain output to separate logfile

2021-12-22 Thread Timofey
Hello, hackers! Now, all of auto_explain output is directed to postgres's log and it is not comfortably to extract on big highloaded systems. My proposal is add option to auto_explain to log data to separate logfile. In my patch I plan to (re)open file every time associated guc variable is cha

Re: parallel vacuum comments

2021-12-22 Thread Amit Kapila
On Wed, Dec 22, 2021 at 6:22 PM Amit Kapila wrote: > > On Wed, Dec 22, 2021 at 5:39 PM houzj.f...@fujitsu.com > wrote: > > > > > > 2) > > +#include "utils/rel.h" > > +#include "utils/lsyscache.h" > > +#include "utils/memutils.h" > > > > It might be better to keep the header file in alphabetical o

RE: Failed transaction statistics to measure the logical replication progress

2021-12-22 Thread osumi.takami...@fujitsu.com
On Wednesday, December 22, 2021 9:38 PM Wang, Wei/王 威 wrote: > I have a question on the v19-0002 patch: > > When I tested for this patch, I found pg_stat_subscription_workers has some > unexpected data. > For example: > [Publisher] > create table replica_test1(a int, b text); create publication

Re: [Proposal] Fully WAL logged CREATE DATABASE - No Checkpoints

2021-12-22 Thread Ashutosh Sharma
On Wed, Dec 22, 2021 at 5:06 PM Dilip Kumar wrote: > > On Wed, Dec 22, 2021 at 4:26 PM Ashutosh Sharma wrote: > > >> Basically, ALTER TABLE SET TABLESPACE, will register the > >> SYNC_UNLINK_REQUEST for the table files w.r.t the old tablespace, but > >> those will get unlinked during the next che

Re: [Proposal][WIP] Add option to log auto_explain output to separate logfile

2021-12-22 Thread Pavel Stehule
Hi st 22. 12. 2021 v 14:54 odesílatel Timofey napsal: > Hello, hackers! > > Now, all of auto_explain output is directed to postgres's log and it is > not comfortably to extract on big highloaded systems. > My proposal is add option to auto_explain to log data to separate > logfile. In my patch I

Re: pg_upgrade should truncate/remove its logs before running

2021-12-22 Thread Tom Lane
Michael Paquier writes: > On Mon, Dec 20, 2021 at 09:39:26PM -0600, Justin Pryzby wrote: >> Are you suggesting to remove these ? >> -/pg_upgrade_internal.log >> -/loadable_libraries.txt > Yep, it looks so as these are part of the logs, the second one being a > failure state. >> -/reindex_hash.sq

Re: do only critical work during single-user vacuum?

2021-12-22 Thread John Naylor
On Tue, Dec 21, 2021 at 10:39 PM Masahiko Sawada wrote: > > On Wed, Dec 22, 2021 at 6:56 AM Peter Geoghegan wrote: > > > > This new command/facility should probably not be a new flag to the > > VACUUM command, as such. Rather, I think that it should either be an > > SQL-callable function, or a de

Re: Logical replication timeout problem

2021-12-22 Thread Fabrice Chapuis
Hello Amit, I was able to reproduce the timeout problem in the lab. After loading more than 20 millions of rows in a table which is not replicated (insert command ends without error), errors related to logical replication processes appear in the postgres log. Approximately every 5 minutes worker p

Re: Clarifying/rationalizing Vars' varno/varattno/varnoold/varoattno

2021-12-22 Thread Tom Lane
Andrey Lepikhov writes: > On 5/2/2020 01:24, Tom Lane wrote: >> I've not written any actual code, but am close to being ready to. > This thread gives us hope to get started on solving some of the basic > planner problems. > But there is no activity for a long time, as I see. Have You tried to >

Re: pg_upgrade should truncate/remove its logs before running

2021-12-22 Thread Justin Pryzby
On Wed, Dec 22, 2021 at 09:52:26AM -0500, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Dec 20, 2021 at 09:39:26PM -0600, Justin Pryzby wrote: > >> Are you suggesting to remove these ? > >> -/pg_upgrade_internal.log > >> -/loadable_libraries.txt > > > Yep, it looks so as these are part of

Re: Clarifying/rationalizing Vars' varno/varattno/varnoold/varoattno

2021-12-22 Thread Andrey Lepikhov
On 22/12/2021 20:42, Tom Lane wrote: Andrey Lepikhov writes: On 5/2/2020 01:24, Tom Lane wrote: I've not written any actual code, but am close to being ready to. This thread gives us hope to get started on solving some of the basic planner problems. But there is no activity for a long time,

Re: sequences vs. synchronous replication

2021-12-22 Thread Fujii Masao
On 2021/12/22 21:11, Tomas Vondra wrote: Interesting idea, but I think it has a couple of issues :-( Thanks for the review! 1) We'd need to know the LSN of the last WAL record for any given sequence, and we'd need to communicate that between backends somehow. Which seems rather tricky to

Re: sequences vs. synchronous replication

2021-12-22 Thread Tomas Vondra
On 12/21/21 03:49, Tomas Vondra wrote: On 12/21/21 02:01, Tom Lane wrote: Tomas Vondra writes: OK, I did a quick test with two very simple benchmarks - simple select from a sequence, and 'pgbench -N' on scale 1. Benchmark was on current master, patched means SEQ_LOG_VALS was set to 1. But

Re: sequences vs. synchronous replication

2021-12-22 Thread Tomas Vondra
On 12/22/21 18:50, Fujii Masao wrote: On 2021/12/22 21:11, Tomas Vondra wrote: Interesting idea, but I think it has a couple of issues :-( Thanks for the review! 1) We'd need to know the LSN of the last WAL record for any given sequence, and we'd need to communicate that between backends s

Re: track_io_timing default setting

2021-12-22 Thread Stephen Frost
Greetings, * Laurenz Albe (laurenz.a...@cybertec.at) wrote: > On Fri, 2021-12-10 at 10:20 -0500, Tom Lane wrote: > > Jeff Janes writes: > > > Can we change the default setting of track_io_timing to on? > > > > That adds a very significant amount of overhead on some platforms > > (gettimeofday is

Re: track_io_timing default setting

2021-12-22 Thread Peter Geoghegan
On Wed, Dec 22, 2021 at 11:16 AM Stephen Frost wrote: > > I set "track_io_timing" to "on" all the time, same as "log_lock_waits", > > so I'd want them both on by default. > > Same. I'd also push back and ask what modern platforms still require a > kernel call for gettimeofday, and are we really d

Re: \d with triggers: more than one row returned by a subquery used as an expression

2021-12-22 Thread Justin Pryzby
On Fri, Dec 17, 2021 at 09:43:56AM -0600, Justin Pryzby wrote: > I want to mention that the 2nd problem I mentioned here is still broken. > https://www.postgresql.org/message-id/20210717010259.gu20...@telsasoft.com > > It happens if non-inheritted triggers on child and parent have the same name.

Re: do only critical work during single-user vacuum?

2021-12-22 Thread Peter Geoghegan
On Tue, Dec 21, 2021 at 6:39 PM Masahiko Sawada wrote: > Even not in the situation where the database has to run as the > single-user mode to freeze tuples, I think there would be some use > cases where users want to run vacuum (in failsafe mode) on tables with > relfrozenxid/relminmxid greater th

Are datcollate/datctype always libc even under --with-icu ?

2021-12-22 Thread Chapman Flack
... ok, I see that the answer is yes, according to the commit comment for eccfef8: Currently, ICU-provided collations can only be explicitly named collations. The global database locales are still always libc-provided. I got there the long way, by first wondering how to tell whether a datcol

Re: Unifying VACUUM VERBOSE and log_autovacuum_min_duration output

2021-12-22 Thread Peter Geoghegan
On Tue, Dec 21, 2021 at 9:46 PM Greg Stark wrote: > Or rather I think a better way to look at it is that the progress > output for the operator should be separated from the metrics logged. > As an operator what I want to see is some progress indicator > ""starting table scan", "overflow at x% of t

Throttling WAL inserts when the standby falls behind more than the configured replica_lag_in_bytes

2021-12-22 Thread SATYANARAYANA NARLAPURAM
Hi Hackers, I am considering implementing RPO (recovery point objective) enforcement feature for Postgres where the WAL writes on the primary are stalled when the WAL distance between the primary and standby exceeds the configured (replica_lag_in_bytes) threshold. This feature is useful particular

Re: Buildfarm support for older versions

2021-12-22 Thread Larry Rosenman
On 12/22/2021 7:20 am, Andrew Dunstan wrote: On 12/21/21 15:06, Larry Rosenman wrote: I filled out that form on the 16th, and haven't gotten a new animal assignment.  Is there a problem with my data? It's a manual process, done when your friendly admins have time. I have approved it now. c

Re: Delay the variable initialization in get_rel_sync_entry

2021-12-22 Thread Kyotaro Horiguchi
At Wed, 22 Dec 2021 13:11:38 +, "houzj.f...@fujitsu.com" wrote in > Hi, > > When reviewing some logical replication patches. I noticed that in > function get_rel_sync_entry() we always invoke get_rel_relispartition() > and get_rel_relkind() at the beginning which could cause unnecessary >

Re: Buildfarm support for older versions

2021-12-22 Thread Larry Rosenman
On 12/22/2021 7:16 pm, Larry Rosenman wrote: On 12/22/2021 7:20 am, Andrew Dunstan wrote: On 12/21/21 15:06, Larry Rosenman wrote: I filled out that form on the 16th, and haven't gotten a new animal assignment.  Is there a problem with my data? It's a manual process, done when your friendly

RE: parallel vacuum comments

2021-12-22 Thread houzj.f...@fujitsu.com
On Wed, Dec 22, 2021 9:55 PM Amit Kapila wrote: > On Wed, Dec 22, 2021 at 6:22 PM Amit Kapila > wrote: > > > > On Wed, Dec 22, 2021 at 5:39 PM houzj.f...@fujitsu.com > > wrote: > > > > > > > > > 2) > > > +#include "utils/rel.h" > > > +#include "utils/lsyscache.h" > > > +#include "utils/memutils.

Re: Buildfarm support for older versions

2021-12-22 Thread Tom Lane
Larry Rosenman writes: > REL9_2_STABLE make dies on: > ld: error: relocation R_X86_64_PC32 cannot be used against symbol > _CurrentRuneLocale; recompile with -fPIC > [etc] What configure options did you use? regards, tom lane

Re: Buildfarm support for older versions

2021-12-22 Thread Larry Rosenman
On 12/22/2021 9:34 pm, Tom Lane wrote: Larry Rosenman writes: REL9_2_STABLE make dies on: ld: error: relocation R_X86_64_PC32 cannot be used against symbol _CurrentRuneLocale; recompile with -fPIC [etc] What configure options did you use? regards, tom lane config_op

Re: Buildfarm support for older versions

2021-12-22 Thread Tom Lane
Larry Rosenman writes: > On 12/22/2021 9:34 pm, Tom Lane wrote: >> What configure options did you use? > config_opts =>[ > qw( >--enable-cassert >--enable-debug >--enable-nls >--enable-tap-tests >--with-perl >) >

Re: Buildfarm support for older versions

2021-12-22 Thread Larry Rosenman
On 12/22/2021 9:59 pm, Tom Lane wrote: Larry Rosenman writes: On 12/22/2021 9:34 pm, Tom Lane wrote: What configure options did you use? config_opts =>[ qw( --enable-cassert --enable-debug --enable-nls --enable-tap-tests --with

Re: Buildfarm support for older versions

2021-12-22 Thread Tom Lane
Larry Rosenman writes: > On 12/22/2021 9:59 pm, Tom Lane wrote: >> Does it work if you drop --enable-nls? (It'd likely be worth fixing >> if so, but I'm trying to narrow the possible causes.) > Nope... OK. Since 9.3 succeeds, it seems like it's a link problem we fixed at some point. Can you b

Re: Buildfarm support for older versions

2021-12-22 Thread Larry Rosenman
On 12/22/2021 10:15 pm, Tom Lane wrote: Larry Rosenman writes: On 12/22/2021 9:59 pm, Tom Lane wrote: Does it work if you drop --enable-nls? (It'd likely be worth fixing if so, but I'm trying to narrow the possible causes.) Nope... OK. Since 9.3 succeeds, it seems like it's a link probl

Re: parallel vacuum comments

2021-12-22 Thread Masahiko Sawada
On Wed, Dec 22, 2021 at 10:55 PM Amit Kapila wrote: > > On Wed, Dec 22, 2021 at 6:22 PM Amit Kapila wrote: > > > > On Wed, Dec 22, 2021 at 5:39 PM houzj.f...@fujitsu.com > > wrote: > > > > > > > > > 2) > > > +#include "utils/rel.h" > > > +#include "utils/lsyscache.h" > > > +#include "utils/memut

Re: In-placre persistance change of a relation

2021-12-22 Thread Kyotaro Horiguchi
At Wed, 22 Dec 2021 08:42:14 +, Jakub Wartak wrote in > I think there's slight omission: ... > apparently reindex_index() params cannot be NULL - the same happens with > switching persistent Hmm. a3dc926009 has changed the interface. (But the name is also changed after that.) -reindex_rel

Re: In-placre persistance change of a relation

2021-12-22 Thread Kyotaro Horiguchi
At Thu, 23 Dec 2021 15:01:41 +0900 (JST), Kyotaro Horiguchi wrote in > I added TAP test to excecise the in-place persistence change. We don't need a base table for every index. TAP test revised. regards. -- Kyotaro Horiguchi NTT Open Source Software Center >From 112c077561bb24a0b40995e2d6ada