Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread Masahiko Sawada
On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > I think we should not reinterpret the severity of the error and lower > > it. Especially, in this case, any kind of errors can be thrown. It > > could be such a serious error that FDW developer wants t

Re: Decoding of two-phase xacts missing from CREATE_REPLICATION_SLOT command

2021-06-08 Thread Ajin Cherian
On Mon, Jun 7, 2021 at 3:17 PM Amit Kapila wrote: > Pushed the above patch. Here's an updated patchset that adds back in the option for two-phase in CREATE_REPLICATION_SLOT command and a second patch that adds support for two-phase decoding in pg_recvlogical. regards, Ajin Cherian v2-0001-Add

RE: locking [user] catalog tables vs 2pc vs logical rep

2021-06-08 Thread osumi.takami...@fujitsu.com
On Monday, June 7, 2021 6:22 PM Amit Kapila wrote: > On Mon, Jun 7, 2021 at 10:44 AM Amit Kapila > wrote: > > > > One more comment is that for HEAD, first just create a patch with > > synchronous replication-related doc changes and then write a separate > > patch for prepared transactions. > > >

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread Kyotaro Horiguchi
(I have caught up here. Sorry in advance for possible pointless discussion by me..) At Tue, 8 Jun 2021 00:47:08 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Masahiko Sawada > > I think we should not reinterpret the severity of the error and lower > > it. Especially, in this case, any

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread Kyotaro Horiguchi
At Tue, 8 Jun 2021 16:32:14 +0900, Masahiko Sawada wrote in > On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com > wrote: > > > > From: Masahiko Sawada > > > I think we should not reinterpret the severity of the error and lower > > > it. Especially, in this case, any kind of errors can

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com > wrote: > > Why does the client have to know the error on a remote server, whereas the > global transaction itself is destined to commit? > > It's not necessarily on a remote server. It could be a problem with >

RE: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread tsunakawa.ta...@fujitsu.com
From: Kyotaro Horiguchi > I think the discussion is based the behavior that any process that is > responsible for finishing the 2pc-commit continue retrying remote > commits until all of the remote-commits succeed. Thank you for coming back. We're talking about the first attempt to prepare and

Re: Race condition in recovery?

2021-06-08 Thread Dilip Kumar
On Tue, Jun 8, 2021 at 11:13 AM Dilip Kumar wrote: > > # Wait until the node exits recovery. > $standby->poll_query_until('postgres', "SELECT pg_is_in_recovery() = 'f';") > or die "Timed out while waiting for promotion"; > > I will try to generate a version for 9.6 based on this idea and see how i

Re: Duplicate history file?

2021-06-08 Thread Tatsuro Yamada
Hi Horiguchi-san, This thread should have been started here: https://www.postgresql.org/message-id/20210531.165825.921389284096975508.horikyota.ntt%40gmail.com (To recap: In a replication set using archive, startup tries to restore WAL files from archive before checking pg_wal directory for t

Remove unused code from the KnownAssignedTransactionIdes submodule

2021-06-08 Thread Quan Zongliang
Hi, In the KnownAssignedTransactionIdes sub-module, two lines of unused code were found in a previous change. -- Quan Zongliang CPUG diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index b448533564..99ce8c752a 100644 --- a/src/backend/storage/ipc/procar

Unused function parameter in get_qual_from_partbound()

2021-06-08 Thread houzj.f...@fujitsu.com
Hi, I noticed that the first function parameter in get_qual_from_partbound(**Relation rel**, Relation parent, is not used in the function. Is it better to remove it like the attached patch ? Best regards, houzj 0001-remove-unused-function-parameter-in-get_qual_from_pa.patch Description: 0001

Re: Error on pgbench logs

2021-06-08 Thread Kyotaro Horiguchi
At Tue, 8 Jun 2021 12:09:47 +0900, YoungHwan Joo wrote in > Hello! > > While I was using pgbench from the master branch, I discovered an error on > pgbench logs. > When I run pgbench, the log file contains a lot of redundant 0s. > > I ran git bisect and found out that this error occured since t

GiST operator class for bool

2021-06-08 Thread Emre Hasegeli
It could be useful to use bool in exclusion constraints, but it's currently not nicely supported. The attached patch adds support for bool to the btree_gist extension, so we can do this. I am adding this to the commitfest 2021-07. 0001-btree_gist-Support-bool.patch Description: Binary data

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Sun, Jun 06, 2021 at 11:00:38AM -0700, Peter Geoghegan wrote: > On Sun, Jun 6, 2021 at 9:35 AM Justin Pryzby wrote: > > I'll leave the instance running for a little bit before restarting (or > > kill-9) > > in case someone requests more info. > > How about dumping the page image out, and shar

Re: GiST operator class for bool

2021-06-08 Thread Andrey Borodin
Hi! > 8 июня 2021 г., в 13:48, Emre Hasegeli написал(а): > > It could be useful to use bool in exclusion constraints, but it's > currently not nicely supported. The attached patch adds support for > bool to the btree_gist extension, so we can do this. > > I am adding this to the commitfest 202

Re: Unused function parameter in get_qual_from_partbound()

2021-06-08 Thread David Rowley
On Tue, 8 Jun 2021 at 21:50, houzj.f...@fujitsu.com wrote: > I noticed that the first function parameter in > get_qual_from_partbound(**Relation rel**, Relation parent, > is not used in the function. > > Is it better to remove it like the attached patch ? Going by [1] it was used when it went in

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Sun, Jun 06, 2021 at 01:59:10PM -0400, Tom Lane wrote: > Matthias van de Meent writes: > > On Sun, 6 Jun 2021 at 18:35, Justin Pryzby wrote: > >> However, I also found an autovacuum chewing 100% CPU, and it appears the > >> problem is actually because autovacuum has locked a page of pg-statist

Re: Decoding speculative insert with toast leaks memory

2021-06-08 Thread Dilip Kumar
On Mon, Jun 7, 2021 at 6:45 PM Dilip Kumar wrote: >> >> >> 2. In the test, there seems to be an assumption that we can unlock s2 >> and s3 one after another, and then both will start waiting on s-1 but >> isn't it possible that before s2 start waiting on s1, s3 completes its >> insertion and then

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Matthias van de Meent
On Tue, 8 Jun 2021 at 13:03, Justin Pryzby wrote: > > On Sun, Jun 06, 2021 at 11:00:38AM -0700, Peter Geoghegan wrote: > > On Sun, Jun 6, 2021 at 9:35 AM Justin Pryzby wrote: > > > I'll leave the instance running for a little bit before restarting (or > > > kill-9) > > > in case someone requests

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 01:54:41PM +0200, Matthias van de Meent wrote: > On Tue, 8 Jun 2021 at 13:03, Justin Pryzby wrote: > > > > On Sun, Jun 06, 2021 at 11:00:38AM -0700, Peter Geoghegan wrote: > > > On Sun, Jun 6, 2021 at 9:35 AM Justin Pryzby wrote: > > > > I'll leave the instance running for

Re: Logical replication keepalive flood

2021-06-08 Thread Abbas Butt
Hi Kyotaro, I have tried to test your patches. Unfortunately even after applying the patches the WAL Sender is still sending too frequent keepalive messages. In my opinion the fix is to make sure that wal_sender_timeout/2 has passed before sending the keepalive message in the code fragment I had sh

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Matthias van de Meent
On Tue, 8 Jun 2021 at 14:11, Justin Pryzby wrote: > > On Tue, Jun 08, 2021 at 01:54:41PM +0200, Matthias van de Meent wrote: > > On Tue, 8 Jun 2021 at 13:03, Justin Pryzby wrote: > > > > > > On Sun, Jun 06, 2021 at 11:00:38AM -0700, Peter Geoghegan wrote: > > > > On Sun, Jun 6, 2021 at 9:35 AM Ju

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-06-08 Thread vignesh C
On Tue, Jun 8, 2021 at 1:34 PM osumi.takami...@fujitsu.com wrote: > > On Monday, June 7, 2021 6:22 PM Amit Kapila wrote: > > On Mon, Jun 7, 2021 at 10:44 AM Amit Kapila > > wrote: > > > > > > One more comment is that for HEAD, first just create a patch with > > > synchronous replication-related

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 02:27:14PM +0200, Matthias van de Meent wrote: > Thanks for the information! I created an apparently-complete core file by first doing this: | echo 127 |sudo tee /proc/22591/coredump_filter *and updated wiki:Developer_FAQ to work with huge pages I'm planning to kill the p

Re: Make unlogged table resets detectable

2021-06-08 Thread Robert Haas
On Mon, Jun 7, 2021 at 11:46 PM Michael Paquier wrote: > On Mon, Jun 07, 2021 at 02:56:57PM -0400, Tom Lane wrote: > > +1. I'd support recording the time of the last crash recovery, as > > well as having a counter. I think an LSN would not be as useful > > as a timestamp. > > One could guess a t

Re: automatically generating node support functions

2021-06-08 Thread David Rowley
On Tue, 8 Jun 2021 at 08:28, Peter Eisentraut wrote: > > I wrote a script to automatically generate the node support functions > (copy, equal, out, and read, as well as the node tags enum) from the > struct definitions. Thanks for working on this. I agree that it would be nice to see improvements

Re: SSL SNI

2021-06-08 Thread Peter Eisentraut
On 08.06.21 08:54, Michael Paquier wrote: On Mon, Jun 07, 2021 at 11:34:24AM -0400, Tom Lane wrote: Yeah, I'd include the empty-string test just because it's standard practice in this area of libpq. Whether those tests are actually triggerable in every case is obscure, but ... Checking after

Re: Misplaced superuser check in pg_log_backend_memory_contexts()

2021-06-08 Thread Fujii Masao
On 2021/06/08 11:49, Michael Paquier wrote: On Sun, Jun 06, 2021 at 11:13:40AM -0400, Tom Lane wrote: Julien Rouhaud writes: However +1 for the patch, as it seems more consistent to always get a permission failure if you're not a superuser. Yeah, it's just weird if such a check is not the

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-06-08 Thread Alvaro Herrera
On 2021-Jun-06, Michael Paquier wrote: > On Fri, Jun 04, 2021 at 07:21:11PM -0400, Alvaro Herrera wrote: > The test has been done with four configurations, and here are the > results: > 1) HEAD: 9659ms > 2) REL_13_STABLE: 8310ms. > 3) Alvaro's patch, as of > https://postgr.es/m/202106042321.6jx54

Re: [bug?] Missed parallel safety checks, and wrong parallel safety

2021-06-08 Thread Robert Haas
On Mon, Jun 7, 2021 at 11:33 PM Amit Kapila wrote: > Note the error is raised after applying the patch, without the patch, > the above won't show any error (error message could be improved here). > Such cases can lead to unpredictable behavior without a patch because > we won't be able to detect t

Re: GiST operator class for bool

2021-06-08 Thread Emre Hasegeli
> But patch that you propose does not support sorting build added in PG14. It looks like the change to btree_gist is not committed yet. I'll rebase my patch once it's committed. It was a long thread. I couldn't read all of it. Though, the last patches felt to me like a part of what's already b

Re: A modest proposal vis hierarchical queries: MINUS in the column list

2021-06-08 Thread Andrew Dunstan
On 6/7/21 6:10 PM, Tom Lane wrote: > "David G. Johnston" writes: >> On Mon, Jun 7, 2021 at 1:54 PM Mark Zellers >> wrote: >>> What if you could use the MINUS keyword in the column >>> list of a select statement to remove a column from the result set returned >>> to the client? >> I asked this a

Re: Multiple hosts in connection string failed to failover in non-hot standby mode

2021-06-08 Thread Tom Lane
Michael Paquier writes: > Now, I also see that using pgwin32_setenv() instead of > src/port/setenv.c causes cl to be confused once we update > ecpg_regression.proj because it cannot find setenv(). Bringing the > question, why is it necessary to have both setenv.c and > pgwin32_setenv() on HEAD?

Re: Misplaced superuser check in pg_log_backend_memory_contexts()

2021-06-08 Thread torikoshia
On 2021-06-08 11:49, Michael Paquier wrote: On Sun, Jun 06, 2021 at 11:13:40AM -0400, Tom Lane wrote: Julien Rouhaud writes: However +1 for the patch, as it seems more consistent to always get a permission failure if you're not a superuser. Yeah, it's just weird if such a check is not the fi

Re: A modest proposal vis hierarchical queries: MINUS in the column list

2021-06-08 Thread Tom Lane
Andrew Dunstan writes: > On 6/7/21 6:10 PM, Tom Lane wrote: >> Note that it's not like SQL hasn't heard of projections before. >> You can always do "SELECT a, b, d FROM subquery-yielding-a-b-c-d". >> So the proposed syntax would save a small amount of typing, but >> it's not adding any real new fu

Re: SQL-standard function body

2021-06-08 Thread Tom Lane
Julien Rouhaud writes: > On Mon, Jun 07, 2021 at 03:24:33PM -0400, Tom Lane wrote: >> Concretely, I think the right fix is per attached. > +1, I agree that this approach is better. Pushed. regards, tom lane

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Alvaro Herrera
On 2021-Jun-06, Justin Pryzby wrote: > However, I also found an autovacuum chewing 100% CPU, and it appears the > problem is actually because autovacuum has locked a page of pg-statistic, and > every other process then gets stuck waiting in the planner. I checked a few > and found these: > [...]

Re: Race condition in recovery?

2021-06-08 Thread Robert Haas
On Tue, Jun 8, 2021 at 4:47 AM Dilip Kumar wrote: > I have changed for as per 9.6 but I am seeing some crash (both > with/without fix), I could not figure out the reason, it did not > generate any core dump, although I changed pg_ctl in PostgresNode.pm > to use "-c" so that it can generate core bu

Re: A modest proposal vis hierarchical queries: MINUS in the column list

2021-06-08 Thread Peter Eisentraut
On 08.06.21 04:50, Julien Rouhaud wrote: On Mon, Jun 07, 2021 at 06:10:58PM -0400, Tom Lane wrote: I'm fairly disinclined to do anything about it though, because I'm afraid of the SQL committee standardizing some other syntax for the same idea in future (or maybe worse, commandeering the same k

Re: Make unlogged table resets detectable

2021-06-08 Thread Tom Lane
Robert Haas writes: > On Mon, Jun 7, 2021 at 11:46 PM Michael Paquier wrote: >> On Mon, Jun 07, 2021 at 02:56:57PM -0400, Tom Lane wrote: >>> +1. I'd support recording the time of the last crash recovery, as >>> well as having a counter. I think an LSN would not be as useful >>> as a timestamp.

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Andres Freund
Hi, On 2021-06-08 14:27:14 +0200, Matthias van de Meent wrote: > heap_prune_satisfies_vacuum considers 1 more transaction to be > unvacuumable, and thus indeed won't vacuum that tuple that > HeapTupleSatisfiesVacuum does want to be vacuumed. > > The new open question is now: Why is > GlobalVisCat

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 12:06:02PM -0400, Alvaro Herrera wrote: > On 2021-Jun-06, Justin Pryzby wrote: > > > However, I also found an autovacuum chewing 100% CPU, and it appears the > > problem is actually because autovacuum has locked a page of pg-statistic, > > and > > every other process then

Re: automatically generating node support functions

2021-06-08 Thread Peter Eisentraut
On 08.06.21 15:40, David Rowley wrote: It's almost 2 years ago now, but I'm wondering if you saw what Andres proposed in [1]? The idea was basically to make a metadata array of the node structs so that, instead of having to output large amounts of .c code to do read/write/copy/equals, instead ju

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Alvaro Herrera
On 2021-Jun-08, Justin Pryzby wrote: > On Tue, Jun 08, 2021 at 12:06:02PM -0400, Alvaro Herrera wrote: > > On 2021-Jun-06, Justin Pryzby wrote: > > > > > However, I also found an autovacuum chewing 100% CPU, and it appears the > > > problem is actually because autovacuum has locked a page of pg-s

Re: Fdw batch insert error out when set batch_size > 65535

2021-06-08 Thread Tomas Vondra
On 5/31/21 6:01 AM, Bharath Rupireddy wrote: > On Mon, May 31, 2021 at 1:21 AM Tomas Vondra > wrote: >> >> Hi, >> >> I took at this patch today. I did some minor changes, mostly: >> >> 1) change the code limiting batch_size from >> >> if (fmstate->p_nums > 0 && >>(batch_size * fmsta

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Peter Geoghegan
Reminds me of the other bug that you also reported about a year ago, Justin - which was never fixed. The one with the duplicate tids from a cic (see pg 14 open item). Is this essentially the same environment as the one that led to your other bug report? Peter Geoghegan (Sent from my phone)

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 02:01:51PM -0400, Alvaro Herrera wrote: > On 2021-Jun-08, Justin Pryzby wrote: > > > On Tue, Jun 08, 2021 at 12:06:02PM -0400, Alvaro Herrera wrote: > > > On 2021-Jun-06, Justin Pryzby wrote: > > > > > > > However, I also found an autovacuum chewing 100% CPU, and it appear

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 11:40:31AM -0700, Peter Geoghegan wrote: > Reminds me of the other bug that you also reported about a year ago, > Justin - which was never fixed. The one with the duplicate tids from a cic > (see pg 14 open item). > > Is this essentially the same environment as the one tha

Re: DELETE CASCADE

2021-06-08 Thread David Christensen
On Mon, Jun 7, 2021 at 2:54 AM Peter Eisentraut < peter.eisentr...@enterprisedb.com> wrote: > On 05.06.21 14:21, David Christensen wrote: > > > >> On Jun 5, 2021, at 2:30 AM, Peter Eisentraut < > peter.eisentr...@enterprisedb.com> wrote: > >> > >> On 03.06.21 22:49, David Christensen wrote: > >>>

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 12:34:04PM -0500, Justin Pryzby wrote: > On Tue, Jun 08, 2021 at 12:06:02PM -0400, Alvaro Herrera wrote: > > On 2021-Jun-06, Justin Pryzby wrote: > > > > > However, I also found an autovacuum chewing 100% CPU, and it appears the > > > problem is actually because autovacuum

Re: Make unlogged table resets detectable

2021-06-08 Thread Jeff Davis
On Tue, 2021-06-08 at 12:52 -0400, Tom Lane wrote: > Yeah. If there are actually use-cases for knowing both things, then > we ought to record both. However, it's not real clear to me why > LSN would be interesting. Let me expand on my use case: in a sharded environment, how do you figure out if

Re: DELETE CASCADE

2021-06-08 Thread David Christensen
> > > So basically where we are dispatching to the CASCADE guts, first check > session user’s DELETE permission and throw the normal permissions error if > they can’t delete? > > Actually, you also need appropriate SELECT permissions that correspond > to the WHERE clause of the DELETE statement. >

Re: Make unlogged table resets detectable

2021-06-08 Thread Tom Lane
Jeff Davis writes: > On Tue, 2021-06-08 at 12:52 -0400, Tom Lane wrote: >> Yeah. If there are actually use-cases for knowing both things, then >> we ought to record both. However, it's not real clear to me why >> LSN would be interesting. > Let me expand on my use case: in a sharded environment

Re: Decoding of two-phase xacts missing from CREATE_REPLICATION_SLOT command

2021-06-08 Thread Jeff Davis
On Tue, 2021-06-08 at 17:41 +1000, Ajin Cherian wrote: > Here's an updated patchset that adds back in the option for two-phase > in CREATE_REPLICATION_SLOT command and a second patch that adds > support for > two-phase decoding in pg_recvlogical. A few things: * I suggest putting the TWO_PHASE ke

Re: Race condition in recovery?

2021-06-08 Thread Robert Haas
On Tue, Jun 8, 2021 at 12:26 PM Robert Haas wrote: > I think the problem is here: > > Can't locate object method "lsn" via package "PostgresNode" at > t/025_stuck_on_old_timeline.pl line 84. > > When that happens, it bails out, and cleans everything up, doing an > immediate shutdown of all the nod

Re: Remove server and libpq support for the version 2 wire protocol

2021-06-08 Thread Bruce Momjian
On Tue, Jun 8, 2021 at 09:13:29AM +0900, Tatsuo Ishii wrote: > In release-14.sgml: > > > > > Remove server and libpq support > for the version 2 wire protocol > (Heikki Linnakangas) > > > > This was last used as the default in Postgres 7.2 (year 2002).

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Alvaro Herrera
On 2021-Jun-08, Justin Pryzby wrote: > They're all zero except for: > > $201 = 1 '\001' > $202 = 3 '\003' > $203 = 1 '\001' > > src/include/storage/proc.h-#define PROC_IS_AUTOVACUUM 0x01 > /* is it an autovac worker? */ > src/include/storage/proc.h-#define PROC

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Peter Geoghegan
On Tue, Jun 8, 2021 at 12:27 PM Justin Pryzby wrote: > > They're running this: > > | PGOPTIONS="--deadlock_timeout=333ms -cstatement-timeout=3600s" psql -c > > "REINDEX INDEX CONCURRENTLY $i" > > And if it times out, it then runs: $PSQL "DROP INDEX CONCURRENTLY $bad" > ... > > $ date -d @16231212

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 01:52:40PM -0700, Peter Geoghegan wrote: > On Tue, Jun 8, 2021 at 12:27 PM Justin Pryzby wrote: > > > They're running this: > > > | PGOPTIONS="--deadlock_timeout=333ms -cstatement-timeout=3600s" psql -c > > > "REINDEX INDEX CONCURRENTLY $i" > > > And if it times out, it th

Re: Make unlogged table resets detectable

2021-06-08 Thread Jeff Davis
On Tue, 2021-06-08 at 16:08 -0400, Tom Lane wrote: > Since we don't put LSNs into unlogged tables, nor would the different > shards be likely to have equivalent LSNs, I'm not seeing that LSN is > remarkably better for this than a timestamp. It requires some other bookkeeping on the part of the sha

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Peter Geoghegan
On Tue, Jun 8, 2021 at 2:23 PM Justin Pryzby wrote: > I'm not sure what you're suggesting ? Maybe I should add some NOTICES there. Here is one approach that might work: Can you check if the assertion added by the attached patch fails very quickly with your test case? This does nothing more than

logical replication of truncate command with trigger causes Assert

2021-06-08 Thread Mark Dilger
Hackers, On master, when a statement level trigger is fired for a replicated truncate command, the following stack trace is generated: TRAP: FailedAssertion("portal != NULL", File: "pquery.c", Line: 1760, PID: 93854) 0 postgres0x000108e269f2 ExceptionalConditio

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Peter Geoghegan
On Tue, Jun 8, 2021 at 4:03 AM Justin Pryzby wrote: > postgres=# SELECT lp, lp_off, lp_flags, lp_len, t_xmin, t_xmax, t_field3, > t_ctid, t_infomask2, t_infomask, t_hoff, t_bits, t_oid FROM > heap_page_items(pg_read_binary_file('/tmp/dump_block.page')); > lp | lp_off | lp_flags | lp_len | t_xm

Re: logical replication of truncate command with trigger causes Assert

2021-06-08 Thread Tom Lane
Mark Dilger writes: > On master, when a statement level trigger is fired for a replicated truncate > command, the following stack trace is generated: Hmm. > I believe the issue was introduced in commit 84f5c2908da which added > EnsurePortalSnapshotExists. That's not going to work in the case

Re: array_cat anycompatible change is breaking xversion upgrade tests (v14 release notes)

2021-06-08 Thread Justin Pryzby
@Bruce: Would you add something about this to the release notes before beta2? I added it as an OpenItem. On Tue, May 25, 2021 at 11:14:58AM -0500, Justin Pryzby wrote: > On Thu, May 20, 2021 at 07:35:10PM -0400, Tom Lane wrote: > > Justin Pryzby writes: > > > On Wed, Nov 04, 2020 at 07:43:51PM -

Re: logical replication of truncate command with trigger causes Assert

2021-06-08 Thread Mark Dilger
> On Jun 8, 2021, at 3:55 PM, Tom Lane wrote: > > The right way to say that is "commit 84f5c2908da exposed the pre-existing > unsafe behavior of this code". It's not okay to run arbitrary user code > without holding a snapshot to protect TOAST dereference operations. Sure, I didn't expect th

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Peter Geoghegan
On Tue, Jun 8, 2021 at 5:27 AM Matthias van de Meent wrote: > > (gdb) p *vacrel > > $56 = {... OldestXmin = 926025113, ...} > > > > (gdb) p GlobalVisCatalogRels > > $57 = {definitely_needed = {value = 926025113}, maybe_needed = {value = > > 926025112}} > > This maybe_needed is older than the Olde

Re: logical replication of truncate command with trigger causes Assert

2021-06-08 Thread Tom Lane
Mark Dilger writes: > On Jun 8, 2021, at 3:55 PM, Tom Lane wrote: >> I suppose that either apply_dispatch or LogicalRepApplyLoop needs to >> grow some more snapshot management logic, but I've not looked at that >> code much, so I don't have an opinion on just where to add it. > I was looking at

Re: array_cat anycompatible change is breaking xversion upgrade tests (v14 release notes)

2021-06-08 Thread Bruce Momjian
On Tue, Jun 8, 2021 at 05:56:18PM -0500, Justin Pryzby wrote: > @Bruce: Would you add something about this to the release notes before beta2? > > I added it as an OpenItem. OK, see below. > On Tue, May 25, 2021 at 11:14:58AM -0500, Justin Pryzby wrote: > > On Thu, May 20, 2021 at 07:35:10PM -04

Re: row filtering for logical replication

2021-06-08 Thread Peter Smith
On Mon, May 10, 2021 at 11:42 PM Euler Taveira wrote: > > On Mon, May 10, 2021, at 5:19 AM, Peter Smith wrote: > > AFAIK this is the latest patch available, but FYI it no longer applies > cleanly on HEAD. > > Peter, the last patch is broken since f3b141c4825. I'm still working on it for > the next

Re: array_cat anycompatible change is breaking xversion upgrade tests (v14 release notes)

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 08:02:46PM -0400, Bruce Momjian wrote: > This involves creating an aggreate that _uses_ these array functions as > their state transition function? Yes -- Justin

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Tom Lane
Peter Geoghegan writes: > On Tue, Jun 8, 2021 at 5:27 AM Matthias van de Meent >>> (gdb) p GlobalVisCatalogRels >>> $57 = {definitely_needed = {value = 926025113}, maybe_needed = {value = >>> 926025112}} >> This maybe_needed is older than the OldestXmin, which indeed gives us >> this problematic

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 02:38:37PM -0700, Peter Geoghegan wrote: > On Tue, Jun 8, 2021 at 2:23 PM Justin Pryzby wrote: > > I'm not sure what you're suggesting ? Maybe I should add some NOTICES > > there. > > Here is one approach that might work: Can you check if the assertion > added by the att

RE: Unused function parameter in get_qual_from_partbound()

2021-06-08 Thread houzj.f...@fujitsu.com
On Tuesday, June 8, 2021 7:30 PM David Rowley > On Tue, 8 Jun 2021 at 21:50, houzj.f...@fujitsu.com > wrote: > > I noticed that the first function parameter in > > get_qual_from_partbound(**Relation rel**, Relation parent, is not used in > > the > function. > > > > Is it better to remove it like

Re: alter table set TABLE ACCESS METHOD

2021-06-08 Thread Jeff Davis
On Thu, 2021-06-03 at 14:36 -0700, Jeff Davis wrote: > Do we have general agreement on this point? Did I miss another > purpose > of detoasting in tablecmds.c, or can we just remove that part of the > patch? Per discussion with Justin, I'll drive this patch. I merged the CF entries into https://co

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Peter Geoghegan
On Tue, Jun 8, 2021 at 5:11 PM Tom Lane wrote: > I wonder if this is a variant of the problem shown at > > https://www.postgresql.org/message-id/2591376.1621196582%40sss.pgh.pa.us > > where maybe_needed was visibly quite insane. This value is > less visibly insane, but it's still wrong. It might

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Peter Geoghegan
On Tue, Jun 8, 2021 at 5:18 PM Justin Pryzby wrote: > I reproduced the issue on a new/fresh cluster like this: > > ./postgres -D data -c autovacuum_naptime=1 -c > autovacuum_analyze_scale_factor=0.005 -c log_autovacuum_min_duration=-1 > psql -h /tmp postgres -c "CREATE TABLE t(i int); INSERT INTO

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Justin Pryzby
On Tue, Jun 08, 2021 at 05:44:15PM -0700, Peter Geoghegan wrote: > On Tue, Jun 8, 2021 at 5:11 PM Tom Lane wrote: > > I wonder if this is a variant of the problem shown at > > > > https://www.postgresql.org/message-id/2591376.1621196582%40sss.pgh.pa.us > > > > where maybe_needed was visibly quite

Re: Misplaced superuser check in pg_log_backend_memory_contexts()

2021-06-08 Thread Michael Paquier
On Wed, Jun 09, 2021 at 12:25:51AM +0900, torikoshia wrote: > BTW, I did the same thing in another patch I'm proposing[1], so I'll fix > that as well. Yes, it would be better to be consistent here. -- Michael signature.asc Description: PGP signature

Re: Multiple hosts in connection string failed to failover in non-hot standby mode

2021-06-08 Thread Michael Paquier
On Tue, Jun 08, 2021 at 11:21:34AM -0400, Tom Lane wrote: > IIUC, what you are proposing to do is replace pgwin32_setenv with > src/port/setenv.c. I don't think that's an improvement. setenv.c > leaks memory on repeat calls, because it cannot know what > pgwin32_setenv knows about how putenv work

Adjust pg_regress output for new long test names

2021-06-08 Thread Thomas Munro
Hi, test deadlock-simple ... ok 20 ms test deadlock-hard... ok10624 ms test deadlock-soft... ok 147 ms test deadlock-soft-2 ... ok 5154 ms test deadlock-parallel... ok 132 ms test deta

Re: Adjust pg_regress output for new long test names

2021-06-08 Thread Julien Rouhaud
On Wed, Jun 09, 2021 at 01:57:45PM +1200, Thomas Munro wrote: > Hi, > > test deadlock-simple ... ok 20 ms > test deadlock-hard... ok10624 ms > test deadlock-soft... ok 147 ms > test deadlock-soft-2 ... ok

Re: Adjust pg_regress output for new long test names

2021-06-08 Thread Michael Paquier
On Wed, Jun 09, 2021 at 10:17:35AM +0800, Julien Rouhaud wrote: > On Wed, Jun 09, 2021 at 01:57:45PM +1200, Thomas Munro wrote: >> Any objections to making these new tests line up with the rest? > > No objection, as the output is still way under 80 characters. +1. -- Michael signature.asc Descr

Re: Logical replication keepalive flood

2021-06-08 Thread Kyotaro Horiguchi
Hi. On 2021/06/08 21:21, Abbas Butt wrote: Hi Kyotaro, I have tried to test your patches. Unfortunately even after applying the patches the WAL Sender is still sending too frequent keepalive messages. Sorry for the bogus patch.  I must have seen something impossible. The keep-alive packet is

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Masahiko Sawada
On Wed, Jun 9, 2021 at 2:17 AM Andres Freund wrote: > > Hi, > > On 2021-06-08 14:27:14 +0200, Matthias van de Meent wrote: > > heap_prune_satisfies_vacuum considers 1 more transaction to be > > unvacuumable, and thus indeed won't vacuum that tuple that > > HeapTupleSatisfiesVacuum does want to be

Re: Multiple hosts in connection string failed to failover in non-hot standby mode

2021-06-08 Thread Tom Lane
Michael Paquier writes: > On Tue, Jun 08, 2021 at 11:21:34AM -0400, Tom Lane wrote: >> IIUC, what you are proposing to do is replace pgwin32_setenv with >> src/port/setenv.c. I don't think that's an improvement. setenv.c >> leaks memory on repeat calls, because it cannot know what >> pgwin32_set

Re: pg14b1 stuck in lazy_scan_prune/heap_page_prune of pg_statistic

2021-06-08 Thread Michael Paquier
On Tue, Jun 08, 2021 at 05:47:28PM -0700, Peter Geoghegan wrote: > I don't have time to try this out myself today, but offhand I'm pretty > confident that this is sufficient to reproduce the underlying bug > itself. And if that's true then I guess it can't have anything to do > with the pg_upgrade/

Re: Adjust pg_regress output for new long test names

2021-06-08 Thread Tom Lane
Thomas Munro writes: > test detach-partition-concurrently-1 ... ok 553 ms > test detach-partition-concurrently-2 ... ok 234 ms > test detach-partition-concurrently-3 ... ok 2389 ms > test detach-partition-concurrently-4 ... ok 1876 ms > Any objections to making t

Re: Duplicate history file?

2021-06-08 Thread Kyotaro Horiguchi
On 2021/06/08 18:19, Tatsuro Yamada wrote: I've tried your patch. Unfortunately, it didn't seem to have any good effect on the script I sent to reproduce the problem. Oops! The patch forgot about history files. I checked the attached with your repro script and it works fine. I understand th

Re: Unused function parameter in get_qual_from_partbound()

2021-06-08 Thread Michael Paquier
On Wed, Jun 09, 2021 at 12:28:48AM +, houzj.f...@fujitsu.com wrote: > OK, Thanks for the explanation. > Added to CF: https://commitfest.postgresql.org/33/3159/ At first glance, this looked to me like breaking something just for sake of breaking it, but removing the rel argument could be helpfu

Re: Adjust pg_regress output for new long test names

2021-06-08 Thread Noah Misch
On Wed, Jun 09, 2021 at 01:57:45PM +1200, Thomas Munro wrote: > test deadlock-simple ... ok 20 ms > test deadlock-hard... ok10624 ms > test deadlock-soft... ok 147 ms > test deadlock-soft-2 ... ok 5154 ms >

Re: locking [user] catalog tables vs 2pc vs logical rep

2021-06-08 Thread Amit Kapila
On Tue, Jun 8, 2021 at 6:24 PM vignesh C wrote: > > Thanks for the updated patch. > > I have few comments: > 1) Should we list the actual system tables like pg_class,pg_trigger, > etc instead of any other catalog table? > User has issued an explicit LOCK on pg_class (or any other catalog table) >

Re: Adjust pg_regress output for new long test names

2021-06-08 Thread Thomas Munro
[Responding to two emails in one] On Wed, Jun 9, 2021 at 2:44 PM Tom Lane wrote: > ... or we could shorten those file names. I recall an episode > awhile ago where somebody complained that their version of "tar" > couldn't handle some of the path names in our tarball, so > keeping things from ge

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-06-08 Thread Michael Paquier
On Tue, Jun 08, 2021 at 10:39:24AM -0400, Alvaro Herrera wrote: > My unverified guess is that this code causes too many pipeline stalls > while executing the big per-column loop. Maybe it would be better to > scan the attribute array twice: one to collect all data from > Form_pg_attribute for each

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-06-08 Thread Tom Lane
Michael Paquier writes: > On Tue, Jun 08, 2021 at 10:39:24AM -0400, Alvaro Herrera wrote: >> Maybe at this point reverting is the only solution. > That's a safe bet at this point. It would be good to conclude this > one by beta2 IMO. I still think it's a really dubious argument that anybody wou

Re: logical decoding bug: segfault in ReorderBufferToastReplace()

2021-06-08 Thread Amit Kapila
On Wed, Jun 9, 2021 at 12:06 AM Jeremy Schneider wrote: > > On 6/4/21 23:42, Amit Kapila wrote: > > On 2021-Jun-04, Jeremy Schneider wrote: > > ERROR: XX000: could not open relation with OID 0 > LOCATION: ReorderBufferToastReplace, reorderbuffer.c:305 > > Even, if this fixes the issue, I guess it

Re: Error on pgbench logs

2021-06-08 Thread Michael Paquier
On Tue, Jun 08, 2021 at 06:59:04PM +0900, Kyotaro Horiguchi wrote: > The cause is that the time unit is changed to usec but the patch > forgot to convert agg_interval into the same unit in doLog. I tempted > to change it into pg_time_usec_t but it seems that it is better that > the unit is same wit

Re: Fdw batch insert error out when set batch_size > 65535

2021-06-08 Thread Bharath Rupireddy
On Wed, Jun 9, 2021 at 12:04 AM Tomas Vondra wrote: > No, the "Int16" refers to the FE/BE documentation, where we use Int16. > But in the C code we interpret it as uint16. Hm. I see that in protocol.sgml Int16 is being used. > I've added a simple regression test to postgres_fdw, testing that bat

Re: Transactions involving multiple postgres foreign servers, take 2

2021-06-08 Thread Masahiko Sawada
On Tue, Jun 8, 2021 at 5:28 PM tsunakawa.ta...@fujitsu.com wrote: > > From: Masahiko Sawada > > On Tue, Jun 8, 2021 at 9:47 AM tsunakawa.ta...@fujitsu.com > > wrote: > > > Why does the client have to know the error on a remote server, whereas the > > global transaction itself is destined to comm

Re: [HACKERS] logical decoding of two-phase transactions

2021-06-08 Thread Greg Nancarrow
On Tue, Jun 8, 2021 at 4:12 PM Peter Smith wrote: > > Please find attached the latest patch set v83* > Some feedback for the v83 patch set: v83-0001: (1) doc/src/sgml/protocol.sgml (i) Remove extra space: BEFORE: + The transaction will be decoded and transmitted at AFTER: + T

  1   2   >