Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Amit Kapila
On Thu, May 27, 2021 at 9:02 AM Amit Kapila wrote: > > On Thu, Mar 25, 2021 at 11:04 AM Ashutosh Bapat > wrote: > > > > Hi All, > > We saw OOM in a system where WAL sender consumed Gigabttes of memory > > which was never released. Upon investigation, we found out that there > > were many ReorderB

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-26 Thread Michael Paquier
On Wed, May 26, 2021 at 11:34:53PM -0400, Tom Lane wrote: > So maybe we should just dump the promise that VACUUM FULL will recompress > everything? I'd be in favor of that actually, because it seems 100% > outside the charter of either VACUUM FULL or CLUSTER. Hmm. You are right that by default t

Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Dilip Kumar
On Thu, May 27, 2021 at 9:03 AM Amit Kapila wrote: > > On Thu, Mar 25, 2021 at 11:04 AM Ashutosh Bapat > wrote: > > > > Hi All, > > We saw OOM in a system where WAL sender consumed Gigabttes of memory > > which was never released. Upon investigation, we found out that there > > were many ReorderB

Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Dilip Kumar
On Thu, May 27, 2021 at 9:26 AM Amit Kapila wrote: > > On Thu, May 27, 2021 at 9:02 AM Amit Kapila wrote: > > > > On Thu, Mar 25, 2021 at 11:04 AM Ashutosh Bapat > > wrote: > > > > > > Hi All, > > > We saw OOM in a system where WAL sender consumed Gigabttes of memory > > > which was never releas

Re: Move pg_attribute.attcompression to earlier in struct for reduced size?

2021-05-26 Thread Tom Lane
Michael Paquier writes: > On Wed, May 26, 2021 at 11:34:53PM -0400, Tom Lane wrote: >> So maybe we should just dump the promise that VACUUM FULL will recompress >> everything? I'd be in favor of that actually, because it seems 100% >> outside the charter of either VACUUM FULL or CLUSTER. > Hmm.

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Amit Kapila
On Wed, May 26, 2021 at 5:51 PM Amit Kapila wrote: > > On Wed, May 26, 2021 at 5:28 PM Bharath Rupireddy > wrote: > > > > On Fri, May 21, 2021 at 3:46 PM Amit Kapila wrote: > > > > > > On Fri, Mar 19, 2021 at 11:02 AM Bharath Rupireddy > > > wrote: > > > > > > > > > The other possibility could

Re: Speed up pg_checksums in cases where checksum already set

2021-05-26 Thread Michael Paquier
On Wed, May 26, 2021 at 09:29:43PM -0500, Justin Pryzby wrote: > In one of the checksum patches, there was an understanding that the pages > should be written even if the checksum is correct, to handle replicas. > > From the v19 patch: > https://www.postgresql.org/message-id/F7AFCFCD-8F77-4546-8D4

Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Amit Kapila
On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: > > On Thu, May 27, 2021 at 9:26 AM Amit Kapila wrote: > > > > > > > > Can we consider the possibility to destroy the toast_hash in > > > ReorderBufferCleanupTXN/ReorderBufferTruncateTXN? It will delay the > > > clean up of memory till the end of

Re: Skip partition tuple routing with constant partition key

2021-05-26 Thread Amit Langote
Hi, On Thu, May 27, 2021 at 2:30 AM Zhihong Yu wrote: >> >> Hi, Amit: > > > For ConvertTupleToPartition() in > 0001-ExecFindPartition-cache-last-used-partition-v3.patch: > > + if (tempslot != NULL) > + ExecClearTuple(tempslot); > > If tempslot and parent_slot point to the same sl

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Bharath Rupireddy
On Thu, May 27, 2021 at 9:43 AM Amit Kapila wrote: > > I have read it but I think we should try to ensure practically what is > > happening because it is possible that first time worker checked in FSM > > without taking relation extension lock, it didn't find any free page, > > and then when it tr

Re: Skipping logical replication transactions on subscriber side

2021-05-26 Thread Masahiko Sawada
On Wed, May 26, 2021 at 3:43 PM Amit Kapila wrote: > > On Tue, May 25, 2021 at 12:26 PM Masahiko Sawada > wrote: > > > > On Mon, May 24, 2021 at 7:51 PM Amit Kapila wrote: > > > > > > On Mon, May 24, 2021 at 1:32 PM Masahiko Sawada > > > wrote: > > > > > > I think you need to consider few mor

Re: Asymmetric partition-wise JOIN

2021-05-26 Thread Andrey Lepikhov
Next version of the patch. For searching any problems I forced this patch during 'make check' tests. Some bugs were found and fixed. -- regards, Andrey Lepikhov Postgres Professional From 101614b504b0b17e201d2375c8af61cfc671e51d Mon Sep 17 00:00:00 2001 From: Andrey Lepikhov Date: Fri, 2 Apr 2

Re: Decoding speculative insert with toast leaks memory

2021-05-26 Thread Dilip Kumar
On Thu, May 27, 2021 at 9:47 AM Amit Kapila wrote: > > On Thu, May 27, 2021 at 9:40 AM Dilip Kumar wrote: > > True, but if you do this clean-up in ReorderBufferCleanupTXN then you > don't need to take care at separate places. Also, toast_hash is stored > in txn so it appears natural to clean it u

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Bharath Rupireddy
On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com wrote: > I am afraid that the using the FSM seems not get a stable performance gain(at > least on my machine), > I will take a deep look into this to figure out the difference. A naive idea > it that the benefit that bulk extension > bring i

Re: Skip partition tuple routing with constant partition key

2021-05-26 Thread Zhihong Yu
On Wed, May 26, 2021 at 9:22 PM Amit Langote wrote: > Hi, > > On Thu, May 27, 2021 at 2:30 AM Zhihong Yu wrote: > >> > >> Hi, Amit: > > > > > > For ConvertTupleToPartition() in > 0001-ExecFindPartition-cache-last-used-partition-v3.patch: > > > > + if (tempslot != NULL) > > + Exec

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Dilip Kumar
On Thu, May 27, 2021 at 10:16 AM Bharath Rupireddy wrote: > > On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com > wrote: > > I am afraid that the using the FSM seems not get a stable performance > > gain(at least on my machine), > > I will take a deep look into this to figure out the differ

Re: Skip partition tuple routing with constant partition key

2021-05-26 Thread Amit Langote
On Thu, May 27, 2021 at 1:55 PM Zhihong Yu wrote: > For CanUseSavedPartitionForTuple, nit: you can check > !dispatch->savedPartResultInfo at the beginning and return early. > This would save some indentation. Sure, see the attached. -- Amit Langote EDB: http://www.enterprisedb.com v5-0002-Ex

Re: Skipping logical replication transactions on subscriber side

2021-05-26 Thread Amit Kapila
On Thu, May 27, 2021 at 9:56 AM Masahiko Sawada wrote: > > On Wed, May 26, 2021 at 3:43 PM Amit Kapila wrote: > > > > On Tue, May 25, 2021 at 12:26 PM Masahiko Sawada > > wrote: > > > > > > On Mon, May 24, 2021 at 7:51 PM Amit Kapila > > > wrote: > > > > > > > > On Mon, May 24, 2021 at 1:32 P

Re: Skip partition tuple routing with constant partition key

2021-05-26 Thread Amit Langote
On Thu, May 27, 2021 at 11:47 AM houzj.f...@fujitsu.com wrote: > About teaching relcache about caching the target partition. > > David-san suggested cache the partidx in PartitionDesc. > And it will need looping and checking the cached value at each level. > I was thinking can we cache a partidx l

Re: Incorrect GUC descriptions in docs and postgresql.conf.sample

2021-05-26 Thread Michael Paquier
On Wed, May 26, 2021 at 11:10:22AM +0900, Michael Paquier wrote: > There is one point where you are right here: log_lock_waits has no > need to be changed. Looks like I checked too many things at once. Fixed that, did one extra round of review, and applied. -- Michael signature.asc Description:

RE: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread tsunakawa.ta...@fujitsu.com
From: Dilip Kumar > I think some other cause of contention on relation extension locks are > 1. CTAS is using a buffer strategy and due to that, it might need to > evict out the buffer frequently for getting the new block in. Maybe > we can identify by turning off the buffer strategy for CTAS and

Re: Race condition in recovery?

2021-05-26 Thread Dilip Kumar
On Thu, May 27, 2021 at 6:19 AM Kyotaro Horiguchi wrote: > > At Wed, 26 May 2021 22:08:32 +0530, Dilip Kumar wrote > in > > On Wed, 26 May 2021 at 10:06 PM, Robert Haas wrote: > > > > > On Wed, May 26, 2021 at 12:26 PM Dilip Kumar > > > wrote: > > > > I will check if there is any timing depend

Re: Bracket, brace, parenthesis

2021-05-26 Thread Kyotaro Horiguchi
At Fri, 14 May 2021 10:04:57 -0400, Tom Lane wrote in > Kyotaro Horiguchi writes: > > I'm not sure how much we (or people) are strcit on the distinction > > between the $SUBJECT, isn't '{' a brace generally? > > +1. I tend to write "square bracket" or "curly brace" when I want to > be extra cl

Re: Race condition in recovery?

2021-05-26 Thread Dilip Kumar
On Wed, May 26, 2021 at 9:40 PM Robert Haas wrote: > ...which has a clear race condition. > src/test/recovery/t/023_pitr_prepared_xact.pl has logic to wait for a > WAL file to be archived, so maybe we can steal that logic and use it > here. Yeah, done that, I think we can use exact same logic fo

Re: Skipping logical replication transactions on subscriber side

2021-05-26 Thread Masahiko Sawada
On Wed, May 26, 2021 at 6:11 PM Amit Kapila wrote: > > On Tue, May 25, 2021 at 6:12 PM Masahiko Sawada wrote: > > > > On Tue, May 25, 2021 at 7:21 PM Bharath Rupireddy > > wrote: > > > > > > If there's no way to get the "correct LSN", then why can't we just > > > print that LSN in the error cont

Re: Forget close an open relation in ReorderBufferProcessTXN()

2021-05-26 Thread Amit Kapila
On Fri, May 21, 2021 at 1:12 PM Amit Langote wrote: > > Hmm, maybe get_rel_syn_entry() should explicitly set map to NULL when > first initializing an entry. It's possible that without doing so, the > map remains set to a garbage value, which causes the invalidation > callback that runs into such

Re: Race condition in recovery?

2021-05-26 Thread Kyotaro Horiguchi
At Thu, 27 May 2021 11:44:47 +0530, Dilip Kumar wrote in > Maybe we can somehow achieve that without a broken archive command, > but I am not sure how it is enough to just delete WAL from pg_wal? I > mean my original case was that > 1. Got the new history file from the archive but did not get t

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Dilip Kumar
On Thu, 27 May 2021 at 11:32 AM, tsunakawa.ta...@fujitsu.com < tsunakawa.ta...@fujitsu.com> wrote: > From: Dilip Kumar > > I think some other cause of contention on relation extension locks are > > 1. CTAS is using a buffer strategy and due to that, it might need to > > evict out the buffer frequ

RE: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread houzj.f...@fujitsu.com
From: Bharath Rupireddy Sent: Thursday, May 27, 2021 12:46 PM > On Thu, May 27, 2021 at 7:12 AM houzj.f...@fujitsu.com > wrote: > > I am afraid that the using the FSM seems not get a stable performance > > gain(at least on my machine), I will take a deep look into this to > > figure out the diffe

Re: Removing unneeded self joins

2021-05-26 Thread Andrey V. Lepikhov
On 5/8/21 2:00 AM, Hywel Carver wrote: On Fri, May 7, 2021 at 8:23 AM Andrey Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: Here I didn't work on 'unnecessary IS NOT NULL filter'. I've tested the new patch, and it is giving the same improved behaviour as the old patch. Thank you for t

RE: Skip partition tuple routing with constant partition key

2021-05-26 Thread houzj.f...@fujitsu.com
From: Amit Langote Sent: Thursday, May 27, 2021 1:54 PM > On Thu, May 27, 2021 at 11:47 AM houzj.f...@fujitsu.com > wrote: > > About teaching relcache about caching the target partition. > > > > David-san suggested cache the partidx in PartitionDesc. > > And it will need looping and checking the

Re: Parallel Inserts in CREATE TABLE AS

2021-05-26 Thread Bharath Rupireddy
On Thu, May 27, 2021 at 12:19 PM houzj.f...@fujitsu.com wrote: > BTW, I checked my test results, I was testing INSERT INTO unlogged table. What do you mean by "testing INSERT INTO"? Is it that you are testing the timings for parallel inserts in INSERT INTO ... SELECT command? If so, why should we

<    1   2