Re: range_agg

2020-10-08 Thread Pavel Stehule
čt 24. 9. 2020 v 2:05 odesílatel Paul A Jungwirth < p...@illuminatedcomputing.com> napsal: > On Sun, Aug 16, 2020 at 12:55 PM Paul A Jungwirth > wrote: > > This is rebased on the current master, including some changes to doc > > tables and pg_upgrade handling of type oids. > > Here is a rebased v

Re: Parallel copy

2020-10-08 Thread Amit Kapila
On Thu, Oct 8, 2020 at 8:43 AM Greg Nancarrow wrote: > > On Thu, Oct 8, 2020 at 5:44 AM vignesh C wrote: > > > Attached v6 patch with the fixes. > > > > Hi Vignesh, > > I noticed a couple of issues when scanning the code in the following patch: > > v6-0003-Allow-copy-from-command-to-process-d

RE: [PoC] Non-volatile WAL buffer

2020-10-08 Thread Deng, Gang
Hi Takashi, There are some differences between our HW/SW configuration and test steps. I attached postgresql.conf I used for your reference. I would like to try postgresql.conf and steps you provided in the later days to see if I can find cause. I also ran pgbench and postgres server on the sa

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-08 Thread Kyotaro Horiguchi
At Fri, 9 Oct 2020 02:33:37 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Masahiko Sawada > > What about temporary network failures? I think there are users who > > don't want to give up resolving foreign transactions failed due to a > > temporary network failure. Or even they might wan

Re: Wrong example in the bloom documentation

2020-10-08 Thread Daniel Westermann (DWE)
Hi Bruce, Tom, On Thu, Oct  8, 2020 at 03:43:32PM -0400, Tom Lane wrote: >> "Daniel Westermann (DWE)" writes: >> >> I was hoping someone more experienced with this would comment, but >> >> seeing none, I will apply it in a day or two to all supported versions? >> >> Have you tested this output ba

Re: Parallel copy

2020-10-08 Thread Amit Kapila
On Thu, Oct 8, 2020 at 12:14 AM vignesh C wrote: > > On Mon, Sep 28, 2020 at 12:19 PM Amit Kapila wrote: > > > > + */ > > > > +typedef struct ParallelCopyLineBoundary > > > > > > > > Are we doing all this state management to avoid using locks while > > > > processing lines? If so, I think we can

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Fri, Oct 9, 2020 at 10:29 AM Amit Kapila wrote: > > On Fri, Oct 9, 2020 at 10:06 AM Dilip Kumar wrote: > > > > On Fri, Oct 9, 2020 at 8:40 AM Amit Kapila wrote: > > > > > > On Thu, Oct 8, 2020 at 2:34 PM Simon Riggs wrote: > > > > > > > > On Thu, 8 Oct 2020 at 09:47, Dilip Kumar wrote: > >

Re: Parallel copy

2020-10-08 Thread Amit Kapila
On Thu, Oct 8, 2020 at 12:14 AM vignesh C wrote: > > On Mon, Sep 28, 2020 at 12:19 PM Amit Kapila wrote: > > > > > > I am convinced by the reason given by Kyotaro-San in that another > > thread [1] and performance data shown by Peter that this can't be an > > independent improvement and rather in

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Amit Kapila
On Fri, Oct 9, 2020 at 10:06 AM Dilip Kumar wrote: > > On Fri, Oct 9, 2020 at 8:40 AM Amit Kapila wrote: > > > > On Thu, Oct 8, 2020 at 2:34 PM Simon Riggs wrote: > > > > > > On Thu, 8 Oct 2020 at 09:47, Dilip Kumar wrote: > > > > > > > > This script will wait 10 seconds after INSERT exits > >

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Fri, Oct 9, 2020 at 8:40 AM Amit Kapila wrote: > > On Thu, Oct 8, 2020 at 2:34 PM Simon Riggs wrote: > > > > On Thu, 8 Oct 2020 at 09:47, Dilip Kumar wrote: > > > > > > This script will wait 10 seconds after INSERT exits > > > > before executing TRUNCATE, please wait for it to run. > > > > Ha

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread David Rowley
On Fri, 9 Oct 2020 at 15:06, Tom Lane wrote: > I notice there are some other ad-hoc isnan() checks scattered > about costsize.c, too. Maybe we should indeed consider fixing > clamp_row_estimate to get rid of inf (and nan too, I suppose) > so that we'd not need those. I don't recall the exact cas

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-08 Thread Amit Kapila
On Fri, Oct 9, 2020 at 5:45 AM Peter Smith wrote: > > On Thu, Oct 8, 2020 at 5:25 PM Amit Kapila wrote: > > > COMMENT > > > Line 177 > > > +logicalrep_read_prepare(StringInfo in, LogicalRepPrepareData * > > > prepare_data) > > > > > > prepare_data->prepare_type = flags; > > > This code may be OK

RE: extension patch of CREATE OR REPLACE TRIGGER

2020-10-08 Thread osumi.takami...@fujitsu.com
Hello > > * A lesser point is that I think you're overcomplicating the code by > > applying heap_modify_tuple. You might as well just build the new > > tuple normally in all cases, and then apply either CatalogTupleInsert or > CatalogTupleUpdate. > > > > * Also, the search for an existing trigge

Re: Minor documentation error regarding streaming replication protocol

2020-10-08 Thread Bruce Momjian
On Fri, Oct 9, 2020 at 08:52:50AM +0900, Michael Paquier wrote: > On Thu, Oct 08, 2020 at 04:23:06PM -0400, Bruce Momjian wrote: > > I have looked at this. It seems SendTimeLineHistory() is sending raw > > bytes from the history file, with no encoding conversion, and > > ReceiveXlogStream() is re

Re: Fix typos in reorderbuffer.c

2020-10-08 Thread Amit Kapila
On Thu, Oct 8, 2020 at 2:40 PM Masahiko Sawada wrote: > > On Thu, 8 Oct 2020 at 17:37, Amit Kapila wrote: > > > > > So, I feel the > > comments should be accordingly updated. > > +1 for this change. > Thanks, I have pushed this and along with it pushed a typo-fix in logical.c. -- With Regards,

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Amit Kapila
On Thu, Oct 8, 2020 at 2:34 PM Simon Riggs wrote: > > On Thu, 8 Oct 2020 at 09:47, Dilip Kumar wrote: > > > > This script will wait 10 seconds after INSERT exits > > > before executing TRUNCATE, please wait for it to run. > > Has this been tested with anything other than the one test case? > > It

RE: POC: postgres_fdw insert batching

2020-10-08 Thread tsunakawa.ta...@fujitsu.com
From: Tomas Vondra > I'm not sure when I'll have time to work on this again, so if you are > interested and willing to work on it, please go ahead. I'll gladly do > reviews and help you with it. Thank you very much. > I think transferring data to other databases is fine - interoperability > is

Re: Expansion of our checks for connection-loss errors

2020-10-08 Thread Kyotaro Horiguchi
At Thu, 08 Oct 2020 21:41:55 -0400, Tom Lane wrote in > Kyotaro Horiguchi writes: > > At Thu, 08 Oct 2020 15:15:54 -0400, Tom Lane wrote in > >> Accordingly, I propose the attached patch (an expansion of > >> Fujii-san's) that causes us to test for all five errnos anyplace > >> we had been che

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-08 Thread Greg Nancarrow
On Fri, Oct 9, 2020 at 8:41 AM Thomas Munro wrote: > One thing I noticed is that you have logic, variable names and > assertions all over the tree that assume that we can only do parallel > *inserts*. I agree 100% with your plan to make Parallel Insert work > first, it is an excellent goal and if

RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-08 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > What about temporary network failures? I think there are users who > don't want to give up resolving foreign transactions failed due to a > temporary network failure. Or even they might want to wait for > transaction completion until they send a cancel request. If we want t

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-08 Thread Kyotaro Horiguchi
Oops! Sorry for the mistake. At Fri, 09 Oct 2020 11:12:01 +0900 (JST), Kyotaro Horiguchi wrote in > At Fri, 9 Oct 2020 00:41:24 +, "tsunakawa.ta...@fujitsu.com" > wrote in > > From: Jamison, Kirk/ジャミソン カーク > > > > (6) > > > > + bufHdr->tag.blockNum

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-10-08 Thread Kyotaro Horiguchi
At Fri, 9 Oct 2020 00:41:24 +, "tsunakawa.ta...@fujitsu.com" wrote in > From: Jamison, Kirk/ジャミソン カーク > > > (6) > > > + bufHdr->tag.blockNum >= > > > firstDelBlock[j]) > > > + InvalidateBuffer(bufHdr); /* > > > releases s

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread Tom Lane
David Rowley writes: > Are you worried about the costs above the join that triggers that > coming out as NaN with that fix? It appears that's the case. [ pokes at that... ] Yeah, it looks like nestloop cost estimation also has some issues with inf-times-zero producing NaN; it's just not assertin

Re: Expansion of our checks for connection-loss errors

2020-10-08 Thread Tom Lane
Kyotaro Horiguchi writes: > At Thu, 08 Oct 2020 15:15:54 -0400, Tom Lane wrote in >> Accordingly, I propose the attached patch (an expansion of >> Fujii-san's) that causes us to test for all five errnos anyplace >> we had been checking for ECONNRESET. > +1 for the direction. > In terms of conn

Re: Expansion of our checks for connection-loss errors

2020-10-08 Thread Kyotaro Horiguchi
At Thu, 08 Oct 2020 15:15:54 -0400, Tom Lane wrote in > Over in the thread at [1], we've tentatively determined that the > reason buildfarm member lorikeet is currently failing is that its > network stack returns ECONNABORTED for (some?) connection failures, > whereas our code is only expecting E

Remove some unnecessary if-condition

2020-10-08 Thread Hou, Zhijie
Hi I found some likely unnecessary if-condition in code. 1. Some check in else branch seems unnecessary. In (/src/backend/replication/logical/reorderbuffer.c) ① @@ -4068,7 +4068,7 @@ ReorderBufferToastAppendChunk(ReorderBuffer *rb, ReorderBufferTXN *txn, > bool   found; > if (!found) > { >

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-08 Thread tsunakawa.ta...@fujitsu.com
From: Jamison, Kirk/ジャミソン カーク > > (6) > > + bufHdr->tag.blockNum >= > > firstDelBlock[j]) > > + InvalidateBuffer(bufHdr); /* > > releases spinlock */ > > > > The right side of >= should be cur_block. > > Fixed. >= should b

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread David Rowley
On Fri, 9 Oct 2020 at 12:59, Tom Lane wrote: > If we did want to do something here, I'd consider something like > > if (isnan(outer_skip_rows)) > outer_skip_rows = 0; > if (isnan(inner_skip_rows)) > inner_skip_rows = 0; Are you worried about the costs above

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread Michael Paquier
On Thu, Oct 08, 2020 at 01:17:25PM -0400, Tom Lane wrote: > Agreed. In theory there might be some point in removing leaks that happen > per-statement, so as to avoid unreasonable memory bloat when processing an > extremely long input file. In practice nobody has complained about that, > and if so

Re: [HACKERS] logical decoding of two-phase transactions

2020-10-08 Thread Peter Smith
On Thu, Oct 8, 2020 at 5:25 PM Amit Kapila wrote: > > COMMENT > > Line 177 > > +logicalrep_read_prepare(StringInfo in, LogicalRepPrepareData * > > prepare_data) > > > > prepare_data->prepare_type = flags; > > This code may be OK but it does seem a bit of an abuse of the flags. > > > > e.g. Are th

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread Tom Lane
David Rowley writes: > I admit it's annoying to add cycles to clamp_row_est() for such insane cases. I poked at this a bit more closely, and noted that the actual problem is that when we do this: outer_skip_rows = rint(outer_path_rows * outerstartsel); we have outer_path_rows = inf, out

Re: Minor documentation error regarding streaming replication protocol

2020-10-08 Thread Michael Paquier
On Thu, Oct 08, 2020 at 04:23:06PM -0400, Bruce Momjian wrote: > I have looked at this. It seems SendTimeLineHistory() is sending raw > bytes from the history file, with no encoding conversion, and > ReceiveXlogStream() is receiving it, again assuming it is just plain > text. I am not sure we rea

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread David Rowley
On Fri, 9 Oct 2020 at 12:16, Tom Lane wrote: > > > Perhaps the right fix is to modify clamp_row_est() with: > > I thought of that too, but as you say, if the rowcount has overflowed a > double then we've got way worse problems. It'd make more sense to try > to keep the count to a saner value in t

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread Tom Lane
David Rowley writes: > The reason it fails is that outer_path_rows has become infinity due to > calc_joinrel_size_estimate continually multiplying in the join > selectivity of 0.05 (due to our 200 default num distinct from lack of > any stats) which after a number of iterations causes the number t

Re: Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread David Rowley
On Fri, 9 Oct 2020 at 08:16, Onder Kalaci wrote: > I hit an assertion failure. When asserts disabled, it works fine even with > more tables (>5000). > > Steps to reproduce: > CREATE TABLE users_table (user_id int, time timestamp, value_1 int, value_2 > int, value_3 float, value_4 bigint); > 250

Re: speed up unicode normalization quick check

2020-10-08 Thread John Naylor
On Thu, Oct 8, 2020 at 8:29 AM Michael Paquier wrote: > On Thu, Oct 08, 2020 at 04:52:18AM -0400, John Naylor wrote: > > Looks fine overall, but one minor nit: I'm curious why you made a > separate > > section in the pgindent exclusions. The style in that file seems to be > one > > comment per ca

Re: POC: postgres_fdw insert batching

2020-10-08 Thread Tomas Vondra
On Thu, Oct 08, 2020 at 02:40:10AM +, tsunakawa.ta...@fujitsu.com wrote: Hello Tomas san, Thank you for picking up this. I'm interested in this topic, too. (As an aside, we'd like to submit a bulk insert patch for ECPG in the near future.) As others referred, Andrey-san's fast COPY to f

Re: [HACKERS] Custom compression methods

2020-10-08 Thread Tomas Vondra
On Thu, Oct 08, 2020 at 02:38:27PM +0530, Dilip Kumar wrote: On Wed, Oct 7, 2020 at 5:00 PM Dilip Kumar wrote: On Wed, Oct 7, 2020 at 10:26 AM Dilip Kumar wrote: > > On Tue, Oct 6, 2020 at 10:21 PM Tomas Vondra > wrote: > > > > On Tue, Oct 06, 2020 at 11:00:55AM +0530, Dilip Kumar wrote: > >

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-08 Thread Thomas Munro
On Tue, Oct 6, 2020 at 10:38 PM Greg Nancarrow wrote: +if (estate->es_plannedstmt->commandType == CMD_INSERT) ... +if ((XactReadOnly || (IsInParallelMode() && queryDesc->plannedstmt->commandType != CMD_INSERT)) && ... +isParallelInsertLeader = nodeModifyTableState->operatio

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread Ranier Vilela
On Thu, Oct 8, 2020 at 10:13:53AM -0700, John W Higgins wrote: >It's not going to win a Turing award - but I thought this project was a >little more friendly then what I've seen in this thread towards a first >time contributor. Instead, it is unfriendly. It takes a lot of motivation to "try" to

Re: Minor documentation error regarding streaming replication protocol

2020-10-08 Thread Bruce Momjian
On Sun, Sep 13, 2020 at 10:25:01PM +0200, Brar Piening wrote: > While implementing streaming replication client functionality for Npgsql > I stumbled upon a minor documentation error at > https://www.postgresql.org/docs/current/protocol-replication.html > > The "content" return value for the TIMEL

Re: Wrong example in the bloom documentation

2020-10-08 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 03:43:32PM -0400, Tom Lane wrote: > "Daniel Westermann (DWE)" writes: > >> I was hoping someone more experienced with this would comment, but > >> seeing none, I will apply it in a day or two to all supported versions? > >> Have you tested this output back to 9.5? > > > I

Re: Wrong example in the bloom documentation

2020-10-08 Thread Tom Lane
"Daniel Westermann (DWE)" writes: >> I was hoping someone more experienced with this would comment, but >> seeing none, I will apply it in a day or two to all supported versions? >> Have you tested this output back to 9.5? > I hoped that as well. No, I tested down to 9.6 because the change happen

Expansion of our checks for connection-loss errors

2020-10-08 Thread Tom Lane
Over in the thread at [1], we've tentatively determined that the reason buildfarm member lorikeet is currently failing is that its network stack returns ECONNABORTED for (some?) connection failures, whereas our code is only expecting ECONNRESET. Fujii Masao therefore proposes that we treat ECONNAB

Assertion failure with LEFT JOINs among >500 relations

2020-10-08 Thread Onder Kalaci
Hi, I hit an assertion failure. When asserts disabled, it works fine even with more tables (>5000). Steps to reproduce: CREATE TABLE users_table (user_id int, time timestamp, value_1 int, value_2 int, value_3 float, value_4 bigint); 250 relations work fine, see the query (too long to copy &

Re: Wrong example in the bloom documentation

2020-10-08 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 06:12:47PM +, Daniel Westermann (DWE) wrote: > Hi Bruce, > > >On Thu, Oct 8, 2020 at 06:34:32AM +, Daniel Westermann (DWE) wrote: > >> Hi, > >> > >> as this does not get any attention on the docs-list, once again here. > >> Any thoughts on this? > > >I was hoping

Re: Wrong example in the bloom documentation

2020-10-08 Thread Daniel Westermann (DWE)
Hi Bruce, >On Thu, Oct 8, 2020 at 06:34:32AM +, Daniel Westermann (DWE) wrote: >> Hi, >> >> as this does not get any attention on the docs-list, once again here. >> Any thoughts on this? >I was hoping someone more experienced with this would comment, but >seeing none, I will apply it in a da

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread Tom Lane
Michael Meskes writes: > On Thu, 2020-10-08 at 12:27 -0400, Bruce Momjian wrote: >> Agreed, but what does the TODO item mean then? >> >> Fix small memory leaks in ecpg >> Memory leaks in a short running application like ecpg are not really >> a problem, but make debugging m

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 10:13:53AM -0700, John W Higgins wrote: > > On Wed, Oct 7, 2020 at 6:35 PM Michael Paquier wrote: > > On Wed, Oct 07, 2020 at 02:31:54PM +0300, Maksim Kita wrote: > > Fix progname memory leak in ecpg client. > > Issue taken from todo list https://wiki.postgres

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread John W Higgins
On Wed, Oct 7, 2020 at 6:35 PM Michael Paquier wrote: > On Wed, Oct 07, 2020 at 02:31:54PM +0300, Maksim Kita wrote: > > Fix progname memory leak in ecpg client. > > Issue taken from todo list https://wiki.postgresql.org/wiki/Todo. > > FWIW, I don't see much point in doing that. > I hope that ev

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 06:58:14PM +0200, Michael Meskes wrote: > On Thu, 2020-10-08 at 12:27 -0400, Bruce Momjian wrote: > > On Thu, Oct 8, 2020 at 10:35:41AM +0900, Michael Paquier wrote: > > > On Wed, Oct 07, 2020 at 02:31:54PM +0300, Maksim Kita wrote: > > > > Fix progname memory leak in ecpg

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread Michael Meskes
On Thu, 2020-10-08 at 12:27 -0400, Bruce Momjian wrote: > On Thu, Oct 8, 2020 at 10:35:41AM +0900, Michael Paquier wrote: > > On Wed, Oct 07, 2020 at 02:31:54PM +0300, Maksim Kita wrote: > > > Fix progname memory leak in ecpg client. > > > Issue taken from todo list https://wiki.postgresql.org/wik

Re: Probably typo in multixact.c

2020-10-08 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 10:26:39AM +0900, Michael Paquier wrote: > On Thu, Oct 08, 2020 at 01:15:35AM +, Hou, Zhijie wrote: > > Hi > > > > In multixact.c I found some comments like the following: > > > > * Similar to AtEOX_MultiXact but for COMMIT PREPARED > > * Discard the local Mu

Re: [PATCH] ecpg: fix progname memory leak

2020-10-08 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 10:35:41AM +0900, Michael Paquier wrote: > On Wed, Oct 07, 2020 at 02:31:54PM +0300, Maksim Kita wrote: > > Fix progname memory leak in ecpg client. > > Issue taken from todo list https://wiki.postgresql.org/wiki/Todo. > > FWIW, I don't see much point in doing that. For on

Re: Wrong example in the bloom documentation

2020-10-08 Thread Bruce Momjian
On Thu, Oct 8, 2020 at 06:34:32AM +, Daniel Westermann (DWE) wrote: > Hi, > > as this does not get any attention on the docs-list, once again here. > Any thoughts on this? I was hoping someone more experienced with this would comment, but seeing none, I will apply it in a day or two to all s

Re: extensible options syntax for replication parser?

2020-10-08 Thread Robert Haas
On Wed, Jun 24, 2020 at 11:51 AM Robert Haas wrote: > Thanks for the review. v2 attached, hopefully fixing the compilation > issue you mentioned. Tushar Ahuja reported to me off-list that my basebackup refactoring patch set was changing whether or not the following message appeared: NOTICE: WAL

Re: Resetting spilled txn statistics in pg_stat_replication

2020-10-08 Thread Amit Kapila
On Thu, Oct 8, 2020 at 7:46 AM Masahiko Sawada wrote: > > On Wed, 7 Oct 2020 at 17:52, Amit Kapila wrote: > > > > > I think after we are done with this the next > > step would be to finish the streaming stats work [1]. We probably need > > to review and add the test case in that patch. If nobody

Re: Transactions involving multiple postgres foreign servers, take 2

2020-10-08 Thread Masahiko Sawada
On Thu, 8 Oct 2020 at 18:05, tsunakawa.ta...@fujitsu.com wrote: > > Sorry to be late to respond. (My PC is behaving strangely after upgrading > Win10 2004) > > From: Masahiko Sawada > > After more thoughts on Tsunakawa-san’s idea it seems to need the > > following conditions: > > > > * At least

Re: partition routing layering in nodeModifyTable.c

2020-10-08 Thread Amit Langote
On Wed, Oct 7, 2020 at 9:07 PM Heikki Linnakangas wrote: > On 07/10/2020 12:50, Amit Langote wrote: > > On Tue, Oct 6, 2020 at 12:45 AM Heikki Linnakangas wrote: > >> It would be better to set it in make_modifytable(), just > >> after calling PlanDirectModify(). > > > > Actually, that's how it wa

Re: speed up unicode normalization quick check

2020-10-08 Thread Michael Paquier
On Thu, Oct 08, 2020 at 04:52:18AM -0400, John Naylor wrote: > Looks fine overall, but one minor nit: I'm curious why you made a separate > section in the pgindent exclusions. The style in that file seems to be one > comment per category. Both parts indeed use PerfectHash.pm, but are generated by

Re: Asynchronous Append on postgres_fdw nodes.

2020-10-08 Thread Andrey Lepikhov
Hi, I want to suggest one more improvement. Currently the is_async_capable_path() routine allow only ForeignPath nodes as async capable path. But in some cases we can allow SubqueryScanPath as async capable too. For example: SELECT * FROM ((SELECT * FROM foreign_1) UNION ALL (SELECT a FROM for

Re: [HACKERS] Runtime Partition Pruning

2020-10-08 Thread Ashutosh Bapat
On Wed, Oct 7, 2020 at 7:00 PM Andy Fan wrote: > > > > On Wed, Oct 7, 2020 at 5:05 PM Andy Fan wrote: >> >> >> >> On Sun, Oct 4, 2020 at 3:10 PM Andy Fan wrote: Now, in my experience, the current system for custom plans vs. generic plans doesn't approach the problem in t

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-10-08 Thread Dmitry Dolgov
> On Thu, Oct 08, 2020 at 09:34:51AM +0800, Andy Fan wrote: > > > Other than that I wanted to ask what are the plans to proceed with this > > patch? It's been a while since the question was raised in which format > > to keep unique key expressions, and as far as I can see no detailed > > suggestion

Re: [Patch] ALTER SYSTEM READ ONLY

2020-10-08 Thread Amul Sul
On Wed, Oct 7, 2020 at 11:19 PM Robert Haas wrote: > > On Wed, Sep 16, 2020 at 3:33 PM Robert Haas wrote: > > I don't mind a loop, but that one looks broken. We have to clear the > > bit before we call the function that processes that type of barrier. > > Otherwise, if we succeed in absorbing the

Wired if-statement in gen_partprune_steps_internal

2020-10-08 Thread Andy Fan
Hi: I found the following code in gen_partprune_steps_internal, which looks the if-statement to be always true since list_length(results) > 1; I added an Assert(step_ids != NIL) and all the test cases passed. if the if-statement is always true, shall we remove it to avoid confusion? gen_part

Re: Asynchronous Append on postgres_fdw nodes.

2020-10-08 Thread Andrey V. Lepikhov
On 10/5/20 11:35 AM, Etsuro Fujita wrote: Hi, I found a small problem. If we have a mix of async and sync subplans when we catch an assertion on a busy connection. Just for example: PLAN Nested Loop (cost=100.00..174316.95 rows=975 width=8) (actual time=5.191..9.262 rows=9 loops=1) J

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-08 Thread k.jami...@fujitsu.com
Hi, > Attached are the updated patches. Sorry there was an error in the 3rd patch. So attached is a rebase one. Regards, Kirk Jamison 0001-v1-Prevent-invalidating-blocks-in-smgrextend-during-recovery.patch Description: 0001-v1-Prevent-invalidating-blocks-in-smgrextend-during-recovery.patch

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-10-08 Thread k.jami...@fujitsu.com
On Thursday, October 8, 2020 3:38 PM, Tsunakawa-san wrote: > Hi Kirk san, Thank you for looking into my patches! > (1) > + * This returns an InvalidBlockNumber when smgr_cached_nblocks is not > + * available and when not in recovery path. > > + /* > + * We cannot believe the result from

Re: Fix typos in reorderbuffer.c

2020-10-08 Thread Masahiko Sawada
On Thu, 8 Oct 2020 at 17:37, Amit Kapila wrote: > > @@ -1432,7 +1432,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, > ReorderBufferTXN *txn) > ReorderBufferCleanupTXN(rb, subtxn); > } > > - /* cleanup changes in the toplevel txn */ > + /* cleanup changes in the txn */ > dlist_foreach_modify

Re: [HACKERS] Custom compression methods

2020-10-08 Thread Dilip Kumar
On Wed, Oct 7, 2020 at 5:00 PM Dilip Kumar wrote: > > On Wed, Oct 7, 2020 at 10:26 AM Dilip Kumar wrote: > > > > On Tue, Oct 6, 2020 at 10:21 PM Tomas Vondra > > wrote: > > > > > > On Tue, Oct 06, 2020 at 11:00:55AM +0530, Dilip Kumar wrote: > > > >On Mon, Oct 5, 2020 at 9:34 PM Tomas Vondra > >

RE: Transactions involving multiple postgres foreign servers, take 2

2020-10-08 Thread tsunakawa.ta...@fujitsu.com
Sorry to be late to respond. (My PC is behaving strangely after upgrading Win10 2004) From: Masahiko Sawada > After more thoughts on Tsunakawa-san’s idea it seems to need the > following conditions: > > * At least postgres_fdw is viable to implement these APIs while > guaranteeing not to happe

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Simon Riggs
On Thu, 8 Oct 2020 at 09:47, Dilip Kumar wrote: > > This script will wait 10 seconds after INSERT exits > > before executing TRUNCATE, please wait for it to run. Has this been tested with anything other than the one test case? It would be good to know how the patch handles a transaction that co

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Thu, Oct 8, 2020 at 2:17 PM Dilip Kumar wrote: > > On Thu, Oct 8, 2020 at 2:05 PM Keisuke Kuroda > wrote: > > > > Hi Dilip, > > > > > I could not see this issue even without the patch, it is taking less > > > than 1s even without the patch. See below results > > > > > > 2020-10-08 13:00:49 BE

Re: Resetting spilled txn statistics in pg_stat_replication

2020-10-08 Thread Amit Kapila
On Thu, Oct 8, 2020 at 1:55 PM Masahiko Sawada wrote: > > On Thu, 8 Oct 2020 at 14:10, Amit Kapila wrote: > > > > > > We can write if we want but there are few things we need to do for > > that like maybe a new function like wait_for_spill_stats which will > > check if the counters have become ze

Re: speed up unicode normalization quick check

2020-10-08 Thread John Naylor
On Thu, Oct 8, 2020 at 2:48 AM Michael Paquier wrote: > On Wed, Oct 07, 2020 at 03:18:44PM +0900, Michael Paquier wrote: > I looked at this one again today, and applied it. I looked at what > MSVC compiler was able to do in terms of optimizationswith > shift-and-add for multipliers, and it is by

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Thu, Oct 8, 2020 at 2:05 PM Keisuke Kuroda wrote: > > Hi Dilip, > > > I could not see this issue even without the patch, it is taking less > > than 1s even without the patch. See below results > > > > 2020-10-08 13:00:49 BEGIN 509 > > 2020-10-08 13:00:49 table nsp_001.part_0001: INSERT:... > >

Fix typos in reorderbuffer.c

2020-10-08 Thread Amit Kapila
@@ -1432,7 +1432,7 @@ ReorderBufferCleanupTXN(ReorderBuffer *rb, ReorderBufferTXN *txn) ReorderBufferCleanupTXN(rb, subtxn); } - /* cleanup changes in the toplevel txn */ + /* cleanup changes in the txn */ dlist_foreach_modify(iter, &txn->changes) { ReorderBufferChange *change; @@ -1533,

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Keisuke Kuroda
Hi Dilip, > I could not see this issue even without the patch, it is taking less > than 1s even without the patch. See below results > > 2020-10-08 13:00:49 BEGIN 509 > 2020-10-08 13:00:49 table nsp_001.part_0001: INSERT:... > 2020-10-08 13:00:49 COMMIT 509 (at 2020-10-08 13:00:48.741986+05:30) >

Re: Resetting spilled txn statistics in pg_stat_replication

2020-10-08 Thread Masahiko Sawada
On Thu, 8 Oct 2020 at 14:10, Amit Kapila wrote: > > On Thu, Oct 8, 2020 at 7:46 AM Masahiko Sawada > wrote: > > > > On Wed, 7 Oct 2020 at 17:52, Amit Kapila wrote: > > > > > > On Wed, Oct 7, 2020 at 11:24 AM Masahiko Sawada > > > wrote: > > > > > > > > On Tue, 6 Oct 2020 at 17:56, Amit Kapila

Re: dynamic result sets support in extended query protocol

2020-10-08 Thread Tatsuo Ishii
Are you proposing to bump up the protocol version (either major or minor)? I am asking because it seems you are going to introduce some new message types. Best regards, -- Tatsuo Ishii SRA OSS, Inc. Japan English: http://www.sraoss.co.jp/index_en.php Japanese:http://www.sraoss.co.jp > I want to

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-08 Thread Bharath Rupireddy
On Thu, Oct 8, 2020 at 1:42 PM Greg Nancarrow wrote: > > On Wed, Oct 7, 2020 at 7:25 PM Greg Nancarrow wrote: > > > > On Wed, Oct 7, 2020 at 12:40 AM Bharath Rupireddy > > wrote: > > > In parallel, we are not doing anything(due to the same reason > > > explained in above comment) to find whether

Re: Parallel INSERT (INTO ... SELECT ...)

2020-10-08 Thread Greg Nancarrow
On Wed, Oct 7, 2020 at 7:25 PM Greg Nancarrow wrote: > > On Wed, Oct 7, 2020 at 12:40 AM Bharath Rupireddy > wrote: > > In parallel, we are not doing anything(due to the same reason > > explained in above comment) to find whether there is a foreign > > partition or not while deciding to go with p

dynamic result sets support in extended query protocol

2020-10-08 Thread Peter Eisentraut
I want to progress work on stored procedures returning multiple result sets. Examples of how this could work on the SQL side have previously been shown [0]. We also have ongoing work to make psql show multiple result sets [1]. This appears to work fine in the simple query protocol. But the

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-10-08 Thread Dilip Kumar
On Fri, Oct 2, 2020 at 12:26 PM Keisuke Kuroda wrote: > > Hi Dilip, Amit, > > > > 5. Can you please once repeat the performance test done by Keisuke-San > > > to see if you have similar observations? Additionally, see if you are > > > also seeing the inconsistency related to the Truncate message r

Re: shared-memory based stats collector

2020-10-08 Thread Kyotaro Horiguchi
At Tue, 06 Oct 2020 10:06:44 +0900 (JST), Kyotaro Horiguchi wrote in > The previous version failed to flush local database stats for certain > condition. That behavior caused useless retries and finally a forced > flush that leads to contention. I fixed that and will measure > performance with t