Re: Yet another fast GiST build

2020-09-27 Thread Heikki Linnakangas
On 21/09/2020 16:29, Andrey M. Borodin wrote: But thing that bothers me now: when we vacuum leaf page, we bump it's NSN. But we do not bump internal page LSN. Does this means we will follow rightlinks after vacuum? It seems superflous. Sorry, I did not understand what you said above. Vacuum d

Re: Report error position in partition bound check

2020-09-27 Thread Amit Langote
On Fri, Sep 25, 2020 at 12:02 AM Tom Lane wrote: > [ cc'ing Peter, since his opinion seems to have got us here in the first > place ] > > Amit Langote writes: > > On Thu, Sep 24, 2020 at 7:19 AM Tom Lane wrote: > >> However, while I was looking at it I couldn't help noticing that > >> transform

Re: Parallel copy

2020-09-27 Thread Amit Kapila
On Wed, Jul 22, 2020 at 7:48 PM vignesh C wrote: > > On Tue, Jul 21, 2020 at 3:54 PM Amit Kapila wrote: > > > > > Review comments: > > === > > > > 0001-Copy-code-readjustment-to-support-parallel-copy > > 1. > > @@ -807,8 +835,11 @@ CopyLoadRawBuf(CopyState cstate) > > else > >

Re: Partition prune with stable Expr

2020-09-27 Thread Jesse Zhang
On Sun, Sep 27, 2020 at 7:52 PM Andy Fan wrote: > > > On Mon, Sep 28, 2020 at 9:15 AM Tom Lane wrote: >> >> Andy Fan writes: >> > Well, that's very interesting. Specific to my user case, >> > SELECT * FROM p WHERE pkey = to_date('2018-12-13', '-mm-dd)'; >> > p has 1500+ partitions and planning

remove useless returns

2020-09-27 Thread Tang, Haiying
Hello Found two more useless "return;" lines from the following code. .src/backend/regex/regcomp.c .src/interfaces/libpq/fe-secure.c Maybe it's better to remove them together? Previous discussion: https://www.postgresql.org/message-id/20191128144653.GA27883@alvherre.pgsql Best Regards, Tang -

a problem about XLogReader callback system

2020-09-27 Thread Wang, Shenhao
Hi, hackers I have a problem about XLogReader callback system In xlog.c, function StartupXLOG xlogreader = XLogReaderAllocate(wal_segment_size, NULL, XL_ROUTINE(.page_read = &XLogPageRead,

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-27 Thread tsunakawa.ta...@fujitsu.com
From: Masahiko Sawada > On Fri, 25 Sep 2020 at 18:21, tsunakawa.ta...@fujitsu.com > wrote: > > Why does the client backend have to create a new connection cache entry > during PREPARE or COMMIT PREPARE? Doesn't the client backend naturally > continue to use connections that it has used in its cu

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-09-27 Thread Keisuke Kuroda
Hi Amit, > You need to refer to XLOG_XACT_INVALIDATIONS, not XLOG_INVALIDATIONS. Thank you, that's right. XLOG_INVALIDATIONS is added at COMMIT, so I need to refer to XLOG_XACT_INVALIDATIONS. > Okay. Feel free to clarify your questions if you have any? Are you > interested in writing a patch for

Re: scram-sha-256 broken with FIPS and OpenSSL 1.0.2

2020-09-27 Thread Michael Paquier
On Fri, Sep 25, 2020 at 12:27:03AM -0400, Tom Lane wrote: > Given the tiny number of complaints to date, it seems sufficient to me > to deal with this in HEAD. Thanks. I have done more tests with the range of OpenSSL versions we support on HEAD, and applied this one. I have noticed that the prev

Re: New statistics for tuning WAL buffer size

2020-09-27 Thread Amit Kapila
On Mon, Sep 28, 2020 at 8:24 AM Kyotaro Horiguchi wrote: > > At Mon, 28 Sep 2020 08:11:23 +0530, Amit Kapila > wrote in > > One other thing that occurred to me today is can't we keep this as > > part of PgStat_GlobalStats? We can use pg_stat_reset_shared('wal'); to > > reset it. It seems to me t

The return value of SPI_connect

2020-09-27 Thread Hou, Zhijie
Hi I found the function SPI_connect() has only one return value(SPI_OK_CONNECT). But I also found that some places check the return value of SPI_connect() like the following: if (SPI_connect() != SPI_OK_CONNECT) elog(ERROR, "SPI_connect failed"); Is it necessary to check

[PATCH] SET search_path += octopus

2020-09-27 Thread Abhijit Menon-Sen
Hi. The first attached patch (0001-Accept-SET-xyz-pqr-to-add-pqr-to-the-current-setting.patch) adds support for commands like the following, applicable to any configuration settings that are represented as a comma-separated list of strings (i.e., GUC_LIST_INPUT): postgres=# SET search_path +=

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-27 Thread Greg Nancarrow
On Sat, Sep 26, 2020 at 3:30 PM Bharath Rupireddy wrote: > I further checked on full txn id and command id. Yes, these are > getting passed to workers via InitializeParallelDSM() -> > SerializeTransactionState(). I tried to summarize what we need to do > in case of parallel inserts in general i.

Re: New statistics for tuning WAL buffer size

2020-09-27 Thread Kyotaro Horiguchi
At Mon, 28 Sep 2020 08:11:23 +0530, Amit Kapila wrote in > One other thing that occurred to me today is can't we keep this as > part of PgStat_GlobalStats? We can use pg_stat_reset_shared('wal'); to > reset it. It seems to me this is a cluster-wide stats and somewhat > similar to some of the oth

Re: Load TIME fields - proposed performance improvement

2020-09-27 Thread Peter Smith
On Sat, Sep 26, 2020 at 2:17 AM Tom Lane wrote: > That led me to refactor the patch as attached. (I'd first thought ... Thanks for refactoring the patch. Everything looks good to me. As expected, observations for the v02 patch gave pretty much the same results as v01 (previous mail). v02 perf

Re: Partition prune with stable Expr

2020-09-27 Thread Andy Fan
On Mon, Sep 28, 2020 at 9:15 AM Tom Lane wrote: > Andy Fan writes: > > Well, that's very interesting. Specific to my user case, > > SELECT * FROM p WHERE pkey = to_date('2018-12-13', '-mm-dd)'; > > p has 1500+ partitions and planning takes lots of time, which is so same > > with SELECT * FR

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-27 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I agree with the above two points. Thank you. I'm relieved to know I didn't misunderstand. > > * Then, add a new function, say, smgrnblocks_cached() that simply returns > the cached block count, and DropRelFileNodeBuffers() uses it instead of > smgrnblocks(). > > > > I am

Re: Small improvements to pg_list.h's linitial(), lsecond(), lthird() etc macros

2020-09-27 Thread David Rowley
Thanks for 9d299a492. On Mon, 28 Sep 2020 at 15:35, Tom Lane wrote: > > Poking around to count remaining uses of those inline functions, > I found a few places that should be using the macros instead, > and fixed them. After that, I notice that list_tail(), > list_third_cell(), and list_fourth_c

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-09-27 Thread Amit Kapila
On Mon, Sep 28, 2020 at 7:50 AM Keisuke Kuroda wrote: > > Hi Amit, > > Thank you for the reply! > > > However, after commit c55040ccd0 (When wal_level=logical, > > write invalidations at command end into WAL so that decoding can use > > this information.) we actually know exactly when we need to e

Re: New statistics for tuning WAL buffer size

2020-09-27 Thread Amit Kapila
On Mon, Sep 28, 2020 at 7:00 AM Masahiro Ikeda wrote: > > On 2020-09-26 19:18, Amit Kapila wrote > > > This makes sense to me. I think even if such background processes have > > to write WAL due to wal_buffers, it will be accounted next time the > > backend sends the stats. > > Thanks for your com

Re: Small improvements to pg_list.h's linitial(), lsecond(), lthird() etc macros

2020-09-27 Thread Tom Lane
Poking around to count remaining uses of those inline functions, I found a few places that should be using the macros instead, and fixed them. After that, I notice that list_tail(), list_third_cell(), and list_fourth_cell() are entirely unreferenced. I'm hesitant to get rid of list_tail(), because

Re: Feature improvement of tab completion for DEALLOCATE

2020-09-27 Thread Fujii Masao
On 2020/09/25 15:18, btnakamichin wrote: 2020-09-25 14:30 に Fujii Masao さんは書きました: On 2020/09/25 13:45, btnakamichin wrote: Hello! I’d like to improve the deallocate tab completion. The deallocate tab completion can’t complement “ALL”. Therefore, this patch fixes the problem. Thanks for

Re: Logical replication CPU-bound with TRUNCATE/DROP/CREATE many tables

2020-09-27 Thread Keisuke Kuroda
Hi Amit, Thank you for the reply! > However, after commit c55040ccd0 (When wal_level=logical, > write invalidations at command end into WAL so that decoding can use > this information.) we actually know exactly when we need to execute > each invalidation. I see, thank you for your explaination.

Re: Small improvements to pg_list.h's linitial(), lsecond(), lthird() etc macros

2020-09-27 Thread Tom Lane
David Rowley writes: > I changed things around to make llast() and the int and oid variant > use a new inline function to get the last cell. > I also pushed the resulting code to master. LGTM. regards, tom lane

Re: Small improvements to pg_list.h's linitial(), lsecond(), lthird() etc macros

2020-09-27 Thread David Rowley
On Mon, 28 Sep 2020 at 12:58, Tom Lane wrote: > > David Rowley writes: > > I'm a bit unsure about llast()'s new double evaluation of the list. > > Perhaps I can add a new inline function named list_last_cell() to get > > around that... Or maybe it doesn't matter. I'm not quite sure what's > > bes

RE: Global snapshots

2020-09-27 Thread tsunakawa.ta...@fujitsu.com
Hi Andrey san, all, From: tsunakawa.ta...@fujitsu.com > And please wait. As below, the patent holder just says that Clock-SI is not > based on the patent and an independent development. He doesn't say > Clock-SI does not overlap with the patent or implementing Clock-SI does not > infringe on th

Re: Asynchronous Append on postgres_fdw nodes.

2020-09-27 Thread Kyotaro Horiguchi
Thanks for reviewing. At Sat, 26 Sep 2020 19:45:39 +0900, Etsuro Fujita wrote in > > Come to think of "complex", ExecAsync stuff in this patch might be > > too-much for a short-term solution until executor overhaul, if it > > comes shortly. (the patch of mine here as a whole is like that, > > t

Re: New statistics for tuning WAL buffer size

2020-09-27 Thread Masahiro Ikeda
On 2020-09-26 19:18, Amit Kapila wrote: On Fri, Sep 25, 2020 at 11:06 PM Fujii Masao wrote: On 2020/09/25 12:06, Masahiro Ikeda wrote: > On 2020-09-18 11:11, Kyotaro Horiguchi wrote: >> At Fri, 18 Sep 2020 09:40:11 +0900, Masahiro Ikeda >> wrote in >>> Thanks. I confirmed that it causes HOT p

Re: Range checks of pg_test_fsync --secs-per-test and pg_test_timing --duration

2020-09-27 Thread Michael Paquier
On Fri, Sep 25, 2020 at 07:52:10AM +0200, Peter Eisentraut wrote: > looks good to me Thanks, applied. -- Michael signature.asc Description: PGP signature

Re: Partition prune with stable Expr

2020-09-27 Thread Tom Lane
Andy Fan writes: > Well, that's very interesting. Specific to my user case, > SELECT * FROM p WHERE pkey = to_date('2018-12-13', '-mm-dd)'; > p has 1500+ partitions and planning takes lots of time, which is so same > with SELECT * FROM p WHERE pkey = '2018-12-13', however the planning > time

Re: calling procedures is slow and consumes extra much memory against calling function

2020-09-27 Thread Tom Lane
Pavel Stehule writes: > I am sending another patch that tries to allow CachedPlans for CALL > statements. I think this patch is very accurate, but it is not nice, > because it is smudging very precious reference counting for CachedPlans. I spent some time testing this. Although the #1 patch gets

Re: Partition prune with stable Expr

2020-09-27 Thread Andy Fan
On Mon, Sep 28, 2020 at 7:15 AM Tom Lane wrote: > Andy Fan writes: > > On Mon, Sep 28, 2020 at 4:46 AM David Rowley > wrote: > >> Thanks for showing an interest in partition pruning. Unfortunately, > >> it's not possible to use stable functions to prune partitions during > >> planning. > > > Si

Re: New statistics for tuning WAL buffer size

2020-09-27 Thread Kyotaro Horiguchi
At Sat, 26 Sep 2020 15:48:49 +0530, Amit Kapila wrote in > On Fri, Sep 25, 2020 at 11:06 PM Fujii Masao > wrote: > > > > On 2020/09/25 12:06, Masahiro Ikeda wrote: > > > On 2020-09-18 11:11, Kyotaro Horiguchi wrote: > > >> At Fri, 18 Sep 2020 09:40:11 +0900, Masahiro Ikeda > > >> wrote in > >

RE: Use appendStringInfoString and appendPQExpBufferStr where possible

2020-09-27 Thread Hou, Zhijie
> Good point. There's another one: > $ git grep -E 'appendStringInfoString.*".{,1}");' > src/backend/utils/adt/ruleutils.c: appendStringInfoString(buf, "("); > I noticed you added a similar thread here. > https://commitfest.postgresql.org/30/ > I think this one could be combined as a singl

Planner making bad choice in alternative subplan decision

2020-09-27 Thread David Rowley
This follows on from what I mentioned in [1]. I'll reiterate below: Really the topic is much more broad than what I wrote above, but the example case I have is around subplan costing which results in the alternative subplan code making a bad choice. So the example case is: create table t (a int,

Re: "cert" + clientcert=verify-ca in pg_hba.conf?

2020-09-27 Thread Kyotaro Horiguchi
Hello. At Fri, 25 Sep 2020 13:30:06 -0400, Bruce Momjian wrote in > On Thu, Sep 24, 2020 at 09:59:50PM -0400, Tom Lane wrote: > > Kyotaro Horiguchi writes: > > > Thank you Bruce, Michael. This is a rebased version. > > > > I really strongly object to all the encoded data in this patch. > > One

Re: Small improvements to pg_list.h's linitial(), lsecond(), lthird() etc macros

2020-09-27 Thread Tom Lane
David Rowley writes: > I'm a bit unsure about llast()'s new double evaluation of the list. > Perhaps I can add a new inline function named list_last_cell() to get > around that... Or maybe it doesn't matter. I'm not quite sure what's > best there. Double evaluation bad, especially in a macro that

Re: Parallel INSERT (INTO ... SELECT ...)

2020-09-27 Thread Greg Nancarrow
On Sun, Sep 27, 2020 at 2:03 AM vignesh C wrote: > > I noticed that we are not having any check for skipping temporary > table insertion. > > You should also include temporary tables check here, as parallel > workers might not have access to temporary tables. > Thanks Vignesh, you are right, I n

Re: Partition prune with stable Expr

2020-09-27 Thread Tom Lane
Andy Fan writes: > On Mon, Sep 28, 2020 at 4:46 AM David Rowley wrote: >> Thanks for showing an interest in partition pruning. Unfortunately, >> it's not possible to use stable functions to prune partitions during >> planning. > Sigh.. I understand you now, I ignored the plan can be cached for l

Re: Partition prune with stable Expr

2020-09-27 Thread Andy Fan
Thank you David for coming:) On Mon, Sep 28, 2020 at 4:46 AM David Rowley wrote: > On Mon, 28 Sep 2020 at 08:59, Andy Fan wrote: > > I find we can't prune partitions in the planner if the qual is a stable > function. > > > IMO, we should do it. Why not? > > Thanks for showing an interest in par

Small improvements to pg_list.h's linitial(), lsecond(), lthird() etc macros

2020-09-27 Thread David Rowley
Over in [1] I complained to Tom that I thought he should use list_nth() instead of linitial() and lsecond(). My reasoning was that we already knew that the list contained 2 elements, and linitial() and lsecond() do some additional checks and return NULL if there are fewer than that amount of elemen

Re: Partition prune with stable Expr

2020-09-27 Thread David Rowley
On Mon, 28 Sep 2020 at 08:59, Andy Fan wrote: > I find we can't prune partitions in the planner if the qual is a stable > function. > IMO, we should do it. Why not? Thanks for showing an interest in partition pruning. Unfortunately, it's not possible to use stable functions to prune partitions

Partition prune with stable Expr

2020-09-27 Thread Andy Fan
Hi: I find we can't prune partitions in the planner if the qual is a stable function. CREATE TABLE measurement ( city_id int not null, logdate date not null, peaktempint, unitsales int ) PARTITION BY RANGE (logdate); CREATE TABLE measurement_y2006m02

Re: AppendStringInfoChar instead of appendStringInfoString

2020-09-27 Thread Justin Pryzby
On Fri, Sep 25, 2020 at 08:49:57AM +, Hou, Zhijie wrote: > In (/src/backend/replication/backup_manifest.c) > > I found the following code: > > appendStringInfoString(&buf, "\n"); > appendStringInfoString(&buf, "\""); Good point. There's another one: $ git grep -E 'appendStringI

Re: Fwd: Extending range type operators to cope with elements

2020-09-27 Thread Tom Lane
Esteban Zimanyi writes: > After a long time (as you can imagine, this year everything has been upside > down ...), you will find enclosed the patch for extending the range > operators so they can cope with range element and element range > in addition to the existing range range. Cool. Please

Re: Get rid of runtime handling of AlternativeSubPlan?

2020-09-27 Thread Tom Lane
David Rowley writes: > On Sun, 27 Sep 2020 at 10:03, Tom Lane wrote: >> And if it's true, why would we change all the call sites >> rather than improving the pg_list.h macros? > Maybe we should. Despite the non-NIL check and length check in > list_head(), list_second_cell(), list_third_cell() fu

Re: Fwd: Extending range type operators to cope with elements

2020-09-27 Thread Esteban Zimanyi
Dear all After a long time (as you can imagine, this year everything has been upside down ...), you will find enclosed the patch for extending the range operators so they can cope with range element and element range in addition to the existing range range. Best regards Esteban -

Re: Get rid of runtime handling of AlternativeSubPlan?

2020-09-27 Thread David Rowley
On Sun, 27 Sep 2020 at 10:03, Tom Lane wrote: > > Thanks for reviewing! > > David Rowley writes: > > 1. I think we should be moving away from using linitial() and second() > > when we know there are two items in the list. Using list_nth() has > > less overhead. > > Uh, really? Yeah. Using liniti

Pedagogical example for KNN usage in GiST indexes?

2020-09-27 Thread Esteban Zimanyi
Dear all Is there a pedagogical example showing KNN processing in GiST indexes ? I am looking for something equivalent to the example in file geo_spgist.c. Thanks for your help Esteban