Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-07 Thread Amit Kapila
On Tue, Sep 8, 2020 at 7:53 AM Masahiko Sawada wrote: > > On Mon, 7 Sep 2020 at 15:24, Amit Kapila wrote: > > I'm still going to work on this patch although I might be slow > response this month. > This is a quite fast response. Thanks for staying on top of it. > > > > > 9. While reviewing this

Inconsistency in determining the timestamp of the db statfile.

2020-09-07 Thread Amit Kapila
We use the timestamp of the global statfile if we are not able to determine it for a particular database either because the entry for that database doesn't exist or there is an error while reading the specific database entry. This was not taken care of while reading other entries like ArchiverStats

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: tsunakawa.ta...@fujitsu.com > (1) > + INIT_BUFFERTAG(newTag, > rnode.node, forkNum[j], firstDelBlock[j]); > > And you need to use i here, too. I remember the books "Code Complete" and/or "Readable Code" suggest to use meaningful loop variable names like

Re: Auto-vectorization speeds up multiplication of large-precision numerics

2020-09-07 Thread Amit Khandekar
On Tue, 8 Sep 2020 at 02:19, Tom Lane wrote: > > I wrote: > > I experimented with a few different ideas such as adding restrict > > decoration to the pointers, and eventually found that what works > > is to write the loop termination condition as "i2 < limit" > > rather than "i2 <= limit". It too

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > if (RelFileNodeEquals(bufHdr->tag.rnode, rnode.node) && > + bufHdr->tag.forkNum == forkNum[j] && > + bufHdr->tag.blockNum >= firstDelBlock[j]) > > Here, I think you need to use 'i' not 'j' for forkNum and > firstDelBlock as those are arrays w.r.t forks. That might fix

Re: Reduce the time required for a database recovery from archive.

2020-09-07 Thread Pavel Stehule
út 8. 9. 2020 v 6:51 odesílatel Dmitry Shulga napsal: > Hello hackers, > > Currently, database recovery from archive is performed sequentially, > by reading archived WAL files and applying their records to the database. > > Overall archive file processing is done one by one, and this might > crea

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Masahiko Sawada
On Mon, 7 Sep 2020 at 17:59, Fujii Masao wrote: > > > > On 2020/08/21 15:25, Masahiko Sawada wrote: > > On Fri, 21 Aug 2020 at 00:36, Fujii Masao > > wrote: > >> > >> > >> > >> On 2020/07/27 15:59, Masahiko Sawada wrote: > >>> On Thu, 23 Jul 2020 at 22:51, Muhammad Usama wrote: > > >

Reduce the time required for a database recovery from archive.

2020-09-07 Thread Dmitry Shulga
Hello hackers, Currently, database recovery from archive is performed sequentially, by reading archived WAL files and applying their records to the database. Overall archive file processing is done one by one, and this might create a performance bottleneck if archived WAL files are delivered slow

Re: Improving connection scalability: GetSnapshotData()

2020-09-07 Thread Thomas Munro
On Tue, Sep 8, 2020 at 4:11 PM Andres Freund wrote: > At first I was very confused as to why none of the existing tests have > found this significant issue. But after thinking about it for a minute > that's because they all use psql, and largely separate psql invocations > for each query :(. Which

Re: 回复:how to create index concurrently on partitioned table

2020-09-07 Thread Michael Paquier
On Mon, Sep 07, 2020 at 09:39:16PM -0500, Justin Pryzby wrote: > Also, my previous revision failed to implement your suggestion to first build > catalog entries with INVALID indexes and to then reindex them. Fixed. - childStmt->oldCreateSubid = InvalidSubTransactionId; - childStmt->oldFirstRe

Re: Improving connection scalability: GetSnapshotData()

2020-09-07 Thread Ian Barwick
On 2020/09/08 13:11, Andres Freund wrote: Hi, On 2020-09-08 13:03:01 +0900, Ian Barwick wrote: (...) I wonder if it's possible to increment "xactCompletionCount" during replay along these lines: *** a/src/backend/access/transam/xact.c --- b/src/backend/access/transam/xact.c ***

Re: Improving connection scalability: GetSnapshotData()

2020-09-07 Thread Andres Freund
Hi, On 2020-09-08 13:03:01 +0900, Ian Barwick wrote: > On 2020/09/03 17:18, Michael Paquier wrote: > > On Sun, Aug 16, 2020 at 02:26:57PM -0700, Andres Freund wrote: > > > So we get some builfarm results while thinking about this. > > > > Andres, there is an entry in the CF for this thread: > > h

Re: Remove page-read callback from XLogReaderState.

2020-09-07 Thread Kyotaro Horiguchi
At Tue, 08 Sep 2020 11:56:29 +0900 (JST), Kyotaro Horiguchi wrote in > Thank you for the comment, Menjo-san, and noticing me of that, Michael. I found why the message I was writing was gone from the draft folder.. Sorry for the garbage. -- Kyotaro Horiguchi NTT Open Source Software Center

Re: Improving connection scalability: GetSnapshotData()

2020-09-07 Thread Ian Barwick
On 2020/09/03 17:18, Michael Paquier wrote: On Sun, Aug 16, 2020 at 02:26:57PM -0700, Andres Freund wrote: So we get some builfarm results while thinking about this. Andres, there is an entry in the CF for this thread: https://commitfest.postgresql.org/29/2500/ A lot of work has been committe

Re: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread Amit Kapila
On Mon, Sep 7, 2020 at 1:33 PM k.jami...@fujitsu.com wrote: > > On Wednesday, September 2, 2020 5:49 PM, Amit Kapila wrote: > > On Wed, Sep 2, 2020 at 9:17 AM Tom Lane wrote: > > > > > > Amit Kapila writes: > > > > Even if the relation is locked, background processes like > > > > checkpointer ca

RE: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread tsunakawa.ta...@fujitsu.com
From: Amit Kapila > I intend to say that the global-visibility work can impact this in a > major way and we have analyzed that to some extent during a discussion > on the other thread. So, I think without having a complete > design/solution that addresses both the 2PC and global-visibility, it > i

Re: Logical Replication - detail message with names of missing columns

2020-09-07 Thread Bharath Rupireddy
On Tue, Sep 8, 2020 at 6:50 AM Kyotaro Horiguchi wrote: > > +1 for objective. However, that can be done simpler way that doesn't > need additional loops by using bitmapset to hold missing remote > attribute numbers. This also make the variable "found" useless. > Thanks. I will look into it and po

Re: Online checksums verification in the backend

2020-09-07 Thread Michael Paquier
On Tue, Sep 08, 2020 at 11:36:45AM +0900, Masahiko Sawada wrote: > On Mon, 7 Sep 2020 at 15:59, Michael Paquier wrote: >> In this case it could be better to unregister from the >> CF app for this entry. > > Well, I sent review comments on this patch and Julien fixed all > comments. So I’d marked

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Amit Kapila
On Tue, Sep 8, 2020 at 8:05 AM Fujii Masao wrote: > > On 2020/09/08 10:34, Amit Kapila wrote: > > On Mon, Sep 7, 2020 at 2:29 PM Fujii Masao > > wrote: > >> > >> IMO it's not easy to commit this 2PC patch at once because it's still large > >> and complicated. So I'm thinking it's better to separ

Re: Remove page-read callback from XLogReaderState.

2020-09-07 Thread Kyotaro Horiguchi
Thank you for the comment, Menjo-san, and noticing me of that, Michael. Sorry for late reply. At Fri, 17 Jul 2020 14:14:44 +0900, Takashi Menjo wrote in > Hi, > > I applied your v15 patchset to master > ed2c7f65bd9f15f8f7cd21ad61602f983b1e72e9. Here are three feedback points > for you: > >

Re: [PATCH] Detect escape of ErrorContextCallback stack pointers (and from PG_TRY() )

2020-09-07 Thread Tom Lane
Craig Ringer writes: > Example here: > https://github.com/ringerc/scrapcode/tree/master/c/clang_return_stack_checks > So I find that actually, the __attribute__((callback(fn)) approach is > unnecessary for the purpose proposed. I tested this by injecting some faults of the described sort. diff -

Re: Global snapshots

2020-09-07 Thread Fujii Masao
On 2020/09/05 3:31, Alexey Kondratov wrote: Hi, On 2020-07-27 09:44, Andrey V. Lepikhov wrote: On 7/27/20 11:22 AM, tsunakawa.ta...@fujitsu.com wrote: US8356007B2 - Distributed transaction management for database systems with multiversioning - Google Patents https://patents.google.com/pat

Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation

2020-09-07 Thread Noah Misch
On Tue, Sep 08, 2020 at 10:43:32AM +0900, Kyotaro Horiguchi wrote: > At Tue, 08 Sep 2020 09:13:53 +0900 (JST), Kyotaro Horiguchi > wrote in > > At Mon, 7 Sep 2020 02:32:55 -0700, Noah Misch wrote in > > > As a PoC, this looks promising. Thanks. Would you add a test case such > > > that > >

Re: 回复:how to create index concurrently on partitioned table

2020-09-07 Thread Justin Pryzby
Thanks for completing and pushing the REINDEX patch and others. Here's a rebasified + fixed version of the others. On Tue, Sep 01, 2020 at 02:51:58PM +0900, Michael Paquier wrote: > The REINDEX patch is progressing its way, so I have looked a bit at > the part for CIC. > > Visibly, the case of m

Re: Online checksums verification in the backend

2020-09-07 Thread Masahiko Sawada
On Mon, 7 Sep 2020 at 15:59, Michael Paquier wrote: > > On Tue, Jul 14, 2020 at 11:08:08AM +0200, Julien Rouhaud wrote: > > On Sun, Jul 12, 2020 at 12:34:03PM -0500, Justin Pryzby wrote: > >> Small language fixes in comments and user-facing docs. > > > > Thanks a lot! I just fixed a small issue (

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Fujii Masao
On 2020/09/08 10:34, Amit Kapila wrote: On Mon, Sep 7, 2020 at 2:29 PM Fujii Masao wrote: IMO it's not easy to commit this 2PC patch at once because it's still large and complicated. So I'm thinking it's better to separate the feature into several parts and commit them gradually. Hmm, I

Re: [PATCH] Detect escape of ErrorContextCallback stack pointers (and from PG_TRY() )

2020-09-07 Thread Craig Ringer
On Fri, 4 Sep 2020 at 14:55, Craig Ringer wrote: > Using a test program: > > return_stack_escape.c:14:3: warning: Address of stack memory associated > with local variable 'g' is still referred to by the global variable > 'guard_ptr' upon returning to the caller. This will be a dangling referenc

Re: Resetting spilled txn statistics in pg_stat_replication

2020-09-07 Thread Masahiko Sawada
On Mon, 7 Sep 2020 at 15:24, Amit Kapila wrote: > > On Thu, Jul 23, 2020 at 11:46 AM Masahiko Sawada > wrote: > > > > I've updated the patch so that the stats collector ignores the > > 'update' message if the slot stats array is already full. > > > > This patch needs a rebase. I don't see this pa

Re: Evaluate expression at planning time for two more cases

2020-09-07 Thread Tom Lane
Surafel Temesgen writes: > [ null_check_on_pkey_optimization_v1.patch ] I took a very brief look at this. > I don’t add NOT NULL constraint optimization to the patch because cached > plan is not invalidation in case of a change in NOT NULL constraint That's actually not a problem, even though s

Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation

2020-09-07 Thread Kyotaro Horiguchi
At Tue, 08 Sep 2020 09:13:53 +0900 (JST), Kyotaro Horiguchi wrote in > At Mon, 7 Sep 2020 02:32:55 -0700, Noah Misch wrote in > > As a PoC, this looks promising. Thanks. Would you add a test case such > > that > > the following demonstrates the bug in the absence of your PoC? > > > > pri

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Amit Kapila
On Mon, Sep 7, 2020 at 2:29 PM Fujii Masao wrote: > > IMO it's not easy to commit this 2PC patch at once because it's still large > and complicated. So I'm thinking it's better to separate the feature into > several parts and commit them gradually. > Hmm, I don't see that we have a consensus on t

Re: Logical Replication - detail message with names of missing columns

2020-09-07 Thread Kyotaro Horiguchi
Thank you for working on this. At Mon, 7 Sep 2020 16:30:59 +0530, Bharath Rupireddy wrote in > Hi, > > I observed that, in logical replication when a subscriber is missing > some columns, it currently emits an error message that says "some" > columns are missing(see logicalrep_rel_open()), but

Re: Division in dynahash.c due to HASH_FFACTOR

2020-09-07 Thread Thomas Munro
On Sat, Sep 5, 2020 at 2:34 AM Alvaro Herrera wrote: > On 2020-Sep-04, Jakub Wartak wrote: > > After removing HASH_FFACTOR PostgreSQL still compiles... Would > > removing it break some external API/extensions ? > > FWIW, HASH_FFACTOR has *never* been used in Postgres core code. > > https://postgr

Re: pgbench and timestamps (bounced)

2020-09-07 Thread Tom Lane
Fabien COELHO writes: > [Resent on hackers for CF registration, sorry for the noise] For the record, the original thread is at https://www.postgresql.org/message-id/flat/CAKVUGgQaZVAUi1Ex41H4wrru%3DFU%2BMfwgjG0aM1br6st7sz31Vw%40mail.gmail.com (I tried but failed to attach that thread to the CF

Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation

2020-09-07 Thread Kyotaro Horiguchi
At Mon, 7 Sep 2020 02:32:55 -0700, Noah Misch wrote in > On Mon, Sep 07, 2020 at 05:40:36PM +0900, Kyotaro Horiguchi wrote: > > At Mon, 07 Sep 2020 13:45:28 +0900 (JST), Kyotaro Horiguchi > > wrote in > > > The cause is that the worker had received pending-sync entry correctly > > > but not ne

Re: Optimising compactify_tuples()

2020-09-07 Thread Thomas Munro
On Mon, Sep 7, 2020 at 7:48 PM David Rowley wrote: > I was wondering today if we could just get rid of the sort in > compactify_tuples() completely. It seems to me that the existing sort > is there just so that the memmove() is done in order of tuple at the > end of the page first. We seem to be j

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-07, Alvaro Herrera wrote: > Ah, it looks like we can get away with initializing the RRI to 0, and > then explicitly handle that case in ExecPartitionCheckEmitError, as in > the attached (which means reindenting, but I left it alone to make it > easy to read). Well, that was silly -- t

Re: A micro-optimisation for walkdir()

2020-09-07 Thread Tom Lane
Thomas Munro writes: > FWIW I just spotted a couple of very suspicious looking failures for > build farm animal "walleye", a "MinGW64 8.1.0" system, that say: walleye's been kind of unstable since the get-go, so I wouldn't put too much faith in reports just from it. regar

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
Ah, it looks like we can get away with initializing the RRI to 0, and then explicitly handle that case in ExecPartitionCheckEmitError, as in the attached (which means reindenting, but I left it alone to make it easy to read). It kinda sucks because we don't report the tuple that causes the error,

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-07, Alvaro Herrera wrote: > Well, they are fake in that the ri_RangeTableIndex they carry is bogus, > which means that ExecBuildSlotValueDescription will misbehave if the > partitioned default partition has a different column order than its > parent. That can be evidenced by changing

Re: default partition and concurrent attach partition

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-04, Amit Langote wrote: Hello > FWIW, I still prefer "minimally valid ResultRelInfo" to "fake > ResultRelInfo", because those aren't really fake, are they? They are > as valid as any other ResultRelInfo as far I can see. I said > "minimally valid" because a fully-valid partition Res

Re: A micro-optimisation for walkdir()

2020-09-07 Thread Thomas Munro
On Tue, Sep 8, 2020 at 2:05 AM Juan José Santamaría Flecha wrote: > On Mon, Sep 7, 2020 at 1:41 PM Thomas Munro wrote: >> Thanks for confirming. I ran the Windows patch through pgindent, >> fixed a small typo, and pushed. > > Great, thanks. Should we include something from this discussion as com

Re: [PATCH] audo-detect and use -moutline-atomics compilation flag for aarch64

2020-09-07 Thread Tom Lane
"Zidenberg, Tsahi" writes: > Outline-atomics is a gcc compilation flag that adds runtime detection of > weather or not the cpu supports atomic instructions. CPUs that don't support > atomic instructions will use the old load-exclusive/store-exclusive > instructions. If a different compilation f

Re: v13: show extended stats target in \d

2020-09-07 Thread Peter Geoghegan
On Sun, Sep 6, 2020 at 1:48 PM Justin Pryzby wrote: > On Sun, Sep 06, 2020 at 01:06:05PM -0700, Peter Geoghegan wrote: > > How about a line break? That seems like a simple solution that takes > > all the competing concerns into account. > Like this ? > Statistics objects: > "public"."t" (ndi

Re: Disk-based hash aggregate's cost model

2020-09-07 Thread Jeff Davis
On Sun, 2020-09-06 at 23:21 +0200, Tomas Vondra wrote: > I've tested the costing changes on the simplified TPC-H query, on two > different machines, and it seems like a clear improvement. Thank you. Committed. > So yeah, the patched costing is much closer to sort (from the point > of > this cost/

Re: Auto-vectorization speeds up multiplication of large-precision numerics

2020-09-07 Thread Tom Lane
I wrote: > I experimented with a few different ideas such as adding restrict > decoration to the pointers, and eventually found that what works > is to write the loop termination condition as "i2 < limit" > rather than "i2 <= limit". It took me a long time to think of > trying that, because it see

Re: Improving connection scalability: GetSnapshotData()

2020-09-07 Thread Andres Freund
Hi, On Mon, Sep 7, 2020, at 07:20, Konstantin Knizhnik wrote: > >> And which pgbench database scale factor you have used? > > 200 > > > > Another thing you could try is to run 2-4 pgench instances in different > > databases. > I tried to reinitialize database with scale 200 but there was no > si

Re: Inaccurate comment, for family-1 polymorphic UNKNOWN type arguments

2020-09-07 Thread Tom Lane
Himanshu Upadhyaya writes: > I observed that we have inaccurate comment in > enforce_generic_type_consistency. Fair point, but that's not the only old comment that's not being careful about it. I applied the attached. regards, tom lane diff --git a/src/backend/parser/pa

Meaning of pg_stats_subscription.latest_end_time

2020-09-07 Thread Simon.Schneider
Hi, having just spoken with Harald Armin Massa of 2ndQuadrant, he encouraged me to ask my question here. This is about publisher/subscriber replication and was originally posted in german to pgsql-de-allgem...@lists.postgresql.org. According to documentation, the field pg_stats_subscription.lat

Re: Improving connection scalability: GetSnapshotData()

2020-09-07 Thread Konstantin Knizhnik
On 06.09.2020 21:52, Andres Freund wrote: Hi, On 2020-09-05 16:58:31 +0300, Konstantin Knizhnik wrote: On 04.09.2020 21:53, Andres Freund wrote: I also used huge_pages=on / configured them on the OS level. Otherwise TLB misses will be a significant factor. As far as I understand there shou

Re: Auto-vectorization speeds up multiplication of large-precision numerics

2020-09-07 Thread Tom Lane
Amit Khandekar writes: > On Mon, 7 Sep 2020 at 11:23, Tom Lane wrote: >> BTW, poking at this further, it seems that the patch only really >> works for gcc. clang accepts the -ftree-vectorize switch, but >> looking at the generated asm shows that it does nothing useful. >> Which is odd, because c

Re: factorial function/phase out postfix operators?

2020-09-07 Thread John Naylor
On Thu, Sep 3, 2020 at 11:50 AM Mark Dilger wrote: > [v7] Ok, I've marked it ready for committer. -- John Naylorhttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Yet another fast GiST build

2020-09-07 Thread Andrey M. Borodin
> 7 сент. 2020 г., в 19:10, Heikki Linnakangas написал(а): > > On 07/09/2020 13:59, Pavel Borisov wrote: > I suppose there is a big jump in integer value (whether signed or > unsigned) as you cross from positive to negative floats, and then the > sort order is reversed. I have no

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-09-07 Thread Tom Lane
Michael Paquier writes: > Is there still something that needs to absolutely be done here knowing > that we have bab1500 that got rid of the root issue? Can the CF entry > be marked as committed? I think there is agreement that we're not going to change cancel_before_shmem_exit's restriction to o

Re: [doc] minor wording improvement in a couple of places

2020-09-07 Thread John Naylor
On Mon, Sep 7, 2020 at 6:30 AM Magnus Hagander wrote: > > On Mon, Sep 7, 2020 at 6:34 AM Ian Barwick > wrote: >>"..., which see for additional details." >> >> which strikes me as a bit odd. Suggested phrasing: >> >>"...; see this file for additional details." >> > > I agree this sounds v

Re: Change a constraint's index - ALTER TABLE ... ALTER CONSTRAINT ... USING INDEX ...

2020-09-07 Thread Alvaro Herrera
On 2020-Sep-07, Laurenz Albe wrote: > This patch would provide a more convenient way to do that. > > Again, I am not sure if that justifies the effort. I have to admit I've seen cases where it'd be useful to have included columns in primary keys. TBH I think if we really wanted the feature of p

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Fujii Masao
On 2020/09/07 17:59, Fujii Masao wrote: On 2020/08/21 15:25, Masahiko Sawada wrote: On Fri, 21 Aug 2020 at 00:36, Fujii Masao wrote: On 2020/07/27 15:59, Masahiko Sawada wrote: On Thu, 23 Jul 2020 at 22:51, Muhammad Usama wrote: On Wed, Jul 22, 2020 at 12:42 PM Masahiko Sawada

Re: Change a constraint's index - ALTER TABLE ... ALTER CONSTRAINT ... USING INDEX ...

2020-09-07 Thread Laurenz Albe
On Fri, 2020-09-04 at 13:31 -0400, Alvaro Herrera wrote: > On 2020-Sep-04, Laurenz Albe wrote: > > On Fri, 2020-09-04 at 10:41 -0400, Alvaro Herrera wrote: > > > > The value I see in this is: > > > > - replacing a primary key index > > > > - replacing the index behind a constraint targeted by a for

Re: 回复:how to create index concurrently on partitioned table

2020-09-07 Thread Anastasia Lubennikova
On 07.09.2020 04:58, Michael Paquier wrote: On Fri, Sep 04, 2020 at 09:51:13AM +0900, Michael Paquier wrote: Glad to hear that, please take the time you need. Attached is a rebased version to address the various conflicts after 844c05a. -- Michael Thank you. With the fix for the cycle in rei

Re: A micro-optimisation for walkdir()

2020-09-07 Thread Juan José Santamaría Flecha
On Mon, Sep 7, 2020 at 1:41 PM Thomas Munro wrote: > On Mon, Sep 7, 2020 at 9:42 PM Magnus Hagander > wrote: > > On Mon, Sep 7, 2020 at 12:27 AM Thomas Munro > wrote: > >> I think the following is a little mysterious, but it does seem to be > >> what people do for this in other projects. It is

Re: shared-memory based stats collector

2020-09-07 Thread Stephen Frost
Greetings, * Stephen Frost (sfr...@snowman.net) wrote: > * Kyotaro Horiguchi (horikyota@gmail.com) wrote: > > The commit 8e19a82640 changed SIGQUIT handler of almost all processes > > not to run atexit callbacks for safety. Archiver process should behave > > the same way for the same reason. E

Re: PG 13 release notes, first draft

2020-09-07 Thread Justin Pryzby
There's some obvious improvements for which I include a patch. >Add alternate version of jsonb_setI() with special NULL handling (Andrew >Dunstan) >The new function, jsonb_set_lax(), allows null new values to either set the >specified key to JSON null, delete the key, raise exception, or ignore

Re: [HACKERS] logical decoding of two-phase transactions

2020-09-07 Thread Amit Kapila
On Mon, Sep 7, 2020 at 10:54 AM Ajin Cherian wrote: > > Hello, > > Trying to revive this patch which attempts to support logical decoding of two > phase transactions. I've rebased and polished Nikhil's patch on the current > HEAD. Some of the logic in the previous patchset has already been commi

Re: Strange behavior with polygon and NaN

2020-09-07 Thread gkokolatos
‐‐‐ Original Message ‐‐‐ On Thursday, 27 August 2020 14:24, Kyotaro Horiguchi wrote: > At Wed, 26 Aug 2020 08:18:49 -0400, Tom Lane t...@sss.pgh.pa.us wrote in > > > Kyotaro Horiguchi horikyota@gmail.com writes: > > > > > At Tue, 25 Aug 2020 19:03:50 -0400, Bruce Momjian br...@m

Re: proposal: possibility to read dumped table's name from file

2020-09-07 Thread Surafel Temesgen
Hi Pavel On Fri, Sep 4, 2020 at 6:22 AM Pavel Stehule wrote: > > Here is updated patch for pg_dump > > pg_dumpall also has –exclude-database=pattern and –no-comments option doesn't that qualify it to benefits from this feature? And please add a test case for this option regards Surafel

Inaccurate comment, for family-1 polymorphic UNKNOWN type arguments

2020-09-07 Thread Himanshu Upadhyaya
Hi Hackers, I observed that we have inaccurate comment in enforce_generic_type_consistency. if (!OidIsValid(elem_typeid)) { if (allow_poly) { elem_typeid = ANYELEMENTOID;

Re: A micro-optimisation for walkdir()

2020-09-07 Thread Thomas Munro
On Mon, Sep 7, 2020 at 9:42 PM Magnus Hagander wrote: > On Mon, Sep 7, 2020 at 12:27 AM Thomas Munro wrote: >> I think the following is a little mysterious, but it does seem to be >> what people do for this in other projects. It is the documented way >> to detect mount points, and I guess IO_REP

Re: Ideas about a better API for postgres_fdw remote estimates

2020-09-07 Thread Andrey V. Lepikhov
On 9/4/20 6:23 PM, Ashutosh Bapat wrote: On Thu, 3 Sep 2020 at 10:44, Andrey V. Lepikhov mailto:a.lepik...@postgrespro.ru>> wrote: On 8/31/20 6:19 PM, Ashutosh Bapat wrote: > On Mon, Aug 31, 2020 at 3:36 PM Andrey V. Lepikhov > mailto:a.lepik...@postgrespro.ru>> wrote: >>

Logical Replication - detail message with names of missing columns

2020-09-07 Thread Bharath Rupireddy
Hi, I observed that, in logical replication when a subscriber is missing some columns, it currently emits an error message that says "some" columns are missing(see logicalrep_rel_open()), but it doesn't specify what the missing column names are. The comment there also says that finding the missing

Re: Parallel copy

2020-09-07 Thread vignesh C
On Tue, Sep 1, 2020 at 3:39 PM Greg Nancarrow wrote: > > Hi Vignesh, > > >Can you share with me the script you used to generate the data & the ddl of > >the table, so that it will help me check that >scenario you faced the > >>problem. > > Unfortunately I can't directly share it (considered comp

Re: Yet another fast GiST build

2020-09-07 Thread Pavel Borisov
> > > >> I suppose there is a big jump in integer value (whether signed or > >> unsigned) as you cross from positive to negative floats, and then the > >> sort order is reversed. I have no idea if either of those things is a > >> problem worth fixing. That made me wonder if there might also be an

Re: Bogus documentation for bogus geometric operators

2020-09-07 Thread Emre Hasegeli
> Emre, the CF bot complains that this does not apply anymore, so please > provide a rebase. By the way, I am a bit confused to see a patch > that adds new operators on a thread whose subject is about > documentation. Rebased version is attached. The subject is about the documentation, but the p

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-07 Thread Andrey V. Lepikhov
On 9/7/20 12:26 PM, Michael Paquier wrote: On Mon, Aug 24, 2020 at 06:19:28PM +0900, Amit Langote wrote: On Mon, Aug 24, 2020 at 4:18 PM Amit Langote wrote: I would Oops, thought I'd continue writing, but hit send before actually doing that. Please ignore. I have some comments on v6, which

Re: [doc] minor wording improvement in a couple of places

2020-09-07 Thread Magnus Hagander
On Mon, Sep 7, 2020 at 6:34 AM Ian Barwick wrote: > Hi > > On these pages: > >https://www.postgresql.org/docs/current/fdw-callbacks.html >https://www.postgresql.org/docs/current/tablesample-method.html > > we have the phrase: > >"..., which see for additional details." > > which strik

Re: Allow CURRENT_ROLE in GRANTED BY

2020-09-07 Thread Asif Rehman
The following review has been posted through the commitfest application: make installcheck-world: tested, passed Implements feature: tested, passed Spec compliant: tested, passed Documentation:tested, passed The patch applies cleanly and looks fine to me. However would

Re: Yet another fast GiST build

2020-09-07 Thread Heikki Linnakangas
On 24/02/2020 10:50, Andrey M. Borodin wrote: On 24 февр. 2020 г., at 01:58, Thomas Munro wrote: On Thu, Feb 20, 2020 at 10:14 AM Thomas Munro wrote: 1. We expect floats to be in IEEE format, and the sort order of IEEE floats is mostly correlated to the binary sort order of the bits reinterp

Re: A micro-optimisation for walkdir()

2020-09-07 Thread Magnus Hagander
On Mon, Sep 7, 2020 at 12:27 AM Thomas Munro wrote: > On Sun, Sep 6, 2020 at 5:23 AM Juan José Santamaría Flecha > wrote: > > On Sat, Sep 5, 2020 at 2:13 AM Andres Freund wrote: > >> > However, it looks like we might be missing a further opportunity > >> > here... Doesn't Windows already give

Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation

2020-09-07 Thread Noah Misch
On Mon, Sep 07, 2020 at 05:40:36PM +0900, Kyotaro Horiguchi wrote: > At Mon, 07 Sep 2020 13:45:28 +0900 (JST), Kyotaro Horiguchi > wrote in > > The cause is that the worker had received pending-sync entry correctly > > but not never created a relcache entry for the relation using > > RelationBui

Re: Transactions involving multiple postgres foreign servers, take 2

2020-09-07 Thread Fujii Masao
On 2020/08/21 15:25, Masahiko Sawada wrote: On Fri, 21 Aug 2020 at 00:36, Fujii Masao wrote: On 2020/07/27 15:59, Masahiko Sawada wrote: On Thu, 23 Jul 2020 at 22:51, Muhammad Usama wrote: On Wed, Jul 22, 2020 at 12:42 PM Masahiko Sawada wrote: On Sat, 18 Jul 2020 at 01:55, Fuji

Re: Online checksums verification in the backend

2020-09-07 Thread Michael Paquier
On Mon, Sep 07, 2020 at 09:38:30AM +0200, Julien Rouhaud wrote: > Did you mean creating a new checksumfuncs.c file? I don't find any > such file in the current tree. Your patch adds checksumfuncs.c, so the subroutines grabbing a given block could just be moved there. > I'm not sure I understand.

Re: v13: CLUSTER segv with wal_level=minimal and parallel index creation

2020-09-07 Thread Kyotaro Horiguchi
At Mon, 07 Sep 2020 13:45:28 +0900 (JST), Kyotaro Horiguchi wrote in > The cause is that the worker had received pending-sync entry correctly > but not never created a relcache entry for the relation using > RelationBuildDesc. So the rd_firstRelfilenodeSubid is not correctly > set. > > I'm inve

Re: Yet another fast GiST build (typo)

2020-09-07 Thread Andrey M. Borodin
> 7 сент. 2020 г., в 12:14, Andrey M. Borodin написал(а): > > Maybe I'm missing something? Like forgot to log 10% of pages, or something > like that... Indeed, there was a bug. I've fixed it, and I still observe same performance gain. Best regards, Andrey Borodin. v2-0001-nbtree-faster-l

Re: Range checks of pg_test_fsync --secs-per-test and pg_test_timing --duration

2020-09-07 Thread Peter Eisentraut
On 2020-09-06 05:04, Michael Paquier wrote: I would allow 0. It's not very useful, but it's not harmful and could be applicable in testing. Hmm, OK. For pg_test_fsync, 0 means infinity, and for pg_test_timing that means stopping immediately (we currently don't allow that). How does this apply

RE: [Patch] Optimize dropping of relation buffers using dlist

2020-09-07 Thread k.jami...@fujitsu.com
On Wednesday, September 2, 2020 5:49 PM, Amit Kapila wrote: > On Wed, Sep 2, 2020 at 9:17 AM Tom Lane wrote: > > > > Amit Kapila writes: > > > Even if the relation is locked, background processes like > > > checkpointer can still touch the relation which might cause > > > problems. Consider a cas

Re: Improvements in Copy From

2020-09-07 Thread Surafel Temesgen
Hi Vignesh On Wed, Jul 1, 2020 at 3:46 PM vignesh C wrote: > Hi, > > While reviewing copy from I identified few improvements for copy from > that can be done : > a) copy from stdin copies lesser amount of data to buffer even though > space is available in buffer because minread was passed as 1

Re: Optimising compactify_tuples()

2020-09-07 Thread David Rowley
On Sun, 6 Sep 2020 at 23:37, David Rowley wrote: > The test replayed ~2.2 GB of WAL. master took 148.581 seconds and > master+0001+0002 took 115.588 seconds. That's about 28% faster. Pretty > nice! I was wondering today if we could just get rid of the sort in compactify_tuples() completely. It se

Re: Issue with cancel_before_shmem_exit while searching to remove a particular registered exit callbacks

2020-09-07 Thread Michael Paquier
On Mon, Aug 10, 2020 at 10:10:08PM -0400, Robert Haas wrote: > That split dates to the parallel query work, and there are some > comments in shmem_exit() about it; see in particular the explanation > in the middle where it says "Call dynamic shared memory callbacks." It > seemed to me that I needed

Re: Online checksums verification in the backend

2020-09-07 Thread Julien Rouhaud
On Mon, Sep 7, 2020 at 8:59 AM Michael Paquier wrote: > > +#include "postgres.h" > + > +#include "access/tupdesc.h" > +#include "common/relpath.h" > #include "storage/block.h" > +#include "utils/relcache.h" > +#include "utils/tuplestore.h" > [...] > +extern bool check_one_block(Relation relation,

Re: PATCH: logical_work_mem and logical streaming of large in-progress transactions

2020-09-07 Thread Dilip Kumar
On Mon, Sep 7, 2020 at 12:00 PM Amit Kapila wrote: > On Sat, Sep 5, 2020 at 8:55 PM Dilip Kumar wrote: > > > > > > I have reviewed the changes and looks fine to me. > > > > Thanks, I have pushed the last patch. Let's wait for a day or so to > see the buildfarm reports and then we can probably cl

Re: [POC] Fast COPY FROM command for the table with foreign partitions

2020-09-07 Thread Michael Paquier
On Mon, Aug 24, 2020 at 06:19:28PM +0900, Amit Langote wrote: > On Mon, Aug 24, 2020 at 4:18 PM Amit Langote wrote: > > I would > > Oops, thought I'd continue writing, but hit send before actually doing > that. Please ignore. > > I have some comments on v6, which I will share later this week.

Re: [PATCH] Keeps tracking the uniqueness with UniqueKey

2020-09-07 Thread Michael Paquier
On Tue, Aug 04, 2020 at 06:59:50AM +0800, Andy Fan wrote: > You are correct, I would include this in the next version patch, Thank you > for this checking! Regression tests are failing with this patch set applied. The CF bot says so, and I can reproduce that locally as well. Could you look at th

Re: New statistics for tuning WAL buffer size

2020-09-07 Thread Fujii Masao
On 2020/09/07 9:58, Masahiro Ikeda wrote: Thanks for the review and advice! On 2020-09-03 16:05, Fujii Masao wrote: On 2020/09/02 18:56, Masahiro Ikeda wrote: +/* -- + * Backend types + * -- You seem to forget to add "*/" into the above comment. This issue could cause the f

Re: Online checksums patch - once again

2020-09-07 Thread Michael Paquier
On Wed, Sep 02, 2020 at 02:22:25PM +0200, Daniel Gustafsson wrote: > I unfortunately haven't had time to read the READ ONLY patch so I can't > comment > on how these two patches do things in relation to each other. > > The main synchronization mechanisms are the use of the inprogress mode where >

Re: Yet another fast GiST build (typo)

2020-09-07 Thread Andrey M. Borodin
> 6 сент. 2020 г., в 18:26, Heikki Linnakangas написал(а): > > On 05/09/2020 14:53, Andrey M. Borodin wrote: >> Thanks for ideas, Heikki. Please see v13 with proposed changes. > > Thanks, that was quick! > >> But I've found out that logging page-by-page slows down GiST build by >> approximate

Re: Auto-vectorization speeds up multiplication of large-precision numerics

2020-09-07 Thread Amit Khandekar
On Mon, 7 Sep 2020 at 11:23, Tom Lane wrote: > > I wrote: > > I made some cosmetic changes to this and committed it. Thanks! > > BTW, poking at this further, it seems that the patch only really > works for gcc. clang accepts the -ftree-vectorize switch, but > looking at the generated asm shows

Re: Online checksums verification in the backend

2020-09-07 Thread Michael Paquier
On Tue, Jul 14, 2020 at 11:08:08AM +0200, Julien Rouhaud wrote: > On Sun, Jul 12, 2020 at 12:34:03PM -0500, Justin Pryzby wrote: >> Small language fixes in comments and user-facing docs. > > Thanks a lot! I just fixed a small issue (see below), PFA updated v10. Sawada-san, you are registered as