Re: print_path is missing GatherMerge and CustomScan support

2018-07-27 Thread Michael Paquier
On Fri, Jul 27, 2018 at 03:40:48PM +0900, Etsuro Fujita wrote: > (I think that at least currently, there is no need for the Gather and > GatherMerge cases in reparameterize_path_by_child, but I don't object to > keeping those as-is there.) Let's keep them. As far as my understanding goes, which i

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-07-27 Thread Amit Langote
On 2018/07/27 1:19, David Rowley wrote: > On 18 July 2018 at 20:29, Amit Langote wrote: >> Let me know what you think of the code in the updated patch. > > Thanks for sending the updated patch. > > I looked over it tonight and made a number of changes: > > 1) Got rid of PARTITION_ROUTING_MAXSIZ

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2018-07-27 Thread Ashutosh Bapat
On Thu, Jul 26, 2018 at 8:37 PM, Dmitry Dolgov <9erthali...@gmail.com> wrote: >> On Mon, 23 Jul 2018 at 10:38, Ashutosh Bapat >> wrote: >> >> On Fri, Jul 20, 2018 at 3:13 AM, Dmitry Dolgov <9erthali...@gmail.com> wrote: >> > >> > It's of course wrong, it's going to be O(max(m, n)) as you said, bu

Re: Early WIP/PoC for inlining CTEs

2018-07-27 Thread David Fetter
On Fri, Jul 27, 2018 at 02:55:26PM +1200, Thomas Munro wrote: > On Thu, Jul 26, 2018 at 7:14 AM, David Fetter wrote: > > Please find attached the next version, which passes 'make check'. > > ... but not 'make check-world' (contrib/postgres_fdw's EXPLAIN is different). Please find attached a patc

Re: [PATCH] Improve geometric types

2018-07-27 Thread Kyotaro HORIGUCHI
Thank you for taking this. At Thu, 26 Jul 2018 17:12:50 +0200, Tomas Vondra wrote in <672f4c42-6742-c1ec-e9a4-1994b815a...@2ndquadrant.com> > On 07/11/2018 07:13 PM, Emre Hasegeli wrote: > > New versions are attached after the patch got in. I noticed > > tests failing on Windows [1] and added

RE: Temporary tables prevent autovacuum, leading to XID wraparound

2018-07-27 Thread Tsunakawa, Takayuki
Thank you, Michael and Horiguchi-san, From: Michael Paquier [mailto:mich...@paquier.xyz] > autovacuum.c is a pretty bad place for stuff as namespace.c holds all the > logic related to temporary tablespaces, so I renamed the routine to > isTempNamespaceInUse and moved it there. I don't have a stro

Re: [HACKERS] [PATCH] Overestimated filter cost and its mitigation

2018-07-27 Thread Ashutosh Bapat
On Thu, Jul 26, 2018 at 10:30 PM, Dean Rasheed wrote: > On 26 July 2018 at 07:12, Ashutosh Bapat > wrote: >> In the patch clauselist_selectivity() gets called repeatedly for every >> new qual added to the clause list. Instead, if we try to combine the >> cost/row estimation with order_qual_clause

Auditing via logical decoding

2018-07-27 Thread Philip Scott
Hi Postgres Hackers, We have been using our own trigger-based audit system at my firm successfully for some years, but the performance penalty is starting to grate a bit and so I have been tasked with seeing if we can make use of the new logical decoding functions to achieve the same thing. I

Re: Auditing via logical decoding

2018-07-27 Thread Jeremy Finzel
On Fri, Jul 27, 2018 at 5:41 AM Philip Scott wrote: > Hi Postgres Hackers, > > We have been using our own trigger-based audit system at my firm > successfully for some years, but the performance penalty is starting to > grate a bit and so I have been tasked with seeing if we can make use of > the

My Skype account (korotkovae) was hacked

2018-07-27 Thread Alexander Korotkov
Hi! Some community members did communicate with me using my Skype account (korotkovae). Today this account was hacked. You should know that any requests to send money sent from my account are fraud. Sorry for inconvenience. I've send request to restore my Skype account, it will be under conside

Inconsistent error message in bt_page_items_bytea().

2018-07-27 Thread Ashutosh Sharma
Hi All, All the pageinspect functions dealing with raw page has the error message as "must be superuser to use raw page function" however, that's not true for bt_page_items_bytea() which has "must be superuser to use pageinspect functions". This seems to me like a copy paste error which got trans

Re: pgbench - very minor bug fix on hash() missing argument

2018-07-27 Thread Fabien COELHO
Hello Michaël, Thanks, committed and back-patched. Ok. I have added some tests for least() and greatest() on the way. Good! Thanks, -- Fabien.

Re: partition tree inspection functions

2018-07-27 Thread Jesper Pedersen
Hi Amit, On 07/26/2018 10:33 PM, Amit Langote wrote: Optional parameter sounds good, so made it get_partition_level(regclass [ , regclass ]) in the updated patch. Although, adding that argument is not without possible surprises its result might evoke. Like, what happens if you try to find the

Re: Removing unneeded self joins

2018-07-27 Thread Alexander Kuzmenkov
Here is a current version of the patch, still rather experimental. Since the previous version, I fixed some bugs and added the possibility to remove a relation even when it is mentioned in target lists. I have to rewrite all references to the removed relation in targetlists and the equivalence

Re: Explain buffers wrong counter with parallel plans

2018-07-27 Thread Amit Kapila
On Thu, Jul 26, 2018 at 7:31 PM, Jonathan S. Katz wrote: > >> On Jul 25, 2018, at 10:25 PM, Amit Kapila wrote: >> >> >> You mean to say the number (Buffers: shared read=442478) in HEAD, >> right? If so, yeah, I am also wondering why the results of the patch >> are different in HEAD and 11beta2.

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Robert Haas
On Thu, Jul 26, 2018 at 11:00 PM, Stephen Frost wrote: > This is a killer point here- clearly the people who have been > contributing to PG aren't going to complain about their contributions > being released as part of some other work which has a different license > or they'd have gone after the m

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-27 Thread Nikhil Sontakke
>> PFA, latest patchset, which completely removes the earlier >> LogicalLock/LogicalUnLock implementation using groupDecode stuff and >> uses the newly suggested approach of checking the currently decoded >> XID for abort in systable_* API functions. Much simpler to code and >> easier to test as we

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Robert Haas
On Wed, Jul 25, 2018 at 10:53 AM, David Fetter wrote: > What made PostgreSQL attractive to those companies in the first place > was a known lack of need to have Extensive Conversations with Legal™ > about licensing and other financial/IP matters. If you think that the lack of a CLA and a patent g

Re: Auditing via logical decoding

2018-07-27 Thread Tomas Vondra
On 07/27/2018 12:44 PM, Philip Scott wrote: Hi Postgres Hackers, We have been using our own trigger-based audit system at my firm successfully for some years, but the performance penalty is starting to grate a bit and so I have been tasked with seeing if we can make use of the new logical

Re: PartitionDispatch's partdesc field

2018-07-27 Thread Robert Haas
On Wed, Jul 25, 2018 at 10:42 PM, Amit Langote wrote: >> Another alternative, which I think might make more sense, is to make >> use pd->key and pd->partdesc in preference to pd->reldesc->rd_partkey >> and pd->reldesc->rd_partdesc. It seems to me that the idea of the >> PartitionDispatch structur

Re: Alter index rename concurrently to

2018-07-27 Thread Robert Haas
On Wed, Jul 18, 2018 at 5:20 AM, Peter Eisentraut wrote: > In the 2012 thread, it was mentioned several times that some thought > that renaming without an exclusive lock was unsafe, but without any > further reasons. I think that was before MVCC catalog snapshots were > implemented, so at that ti

RE: Auditing via logical decoding

2018-07-27 Thread from_postgres
>> We have been using our own trigger-based audit system at my firm >> successfully for some years, but the performance penalty is starting to >> grate a bit and so I have been tasked with seeing if we can make use of >> the new logical decoding functions to achieve the same thing. I thought >>

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Tom Lane
Robert Haas writes: > On Thu, Jul 26, 2018 at 11:00 PM, Stephen Frost wrote: >> This is a killer point here- clearly the people who have been >> contributing to PG aren't going to complain about their contributions >> being released as part of some other work which has a different license >> or t

Re: Auditing via logical decoding

2018-07-27 Thread Pavel Stehule
2018-07-27 16:43 GMT+02:00 : > >> We have been using our own trigger-based audit system at my firm > >> successfully for some years, but the performance penalty is starting to > >> grate a bit and so I have been tasked with seeing if we can make use of > >> the new logical decoding functions to ac

Re: grammar - src/backend/access/heap/README.tuplock

2018-07-27 Thread Alvaro Herrera
I agree with all you say here; these were all my mistakes. Pushed patch, thanks. -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Locking B-tree leafs immediately in exclusive mode

2018-07-27 Thread Alexander Korotkov
On Fri, Jul 27, 2018 at 12:30 AM Simon Riggs wrote: > On 26 July 2018 at 20:59, Alexander Korotkov > wrote: > > > Great, thank you! So, I think the regression is demystified. We can > > now conclude that on our benchmarks this patch doesn't cause > > performance regression larger than measurem

Re: Auditing via logical decoding

2018-07-27 Thread Tomas Vondra
On 07/27/2018 04:43 PM, from_postg...@safetyphil.com wrote: We have been using our own trigger-based audit system at my firm successfully for some years, but the performance penalty is starting to grate a bit and so I have been tasked with seeing if we can make use of the new logical decoding

Re: Auditing via logical decoding

2018-07-27 Thread Stephen Frost
Greetings, * from_postg...@safetyphil.com (from_postg...@safetyphil.com) wrote: > >> We have been using our own trigger-based audit system at my firm > >> successfully for some years, but the performance penalty is starting to > >> grate a bit and so I have been tasked with seeing if we can make

add verbosity to pg_basebackup for sync

2018-07-27 Thread Jeff Janes
On some recent testing, pg_basebackup -Fp was taking an annoying amount of time to finish once the it was done copying the data. Using -v seemed to blame this on "waiting for background process to finish streaming", based on that being the last message on display while the delay was happening. Bu

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Robert Haas
On Fri, Jul 27, 2018 at 10:44 AM, Tom Lane wrote: > However, that does *not* mean that adding patent-related qualifiers to the > license is going to be OK with everybody. An easy counterexample is that > we get code from some of those selfsame companies with private forks, > which then feeds back

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Nico Williams
On Fri, Jul 27, 2018 at 09:30:45AM -0400, Robert Haas wrote: > If you think that the lack of a CLA and a patent grant never causes > extensive conversations with legal, I am quite certain that you are > incorrect. I know of multiple instances where this has been a > concern. > > Other open source

request for new parameter for disable promote (slave only mode)

2018-07-27 Thread Ioseph Kim
Hi. I want to build one master & multi slave environments to use physical replication. Slave nodes have low hardware spec, so I changed max_connection server parameters, and try start slave node. But I could not start slave nodes, because CheckRequiredParameterValues function (in src/backend/a

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Nico Williams
Even assuming you can't change the PG license, you could still: - require disclosure in contributions - require a wide grant in contributions - document all such grants separately from the copyright license Putting the grants in the license is convenient, but it's not required to include paten

Re: Early WIP/PoC for inlining CTEs

2018-07-27 Thread Andrew Gierth
> "Andreas" == Andreas Karlsson writes: >> WITH ctename AS [[NOT] MATERIALIZED] (query) Andreas> I think "NOT MATERIALIZED" would be a bit misleading since the Andreas> planner may choose to materialize anyway, It would certainly be possible to make an explicit NOT MATERIALIZED override

Re: Making "COPY partitioned_table FROM" faster

2018-07-27 Thread Robert Haas
On Wed, Jul 25, 2018 at 10:30 PM, David Rowley wrote: > I agree RANGE partition is probably the most likely case to benefit > from this optimisation, but I just don't think that HASH could never > benefit and LIST probably sits somewhere in the middle. > > HASH partitioning might be useful in case

Re: Deprecating, and scheduling removal of, pg_dump's tar format.

2018-07-27 Thread Robert Haas
On Thu, Jul 26, 2018 at 11:46 PM, Tom Lane wrote: > Andres Freund writes: >> Is there any real reason to retain it? > > As I recall, the principal argument for having it to begin with was > that it's a "non proprietary" format that could be read without any > PG-specific tools. Perhaps the direc

Re: Upper limit arguments of pg_logical_slot_xxx_changes functions accept invalid values

2018-07-27 Thread Robert Haas
On Wed, Jul 11, 2018 at 8:58 PM, Masahiko Sawada wrote: > While reading the replication slot codes, I found a wrong assignment > in pg_logical_slot_get_changes_guts() function as follows. > > if (PG_ARGISNULL(2)) >upto_nchanges = InvalidXLogRecPtr; > else >

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Andres Freund
Hi, On 2018-07-27 11:15:00 -0500, Nico Williams wrote: > Even assuming you can't change the PG license, you could still: > > - require disclosure in contributions That really has no upsides, except poison the area. Either we reject the patch and people doing so can reasonably be expected to kn

Re: Deprecating, and scheduling removal of, pg_dump's tar format.

2018-07-27 Thread Andres Freund
Hi, On 2018-07-27 12:51:17 -0400, Robert Haas wrote: > On Thu, Jul 26, 2018 at 11:46 PM, Tom Lane wrote: > > Andres Freund writes: > >> Is there any real reason to retain it? > > > > As I recall, the principal argument for having it to begin with was > > that it's a "non proprietary" format that

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Nico Williams
On Fri, Jul 27, 2018 at 10:01:40AM -0700, Andres Freund wrote: > On 2018-07-27 11:15:00 -0500, Nico Williams wrote: > > Even assuming you can't change the PG license, you could still: > > > > - require disclosure in contributions > > That really has no upsides, except poison the area. [...] Su

Re: Deprecating, and scheduling removal of, pg_dump's tar format.

2018-07-27 Thread Joshua D. Drake
On 07/27/2018 10:05 AM, Andres Freund wrote: Yeah. I don't mind removing really marginal features to ease maintenance, but I'm not sure that this one is all that marginal or that we'd save that much maintenance by eliminating it. My point is more that it forces users to make choices whenever

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Chapman Flack
On 07/27/2018 01:01 PM, Andres Freund wrote: > the patch and people doing so can reasonably be expected to know about > the patents, making further contributions by them worse. I'm not sure this line of thinking, which seems rooted in notions of tainted or cleanroom development from the copyright

Re: Deprecating, and scheduling removal of, pg_dump's tar format.

2018-07-27 Thread Jeff Janes
On Fri, Jul 27, 2018 at 12:51 PM, Robert Haas wrote: > On Thu, Jul 26, 2018 at 11:46 PM, Tom Lane wrote: > > Andres Freund writes: > >> Is there any real reason to retain it? > > > > As I recall, the principal argument for having it to begin with was > > that it's a "non proprietary" format tha

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Andres Freund
On 2018-07-27 13:33:28 -0400, Chapman Flack wrote: > On 07/27/2018 01:01 PM, Andres Freund wrote: > > > the patch and people doing so can reasonably be expected to know about > > the patents, making further contributions by them worse. > > I'm not sure this line of thinking, which seems rooted in

Re: Explain buffers wrong counter with parallel plans

2018-07-27 Thread Jonathan S. Katz
> On Jul 27, 2018, at 8:31 AM, Amit Kapila wrote: > > On Thu, Jul 26, 2018 at 7:31 PM, Jonathan S. Katz > wrote: >> >>> On Jul 25, 2018, at 10:25 PM, Amit Kapila wrote: >>> >>> >>> You mean to say the number (Buffers: shared read=442478) in HEAD, >>> right? If so, yeah, I am also wondering

Re: Deprecating, and scheduling removal of, pg_dump's tar format.

2018-07-27 Thread Robert Haas
On Fri, Jul 27, 2018 at 1:05 PM, Andres Freund wrote: > My point is more that it forces users to make choices whenever they use > pg_dump. And the tar format has plenty downsides that aren't immediately > apparent. By keeping something with only a small upside around, we > force users to waste ti

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Chapman Flack
On 07/27/2018 01:42 PM, Andres Freund wrote: > On 2018-07-27 13:33:28 -0400, Chapman Flack wrote: >> On 07/27/2018 01:01 PM, Andres Freund wrote: >> >>> the patch and people doing so can reasonably be expected to know about >>> the patents, making further contributions by them worse. >> >> I'm not

Re: How can we submit code patches that implement our (pending) patents?

2018-07-27 Thread Chris Travers
On Fri, Jul 27, 2018, 19:01 Andres Freund wrote: > Hi, > > On 2018-07-27 11:15:00 -0500, Nico Williams wrote: > > Even assuming you can't change the PG license, you could still: > > > > - require disclosure in contributions > > That really has no upsides, except poison the area. Either we rejec

Re: Deprecating, and scheduling removal of, pg_dump's tar format.

2018-07-27 Thread Tom Lane
Jeff Janes writes: > But he isn't proposing getting rid of -Fp, just -Ft. Isn't -Ft is just as > PostgresSQL-specific > as -Fd is? No. The point about -Ft format is that you can extract files that contain SQL text and COPY data, using nothing but standard Unix tools (i.e. tar). So just as with

Re: Deprecating, and scheduling removal of, pg_dump's tar format.

2018-07-27 Thread Andrew Gierth
> "Andres" == Andres Freund writes: >> Why did we invent "custom" format dumps instead of using a standard >> container-file format like tar/cpio/zip/whatever? Andres> Because they're either not all that simple, or don't random Andres> read access inside. But that's just a guess, not fac

Re: [HACKERS] advanced partition matching algorithm for partition-wise join

2018-07-27 Thread Robert Haas
On Fri, Jul 27, 2018 at 3:17 AM, Ashutosh Bapat wrote: > Apart from the complexity there's also a possibility that this > skipping will reduce the efficiency actually in normal cases. Consider > a case where A and B have exactly matching partitions. Current > partition matching algorithm compare a

Re: My Skype account (korotkovae) was hacked

2018-07-27 Thread Robert Haas
On Fri, Jul 27, 2018 at 7:18 AM, Alexander Korotkov wrote: > Some community members did communicate with me using my Skype account > (korotkovae). Today this account was hacked. You should know that > any requests to send money sent from my account are fraud. > Sorry for inconvenience. I've sen

Re: Avoid extra Sort nodes between WindowAggs when sorting can be reused

2018-07-27 Thread Alexander Kuzmenkov
Daniel, Thanks for the update. On 07/25/2018 01:37 AM, Daniel Gustafsson wrote: Hmm, this is missing the eqop fields of SortGroupClause. I haven't tested yet but does the similar degradation happen if two SortGroupCaluses have different eqop and the same other values? I wasn’t able to const

Re: [HACKERS] WAL logging problem in 9.4.3?

2018-07-27 Thread Andrew Dunstan
On 07/18/2018 10:58 AM, Heikki Linnakangas wrote: On 18/07/18 16:29, Robert Haas wrote: On Wed, Jul 18, 2018 at 9:06 AM, Michael Paquier wrote: What's wrong with the approach proposed in http://postgr.es/m/55afc302.1060...@iki.fi ? For back-branches that's very invasive so that seems risk

Re: patch to allow disable of WAL recycling

2018-07-27 Thread Jerry Jelinek
I've setup FreeBSD 11.1 in a VM and I setup a ZFS filesystem to use for the Postgres DB. I ran the following simple benchmark. pgbench -M prepared -c 4 -j 4 -T 60 postgres Since it is in a VM and I can't control what else might be happening on the box, I ran this several times at different times

Re: Segfault logical replication PG 10.4

2018-07-27 Thread Alvaro Herrera
On 2018-Jul-20, Minh-Quan Tran wrote: > Something like this? Can you provide a reproducer for this problem? -- Álvaro Herrerahttps://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Locking B-tree leafs immediately in exclusive mode

2018-07-27 Thread Alexander Korotkov
On Fri, Jul 27, 2018 at 6:11 PM Alexander Korotkov wrote: > On Fri, Jul 27, 2018 at 12:30 AM Simon Riggs wrote: > > On 26 July 2018 at 20:59, Alexander Korotkov > > wrote: > > > > > Great, thank you! So, I think the regression is demystified. We can > > > now conclude that on our benchmarks t

Re: add verbosity to pg_basebackup for sync

2018-07-27 Thread Michael Paquier
On Fri, Jul 27, 2018 at 11:58:42AM -0400, Jeff Janes wrote: > But it was really waiting for the syncs of the new -D dir to finish. The > attached patch adds a -v notice that it is starting to do the sync, with > the wording taken from initdb's equivalent message. This is a good idea. Would we wa

Re: BUG #15182: Canceling authentication due to timeout aka Denial of Service Attack

2018-07-27 Thread Michael Paquier
On Fri, Jul 27, 2018 at 02:40:42PM +, Bossart, Nathan wrote: > On 7/26/18, 11:16 PM, "Michael Paquier" wrote: > I think I'm essentially suggesting what you have in 0002 but without > the new RangeVarGetRelidExtended() callback. I've attached a modified > version of 0002 that seems to fix the

Removing useless \. at the end of copy in pgbench

2018-07-27 Thread Tatsuo Ishii
While populating pgbench_account table by using COPY FROM STDIN, pgbench sends out "\." at the end of the copy data. However this is only necessary in the version 2 of frontend/backend protocol (i.e. the version 3 protocol does not need it). I think we can safely remove the code to save a few CPU c

Re: Removing useless \. at the end of copy in pgbench

2018-07-27 Thread Tom Lane
Tatsuo Ishii writes: > While populating pgbench_account table by using COPY FROM STDIN, > pgbench sends out "\." at the end of the copy data. However this is > only necessary in the version 2 of frontend/backend protocol (i.e. the > version 3 protocol does not need it). I think we can safely remov

Re: Speeding up INSERTs and UPDATEs to partitioned tables

2018-07-27 Thread David Rowley
On 27 July 2018 at 19:11, Amit Langote wrote: > I've attached a delta patch to make the above changes. I'm leaving the > hash table rename up to you though. Thanks for the delta patch. I took all of it, just rewrote a comment slightly. I also renamed the hash table to your suggestion and change

Re: Removing useless \. at the end of copy in pgbench

2018-07-27 Thread Tatsuo Ishii
> Tatsuo Ishii writes: >> While populating pgbench_account table by using COPY FROM STDIN, >> pgbench sends out "\." at the end of the copy data. However this is >> only necessary in the version 2 of frontend/backend protocol (i.e. the >> version 3 protocol does not need it). I think we can safely

Bizarre behavior in libpq's searching of ~/.pgpass

2018-07-27 Thread Tom Lane
I noticed that there's some strange coding in libpq's choice of what hostname to use for searching ~/.pgpass for a password. Historically (pre-v10), it just used the pghost parameter: conn->pgpass = PasswordFromFile(conn->pghost, conn->pgport, conn->

Re: Explain buffers wrong counter with parallel plans

2018-07-27 Thread Amit Kapila
On Fri, Jul 27, 2018 at 11:12 PM, Jonathan S. Katz wrote: > >> On Jul 27, 2018, at 8:31 AM, Amit Kapila wrote: >> >> >> Yeah, that would be better. Today, I have tried the patch on both >> Head and PG11 and I am getting same and correct results. > > I have applied the the patch to PG11beta2 and