Re: libpq compression

2018-06-05 Thread Michael Paquier
On Tue, Jun 05, 2018 at 06:04:21PM +1200, Thomas Munro wrote: > On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik > Speaking of configuration, are you planning to support multiple > compression libraries at the same time? It looks like the current > patch implicitly requires client and server t

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Tomas Vondra
On 06/05/2018 09:22 AM, David Rowley wrote: On 5 June 2018 at 17:04, Tomas Vondra wrote: On 06/05/2018 04:56 AM, David Rowley wrote: Isn't there still a problem determining when the memory exhaustion actually happens though? As far as I know, we've still little knowledge how much memory ea

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread David Rowley
On 5 June 2018 at 17:04, Tomas Vondra wrote: > On 06/05/2018 04:56 AM, David Rowley wrote: >> Isn't there still a problem determining when the memory exhaustion >> actually happens though? As far as I know, we've still little >> knowledge how much memory each aggregate state occupies. >> >> Jeff

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-05 Thread Ashutosh Bapat
On Tue, Jun 5, 2018 at 1:07 PM, Amit Langote wrote: > On 2018/06/05 1:25, Alvaro Herrera wrote: >> In the meantime, my inclination is to fix the documentation to point out >> that AFTER triggers are allowed but BEFORE triggers are not. > > Wasn't that already fixed by bcded2609ade6? > > We don't s

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-05 Thread Amit Langote
On 2018/06/05 1:25, Alvaro Herrera wrote: > In the meantime, my inclination is to fix the documentation to point out > that AFTER triggers are allowed but BEFORE triggers are not. Wasn't that already fixed by bcded2609ade6? We don't say anything about AFTER triggers per se, but the following unde

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Tomas Vondra
On 06/05/2018 07:46 AM, Jeff Davis wrote: On Tue, 2018-06-05 at 07:04 +0200, Tomas Vondra wrote: I expect the eviction strategy to be the primary design challenge of this patch. The other bits will be mostly determined by this one piece. Not sure I agree that this is the primary challenge. Th

Re: Performance regression with PostgreSQL 11 and partitioning

2018-06-05 Thread Amit Langote
On 2018/06/05 13:44, Ashutosh Bapat wrote: > On Tue, Jun 5, 2018 at 5:50 AM, David Rowley wrote: >> On 5 June 2018 at 01:35, Ashutosh Bapat >> wrote: >>> I was wondering if we can get rid of append_rel_list altogether. In >>> your patch, you have saved AppendRelInfos by child_relid. So all the >>

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-05 Thread Amit Langote
On 2018/06/05 16:41, Ashutosh Bapat wrote: > On Tue, Jun 5, 2018 at 1:07 PM, Amit Langote > wrote: >> On 2018/06/05 1:25, Alvaro Herrera wrote: >>> In the meantime, my inclination is to fix the documentation to point out >>> that AFTER triggers are allowed but BEFORE triggers are not. >> >> Wasn't

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Konstantin Knizhnik
On 04.06.2018 21:42, Andres Freund wrote: Hi, On 2018-06-04 16:47:29 +0300, Konstantin Knizhnik wrote: We in PostgresProc were faced with lock extension contention problem at two more customers and tried to use this patch (v13) to address this issue. Unfortunately replacing heavy lock with lw

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 07:22, Masahiko Sawada wrote: On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik wrote: On 26.04.2018 09:10, Masahiko Sawada wrote: On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas wrote: On Tue, Apr 10, 2018 at 9:08 PM, Masahiko Sawada wrote: Never mind. There was a lot of

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Alexander Korotkov
On Tue, Jun 5, 2018 at 12:48 PM Konstantin Knizhnik wrote: > Workload is combination of inserts and selects. > Looks like shared locks obtained by select cause starvation of inserts, > trying to get exclusive relation extension lock. > The problem is fixed by fair lwlock patch, implemented by Ale

Re: Problem while updating a foreign table pointing to a partitioned table on foreign server

2018-06-05 Thread Kyotaro HORIGUCHI
Hello. At Mon, 04 Jun 2018 20:58:28 +0900 (Tokyo Standard Time), Kyotaro HORIGUCHI wrote in <20180604.205828.208262556.horiguchi.kyot...@lab.ntt.co.jp> > It fails on some join-pushdown cases since it doesn't add tid > columns to join tlist. I suppose that build_tlist_to_deparse > needs somethi

Re: [HACKERS] Transactions involving multiple postgres foreign servers

2018-06-05 Thread Masahiko Sawada
On Sat, May 26, 2018 at 12:25 AM, Robert Haas wrote: > On Fri, May 18, 2018 at 11:21 AM, Masahiko Sawada > wrote: >> Regarding to API design, should we use 2PC for a distributed >> transaction if both two or more 2PC-capable foreign servers and >> 2PC-non-capable foreign server are involved with

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Masahiko Sawada
On Tue, Jun 5, 2018 at 6:47 PM, Konstantin Knizhnik wrote: > > > On 05.06.2018 07:22, Masahiko Sawada wrote: >> >> On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik >> wrote: >>> >>> >>> On 26.04.2018 09:10, Masahiko Sawada wrote: On Thu, Apr 26, 2018 at 3:30 AM, Robert Haas wr

Re: why partition pruning doesn't work?

2018-06-05 Thread Amit Langote
Hi Dmitry, Thanks for creating the patch. I looked at it and have some comments. On 2018/06/04 22:30, Dmitry Dolgov wrote: >> On 3 June 2018 at 19:11, Tom Lane wrote: >> Dmitry Dolgov <9erthali...@gmail.com> writes: >>> Just to clarify for myself, for evaluating any stable function here would

Re: pg_replication_slot_advance to return NULL instead of 0/0 if slot not advanced

2018-06-05 Thread Petr Jelinek
On 05/06/18 06:28, Michael Paquier wrote: > On Mon, Jun 04, 2018 at 11:51:35AM +0200, Petr Jelinek wrote: >> On 01/06/18 21:13, Michael Paquier wrote: >>> -startlsn =3D MyReplicationSlot->data.confirmed_flush; >>> +if (OidIsValid(MyReplicationSlot->data.database)) >>> +startlsn

AtEOXact_ApplyLauncher() and subtransactions

2018-06-05 Thread Amit Khandekar
Hi, When a SUBSCRIPTION is altered, then the currently running table-synchronization workers that are no longer needed for the altered subscription, are terminated. This is done by the function AtEOXact_ApplyLauncher() inside CommitTransaction(). So during each ALTER-SUBSCRIPTION command, the on_c

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 13:29, Masahiko Sawada wrote: On Tue, Jun 5, 2018 at 6:47 PM, Konstantin Knizhnik wrote: On 05.06.2018 07:22, Masahiko Sawada wrote: On Mon, Jun 4, 2018 at 10:47 PM, Konstantin Knizhnik wrote: On 26.04.2018 09:10, Masahiko Sawada wrote: On Thu, Apr 26, 2018 at 3:30 AM, Ro

Re: \d t: ERROR: XX000: cache lookup failed for relation

2018-06-05 Thread Teodor Sigaev
Ah, I think this is the missing, essential component: CREATE INDEX ON t(right(i::text,1)) WHERE i::text LIKE '%1'; Finally, I reproduce it with attached script. INSERT 0 99 <- first insertion ERROR: cache lookup failed for relation 1032219 ALTER TABLE ERROR: cache lookup failed for relati

Re: commitfest 2018-07

2018-06-05 Thread Ashutosh Bapat
On Tue, Jun 5, 2018 at 9:02 AM, Tom Lane wrote: > Michael Paquier writes: >> On Mon, Jun 04, 2018 at 07:16:33PM -0400, Peter Eisentraut wrote: >>> There were some discussions about renaming the existing 2018-09 entry >>> versus inserting a new one at -07 and requiring patches to be moved back >>>

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Andres Freund
Hi, On 2018-06-04 22:18:56 -0700, Jeff Davis wrote: > On Mon, 2018-06-04 at 11:52 -0700, Andres Freund wrote: > > I wonder whether, at least for aggregates, the better fix wouldn't be > > to > > switch to feeding the tuples into tuplesort upon memory exhaustion > > and > > doing a sort based aggre

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Andres Freund
Hi, On 2018-06-05 09:35:13 +0200, Tomas Vondra wrote: > But I don't think we've considered copying the whole AllocSet. Maybe that > would work, not sure. Don't think you'd need to fully copy it - you can just have a "wrapper" memory context implementation that does the accounting and then hands t

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Andres Freund
Hi, On 2018-06-05 10:05:35 +0200, Tomas Vondra wrote: > My concern is more about what happens when the input tuple ordering is > inherently incompatible with the eviction strategy, greatly increasing the > amount of data written to disk during evictions. > > Say for example that we can fit 1000 g

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread David Rowley
On 6 June 2018 at 00:45, Andres Freund wrote: > On 2018-06-05 09:35:13 +0200, Tomas Vondra wrote: >> I wonder if an aggregate might use a custom context >> internally (I don't recall anything like that). The accounting capability >> seems potentially useful for other places, and those might not us

Re: why partition pruning doesn't work?

2018-06-05 Thread Dmitry Dolgov
> On 5 June 2018 at 12:31, Amit Langote wrote: > > doesn't look quite right. What says expr is really a Param? The patch > appears to work because, by setting pinfo->execparams to *something*, it > triggers execution-time pruning to run; its contents aren't necessarily > used during execution pr

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Andres Freund
On 2018-06-06 00:53:42 +1200, David Rowley wrote: > On 6 June 2018 at 00:45, Andres Freund wrote: > > On 2018-06-05 09:35:13 +0200, Tomas Vondra wrote: > >> I wonder if an aggregate might use a custom context > >> internally (I don't recall anything like that). The accounting capability > >> seems

Re: plans for PostgreSQL 12

2018-06-05 Thread Andres Freund
Hi, On 2018-06-05 06:32:31 +0200, Pavel Stehule wrote: > ./configure --with-libxml --enable-tap-tests --enable-debug --with-perl > CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer" > > [pavel@nemesis postgresql]$ gcc --version > gcc (GCC) 8.1.1 20180502 (Red Hat 8.1.1-1) > > I executed simple scrip

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Andres Freund
On 2018-06-05 13:09:08 +0300, Alexander Korotkov wrote: > On Tue, Jun 5, 2018 at 12:48 PM Konstantin Knizhnik > wrote: > > Workload is combination of inserts and selects. > > Looks like shared locks obtained by select cause starvation of inserts, > > trying to get exclusive relation extension loc

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread David Rowley
On 6 June 2018 at 00:57, Andres Freund wrote: > On 2018-06-06 00:53:42 +1200, David Rowley wrote: >> On 6 June 2018 at 00:45, Andres Freund wrote: >> > On 2018-06-05 09:35:13 +0200, Tomas Vondra wrote: >> >> I wonder if an aggregate might use a custom context >> >> internally (I don't recall anyt

Re: commitfest 2018-07

2018-06-05 Thread Stephen Frost
Greetings, * Ashutosh Bapat (ashutosh.ba...@enterprisedb.com) wrote: > On Tue, Jun 5, 2018 at 9:02 AM, Tom Lane wrote: > > Michael Paquier writes: > >> On Mon, Jun 04, 2018 at 07:16:33PM -0400, Peter Eisentraut wrote: > >>> There were some discussions about renaming the existing 2018-09 entry >

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Andres Freund
Hi, On 2018-06-06 01:06:39 +1200, David Rowley wrote: > On 6 June 2018 at 00:57, Andres Freund wrote: > > I think it's ok to only handle this gracefully if serialization is > > supported. > > > > But I think my proposal to continue use a hashtable for the already > > known groups, and sorting for

Re: commitfest 2018-07

2018-06-05 Thread Andres Freund
On 2018-06-04 23:32:18 -0400, Tom Lane wrote: > Michael Paquier writes: > > On Mon, Jun 04, 2018 at 07:16:33PM -0400, Peter Eisentraut wrote: > >> There were some discussions about renaming the existing 2018-09 entry > >> versus inserting a new one at -07 and requiring patches to be moved back > >

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Tomas Vondra
On 06/05/2018 02:49 PM, Andres Freund wrote: Hi, On 2018-06-05 10:05:35 +0200, Tomas Vondra wrote: My concern is more about what happens when the input tuple ordering is inherently incompatible with the eviction strategy, greatly increasing the amount of data written to disk during evictions.

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread David Rowley
On 6 June 2018 at 01:09, Andres Freund wrote: > On 2018-06-06 01:06:39 +1200, David Rowley wrote: >> My concern is that only accounting memory for the group and not the >> state is only solving half the problem. It might be fine for >> aggregates that don't stray far from their aggtransspace, but

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Tomas Vondra
On 06/05/2018 03:17 PM, David Rowley wrote: On 6 June 2018 at 01:09, Andres Freund wrote: On 2018-06-06 01:06:39 +1200, David Rowley wrote: My concern is that only accounting memory for the group and not the state is only solving half the problem. It might be fine for aggregates that don't str

install doesn't work on HEAD

2018-06-05 Thread Amit Kapila
Hi, On my win-7, I am facing $SUBJECT. I am consistently getting below error: Copying build output files...Could not copy postgres.exe at install.pl line 26. On digging, I found that it started failing with commit 3a7cc727 (Don't fall off the end of perl functions). It seems that we have added

Re: why partition pruning doesn't work?

2018-06-05 Thread Ashutosh Bapat
On Tue, Jun 5, 2018 at 6:24 PM, Dmitry Dolgov <9erthali...@gmail.com> wrote: >> On 5 June 2018 at 12:31, Amit Langote wrote: >> >> doesn't look quite right. What says expr is really a Param? The patch >> appears to work because, by setting pinfo->execparams to *something*, it >> triggers executi

RE: Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread serge
In our code base we have added a WithStats-Flavor for creating memory contexts. This api accepts a pointer to metric for accounting and it is inherited by all subcontexts unless overridden. So we only needed to change context creation API where we wanted (such as TopTansactionContext, Message Con

Locking B-tree leafs immediately in exclusive mode

2018-06-05 Thread Alexander Korotkov
Hi! Currently _bt_search() always locks leaf buffer in shared mode (aka BT_READ), while caller can relock it later. However, I don't see what prevents _bt_search() from locking leaf immediately in exclusive mode (aka BT_WRITE) when required. When traversing downlink from non-leaf page of level 1

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Tomas Vondra
On 06/05/2018 03:41 PM, se...@rielau.com wrote: In our code base we have added a WithStats-Flavor for creating memory contexts. This api accepts a pointer to metric for accounting and it is inherited by all subcontexts unless overridden. So we only needed to change context creation API where we

Re: [HACKERS] Moving relation extension locks out of heavyweight lock manager

2018-06-05 Thread Alexander Korotkov
On Tue, Jun 5, 2018 at 4:02 PM Andres Freund wrote: > On 2018-06-05 13:09:08 +0300, Alexander Korotkov wrote: > > It appears that buffer replacement happening inside relation > > extension lock is affected by starvation on exclusive buffer mapping > > lwlocks and buffer content lwlocks, caused by

Re: libpq compression

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 08:26, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: Thank you for this notice. Updated and rebased patch is attached. Hi Konstantin, Seems very useful. +1. + rc = inflate(&zs->rx, Z_SYNC_FLUSH); + if (rc != Z_OK) + { + return ZPQ_DECOMPRESS_E

Re: libpq compression

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 09:04, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik wrote: Concerning specification of compression level: I have made many experiments with different data sets and both zlib/zstd and in both cases using compression level higher than default doesn't c

Re: commitfest 2018-07

2018-06-05 Thread Tom Lane
Michael Paquier writes: > Okay. If we tend toward this direction, I propose to do this switch in > two days my time (Thursday afternoon in Tokyo) if there are no > objections, so as anybody has hopefully time to argue back. I think we probably have to wait longer. It's not clear to me when the

Re: Loaded footgun open_datasync on Windows

2018-06-05 Thread Laurenz Albe
Amit Kapila wrote: > On Fri, Jun 1, 2018 at 8:18 PM, Laurenz Albe wrote: > > Amit Kapila wrote: > > > On Fri, Jun 1, 2018 at 3:13 PM, Laurenz Albe > > > wrote: > > > > I recently read our documentation about reliability on Windows: > > > > > > > > > On Windows, if wal_sync_method is open_datasy

Re: install doesn't work on HEAD

2018-06-05 Thread Ashutosh Sharma
Hi, On Tue, Jun 5, 2018 at 7:08 PM, Amit Kapila wrote: > > Hi, > > On my win-7, I am facing $SUBJECT. I am consistently getting below error: > Copying build output files...Could not copy postgres.exe > at install.pl line 26. > > On digging, I found that it started failing with commit 3a7cc727

Re: commitfest 2018-07

2018-06-05 Thread Tom Lane
Andres Freund writes: > I'd rather create a new 2018-07, and just manually move old patches to > it. Otherwise we'll not really focus on the glut of old things, but > everyone just restarts working on their own new thing. I thought the idea was to clear out the underbrush of small, ready-to-go pa

Re: adding tab completions

2018-06-05 Thread Arthur Zakirov
On Sun, Jun 03, 2018 at 10:39:22PM -0500, Justin Pryzby wrote: > Find attached an update which also supports column completion using the legacy > non-parenthesized syntax. Thank you! > BTW..should that be toast.tuple_target ?? I think shouldn't. From the documentation "This parameter cannot be s

Re: plans for PostgreSQL 12

2018-06-05 Thread Pavel Stehule
2018-06-05 15:00 GMT+02:00 Andres Freund : > Hi, > > On 2018-06-05 06:32:31 +0200, Pavel Stehule wrote: > > ./configure --with-libxml --enable-tap-tests --enable-debug --with-perl > > CFLAGS="-ggdb -Og -g3 -fno-omit-frame-pointer" > > > > [pavel@nemesis postgresql]$ gcc --version > > gcc (GCC) 8.1

automating perl compile time checking

2018-06-05 Thread Andrew Dunstan
Here is a follow up patch to last weeks commit allowing all perl files to be checked clean for compile time errors and warnings. The patch contains a simple script to run the checks. The code that finds perl files is put in a function in a single file that is sourced by the three locations

Re: Concurrency bug in UPDATE of partition-key

2018-06-05 Thread Amit Khandekar
Attached is a rebased patch version. Also included it in the upcoming commitfest : https://commitfest.postgresql.org/18/1660/ In the rebased version, the new test cases are added in the existing isolation/specs/partition-key-update-1.spec test. -- Thanks, -Amit Khandekar EnterpriseDB Corporatio

Re: commitfest 2018-07

2018-06-05 Thread Andres Freund
On 2018-06-05 10:20:55 -0400, Tom Lane wrote: > Andres Freund writes: > > I'd rather create a new 2018-07, and just manually move old patches to > > it. Otherwise we'll not really focus on the glut of old things, but > > everyone just restarts working on their own new thing. > > I thought the ide

Re: Code of Conduct plan

2018-06-05 Thread Chris Travers
On Mon, Jun 4, 2018 at 6:59 PM, Joshua D. Drake wrote: > On 06/03/2018 04:08 PM, Gavin Flower wrote: > > My comments: >>> >>> 1) Reiterate my contention that this is a solution is search of problem. >>> Still it looks like it is going forward, so see below. >>> >>> 2) "... engaging in behavior th

Re: commitfest 2018-07

2018-06-05 Thread Joe Conway
On 06/05/2018 10:43 AM, Andres Freund wrote: > On 2018-06-05 10:20:55 -0400, Tom Lane wrote: >> Andres Freund writes: >>> I'd rather create a new 2018-07, and just manually move old patches to >>> it. Otherwise we'll not really focus on the glut of old things, but >>> everyone just restarts workin

Re: [MASSMAIL]Re: Code of Conduct plan

2018-06-05 Thread gilberto . castillo
Hello, Maybe must include policy of money support from several at member from country less earnings. For examplo Cuba. El 2018-06-05 10:45, Chris Travers escribió: On Mon, Jun 4, 2018 at 6:59 PM, Joshua D. Drake wrote: On 06/03/2018 04:08 PM, Gavin Flower wrote: My comments: 1) Reiterat

Re: commitfest 2018-07

2018-06-05 Thread Tom Lane
Joe Conway writes: > On 06/05/2018 10:43 AM, Andres Freund wrote: >> I think we've not fully agreed on that. I'd argue we should manually >> filter things into the next CF. And both small patches and older things >> should qualify. > Would it work to rename 2018-09 to 2018-07 and then make a pas

Re: [MASSMAIL]Re: Code of Conduct plan

2018-06-05 Thread gilberto . castillo
El 2018-06-05 10:54, gilberto.casti...@etecsa.cu escribió: Hello, Maybe must include policy of money support from several at member from country less earnings. El 2018-06-05 10:45, Chris Travers escribió: On Mon, Jun 4, 2018 at 6:59 PM, Joshua D. Drake wrote: On 06/03/2018 04:08 PM, Gavin

Re: I'd like to discuss scaleout at PGCon

2018-06-05 Thread MauMau
From: Simon Riggs > Passing detailed info between servers is exactly what XL does. > > It requires us to define a cluster, exactly as XL does. > > And yes, its a good idea to replicate some tables to all nodes, as XL does. > > So it seems we have at last some agreement that some of the things XL >

Re: commitfest 2018-07

2018-06-05 Thread Stephen Frost
Greetings, * Tom Lane (t...@sss.pgh.pa.us) wrote: > Joe Conway writes: > > On 06/05/2018 10:43 AM, Andres Freund wrote: > >> I think we've not fully agreed on that. I'd argue we should manually > >> filter things into the next CF. And both small patches and older things > >> should qualify. > >

Re: commitfest 2018-07

2018-06-05 Thread Andrew Dunstan
On 06/05/2018 10:43 AM, Andres Freund wrote: On 2018-06-05 10:20:55 -0400, Tom Lane wrote: Andres Freund writes: I'd rather create a new 2018-07, and just manually move old patches to it. Otherwise we'll not really focus on the glut of old things, but everyone just restarts working on their

Re: why partition pruning doesn't work?

2018-06-05 Thread David Rowley
On 5 June 2018 at 22:31, Amit Langote wrote: > Maybe, David (added to cc) has something to say about all this, especially > whether he considers this a feature and not a bug fix. Thanks, Amit. I had missed this thread. Yeah. I admit if I'd thought about this case when I wrote the code, then I'd

Re: commitfest 2018-07

2018-06-05 Thread David Rowley
On 6 June 2018 at 02:20, Tom Lane wrote: > I thought the idea was to clear out the underbrush of small, ready-to-go > patches. How old they are doesn't enter into that. I don't recall a 'fest where small ready to go patches getting anything but priority. -- David Rowley http

Re: commitfest 2018-07

2018-06-05 Thread Andres Freund
On 2018-06-06 03:14:58 +1200, David Rowley wrote: > On 6 June 2018 at 02:20, Tom Lane wrote: > > I thought the idea was to clear out the underbrush of small, ready-to-go > > patches. How old they are doesn't enter into that. > > I don't recall a 'fest where small ready to go patches getting > an

Re: commitfest 2018-07

2018-06-05 Thread Peter Eisentraut
On 6/5/18 09:12, Andres Freund wrote: > I'd rather create a new 2018-07, and just manually move old patches to > it. My concern is whether the commitfest app will handle that well. There is no "move to previous commit fest" button. So you'd have to do it in some evil way, possibly confusing the

Re: commitfest 2018-07

2018-06-05 Thread David Rowley
On 6 June 2018 at 03:17, Andres Freund wrote: > On 2018-06-06 03:14:58 +1200, David Rowley wrote: >> On 6 June 2018 at 02:20, Tom Lane wrote: >> > I thought the idea was to clear out the underbrush of small, ready-to-go >> > patches. How old they are doesn't enter into that. >> >> I don't recall

location reporting in TAP test failures

2018-06-05 Thread Peter Eisentraut
Right now, when a TAP test reports a failure, it looks something like this: # Failed test 'creating a replication slot' # at /../postgresql/src/bin/pg_basebackup/../../../src/test/perl/TestLib.pm line 371. That file location is where we call out to the test function provided by Test::Mo

Re: Code of Conduct plan

2018-06-05 Thread Joshua D. Drake
On 06/05/2018 07:45 AM, Chris Travers wrote: It is my hope that PostgreSQL.Org -Core chooses members for that committee that are exceedingly diverse otherwise it is just an echo chamber for a single ideology and that will destroy this community. If I may suggest:  The committee sho

Re: why partition pruning doesn't work?

2018-06-05 Thread Pavel Stehule
2018-06-05 17:07 GMT+02:00 David Rowley : > On 5 June 2018 at 22:31, Amit Langote > wrote: > > Maybe, David (added to cc) has something to say about all this, > especially > > whether he considers this a feature and not a bug fix. > > Thanks, Amit. I had missed this thread. > > Yeah. I admit if I

RE: Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread serge
Strange. We don't see this overeahd and measure a lot more than just a single metric: Of the grab below only the context->stats += size is need. Could easily be a macro. We also track overall backend size to cap it, and track memory contexts in shared memory (plan cache, function cache, messag

Re: [MASSMAIL]Re: Code of Conduct plan

2018-06-05 Thread gilberto . castillo
O.K, Remember my Country Please. El 2018-06-05 11:29, Joshua D. Drake escribió: On 06/05/2018 07:45 AM, Chris Travers wrote: It is my hope that PostgreSQL.Org -Core chooses members for that committee that are exceedingly diverse otherwise it is just an echo chamber for a sin

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread David Fetter
On Tue, Jun 05, 2018 at 02:56:23PM +1200, David Rowley wrote: > On 5 June 2018 at 06:52, Andres Freund wrote: > > That part has gotten a bit easier since, because we have serialize / > > deserialize operations for aggregates these days. > > True. Although not all built in aggregates have those de

Re: libpq compression

2018-06-05 Thread Konstantin Knizhnik
On 05.06.2018 10:09, Michael Paquier wrote: On Tue, Jun 05, 2018 at 06:04:21PM +1200, Thomas Munro wrote: On Thu, May 17, 2018 at 3:54 AM, Konstantin Knizhnik Speaking of configuration, are you planning to support multiple compression libraries at the same time? It looks like the current pat

Re: Code of Conduct plan

2018-06-05 Thread Magnus Hagander
On Tue, Jun 5, 2018 at 4:45 PM, Chris Travers wrote: > > > On Mon, Jun 4, 2018 at 6:59 PM, Joshua D. Drake > wrote: > >> On 06/03/2018 04:08 PM, Gavin Flower wrote: >> >> My comments: 1) Reiterate my contention that this is a solution is search of problem. Still it looks like it i

buildfarm vs code

2018-06-05 Thread Andrew Dunstan
At my talk at pgcon last Friday [1] I presented some ideas for how people could run a full buildfarm run against their code, including a 4 line recipe using some Docker recipes I had created. Thomas Munro suggested it would be even nicer if you could just point something like Appveypr at the co

Re: I'd like to discuss scaleout at PGCon

2018-06-05 Thread MauMau
From: Robert Haas On Thu, May 31, 2018 at 8:12 AM, MauMau wrote: >> Oh, I didn't know you support FDW approach mainly for analytics. I >> guessed the first target was OLTP read-write scalability. > > That seems like a harder target to me, because you will have an extra > hop involved -- SQL from

Re: buildfarm vs code

2018-06-05 Thread Andrew Dunstan
On 06/05/2018 12:09 PM, Andrew Dunstan wrote: notice at lines 36 and 34 or the Appveyor output, (from lines 16 and 19 of the recipe appveyor.yml) that this recipe does make a couple of adjustments. Of course that's lines 36 and 43. I seem to be getting more dyslectic as I get older ...

Re: [PATCH] Improve geometric types

2018-06-05 Thread Emre Hasegeli
> Those underscore-prefixed names are defined in Microsoft's > [3][4]. So now I'm wondering if win32_port.h needs to > #include if (_MSC_VER < 1800). I don't have the C experience to decide the correct way. There are currently many .c files that are including float.h conditionally or unconditio

Re: I'd like to discuss scaleout at PGCon

2018-06-05 Thread MauMau
From: Ashutosh Bapat > In order to normalize parse trees, we need to at > least replace various OIDs in parse-tree with something that the > foreign server will understand correctly like table name on the > foreign table pointed to by local foreign table OR (schema qualified) > function names and

Re: I'd like to discuss scaleout at PGCon

2018-06-05 Thread MauMau
From: Ashutosh Bapat > In order to normalize parse trees, we need to at > least replace various OIDs in parse-tree with something that the > foreign server will understand correctly like table name on the > foreign table pointed to by local foreign table OR (schema qualified) > function names and

Re: I'd like to discuss scaleout at PGCon

2018-06-05 Thread MauMau
From: Ashutosh Bapat > Each node need to be confiugred and maintained. That requires efforts. > So we need to keep the number of nodes to a minimum. With a > coordinator and worker node segregation, we require at least two nodes > in a cluster and just that configuration doesn't provide much > scal

Re: libpq compression

2018-06-05 Thread Peter Eisentraut
On 6/5/18 03:09, Michael Paquier wrote: > I just had a quick look at this patch, lured by the smell of your latest > messages... And it seems to me that this patch needs a heavy amount of > work as presented. There are a couple of things which are not really > nice, like forcing the presentation

Re: Remove mention in docs that foreign keys on partitioned tables are not supported

2018-06-05 Thread Alvaro Herrera
On 2018-Jun-05, Amit Langote wrote: > On 2018/06/05 16:41, Ashutosh Bapat wrote: > > On Tue, Jun 5, 2018 at 1:07 PM, Amit Langote > > wrote: > >> On 2018/06/05 1:25, Alvaro Herrera wrote: > >>> In the meantime, my inclination is to fix the documentation to point out > >>> that AFTER triggers are

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread David Fetter
On Tue, Jun 05, 2018 at 02:56:23PM +1200, David Rowley wrote: > On 5 June 2018 at 06:52, Andres Freund wrote: > > That part has gotten a bit easier since, because we have serialize > > / deserialize operations for aggregates these days. > > True. Although not all built in aggregates have those de

Re: I'd like to discuss scaleout at PGCon

2018-06-05 Thread MauMau
From: Merlin Moncure > FWIW, Distributed analytical queries is the right market to be in. > This is the field in which I work, and this is where the action is at. > I am very, very, sure about this. My view is that many of the > existing solutions to this problem (in particular hadoop class > solt

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Andres Freund
On 2018-06-05 19:04:11 +0200, David Fetter wrote: > On Tue, Jun 05, 2018 at 02:56:23PM +1200, David Rowley wrote: > > On 5 June 2018 at 06:52, Andres Freund wrote: > > > That part has gotten a bit easier since, because we have serialize > > > / deserialize operations for aggregates these days. > >

[PATCH] Trim trailing whitespace in vim and emacs

2018-06-05 Thread David Fetter
Folks, Here's my attempt to $subject. I've tested with vim, but I'm much less certain I got the EMACS code right. Is there any interest in such a feature? Best, David. --- .dir-locals.el| 3 ++- src/tools/editors/vim.samples | 1 + 2 files changed, 3 insertions(+), 1 deletion(-)

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Tom Lane
David Fetter writes: > On Tue, Jun 05, 2018 at 02:56:23PM +1200, David Rowley wrote: >> True. Although not all built in aggregates have those defined. > Just out of curiosity, which ones don't? As of > 3f85c62d9e825eedd1315d249ef1ad793ca78ed4, pg_aggregate has both of > those as NOT NULL. NOT NU

Re: [PATCH] Trim trailing whitespace in vim and emacs

2018-06-05 Thread Alvaro Herrera
On 2018-Jun-05, David Fetter wrote: Hi David > Here's my attempt to $subject. I've tested with vim, but I'm much less > certain I got the EMACS code right. > > Is there any interest in such a feature? I'd rather have the editor warn me (highlight) such things rather than fix them silently (I wo

Re: Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Jeff Davis
On Tue, 2018-06-05 at 08:39 -0700, se...@rielau.com wrote: > Strange. We don't see this overeahd and measure a lot more than just > a single metric: Let's investigate again. I wasn't able to repro the overhead on x86; Robert saw it on a POWER machine, and it was somewhat noisy. I don't think we we

Variable-length FunctionCallInfoData

2018-06-05 Thread Andres Freund
Hi, While prototyping codegen improvements for JITed expression evaluation, I once more hit the issue that the FunctionCallInfoData structs are really large (936 bytes), despite arguments beyond the fourth barely every being used. I think we should fix that. What I think we should do is convert

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Jeff Davis
On Tue, 2018-06-05 at 05:57 -0700, Andres Freund wrote: > But I think my proposal to continue use a hashtable for the already > known groups, and sorting for additional groups would largely address > that largely, right?  We couldn't deal with groups becoming too > large, > but easily with the numb

Re: commitfest 2018-07

2018-06-05 Thread Jonathan S. Katz
> On Jun 5, 2018, at 10:14 AM, Tom Lane wrote: > > Michael Paquier writes: >> Okay. If we tend toward this direction, I propose to do this switch in >> two days my time (Thursday afternoon in Tokyo) if there are no >> objections, so as anybody has hopefully time to argue back. > > I think we

Re: [PATCH] Trim trailing whitespace in vim and emacs

2018-06-05 Thread Tom Lane
Alvaro Herrera writes: > On 2018-Jun-05, David Fetter wrote: >> Is there any interest in such a feature? > I'd rather have the editor warn me (highlight) such things rather than > fix them silently (I wonder if it'd cause a mess with regression .out > files for example, which I do edit on occasio

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread David Fetter
On Tue, Jun 05, 2018 at 01:27:01PM -0400, Tom Lane wrote: > David Fetter writes: > > On Tue, Jun 05, 2018 at 02:56:23PM +1200, David Rowley wrote: > >> True. Although not all built in aggregates have those defined. > > > Just out of curiosity, which ones don't? As of > > 3f85c62d9e825eedd1315d249

RE: Variable-length FunctionCallInfoData

2018-06-05 Thread serge
Big +1 on this one. Here is what we did. It's very crude, but minimized the amount of pain: It helps that the C compiler treats arrays and pointers the same. I can dig for the complete patch if you want... Cheers Serge /* * This struct is the data actually passed to an fmgr-called function

Re: Variable-length FunctionCallInfoData

2018-06-05 Thread Andres Freund
Hi, On 2018-06-05 10:40:22 -0700, se...@rielau.com wrote: > Big +1 on this one. Cool. > Here is what we did. It's very crude, but minimized the amount of pain: I think I'd rather go for my approach in core though. Needlessly carrying around a bunch of pointers, and adding the necessary indirec

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Jeff Davis
On Tue, 2018-06-05 at 05:42 -0700, Andres Freund wrote: > That's an interesting idea, but it seems simpler to stick to > > hashing > > rather than using a combination strategy. It also seems like it > > would > > take less CPU effort. > Isn't the locality of access going to considerably better with

Re: [PATCH] Trim trailing whitespace in vim and emacs

2018-06-05 Thread David Fetter
On Tue, Jun 05, 2018 at 01:28:54PM -0400, Alvaro Herrera wrote: > On 2018-Jun-05, David Fetter wrote: > Hi David > > > Here's my attempt to $subject. I've tested with vim, but I'm much less > > certain I got the EMACS code right. > > > > Is there any interest in such a feature? > > I'd rather ha

Re: Spilling hashed SetOps and aggregates to disk

2018-06-05 Thread Andres Freund
Hi, On 2018-06-05 10:47:49 -0700, Jeff Davis wrote: > The thing I don't like about it is that it requires running two memory- > hungry operations at once. How much of work_mem do we use for sorted > runs, and how much do we use for the hash table? Is that necessarily true? I'd assume that we'd us

Re: I'd like to discuss scaleout at PGCon

2018-06-05 Thread MauMau
From: Simon Riggs On 1 June 2018 at 16:56, Ashutosh Bapat wrote: >> I think partitioning + FDW provide basic infrastructure for >> distributing data, planning queries working with such data. We need >> more glue to support node management, cluster configuration. So, I >> agree with your statement.

  1   2   >