[HACKERS] 9.4rc bug in percentile_cont

2014-12-12 Thread Andrew Gierth
Just got a report on IRC of a bug in the array version of percentile_cont; if two of the requested percentiles were between the same pair of input rows, the result could be wrong or an error would be generated. e.g. select percentile_cont(array[0.4,0.6]) within group (order by gs) from generate

Re: [HACKERS] Final Patch for GROUPING SETS

2014-12-12 Thread Andrew Gierth
> "Tom" == Tom Lane writes: >> I'd already explained in more detail way back when we posted the >> patch. But to reiterate: the ChainAggregate nodes pass through >> their input data unchanged, but on group boundaries they write >> aggregated result rows to a tuplestore shared by the whole

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Amit Kapila
On Sat, Dec 13, 2014 at 1:13 AM, Josh Berkus wrote: > > On 12/12/2014 11:35 AM, Alvaro Herrera wrote: > > Uh, really? Last I looked at the numbers from SPI treasurer reports, > > they are not impressive enough to hire a full-time engineer, let alone a > > senior one. > > > > The Linux Foundation

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Bruce Momjian
On Fri, Dec 12, 2014 at 08:57:55PM -0500, Tom Lane wrote: > Peter Eisentraut writes: > > On 12/12/14 10:16 AM, Tom Lane wrote: > >> I think pg_upgrade should continue to have SQL scripts that create and > >> delete the SQL function definitions for these. > > > That won't actually work very easily

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
El 12/12/2014 23:09, "Alvaro Herrera" escribió: > > Claudio Freire wrote: > > > Fair enough, but that's not the same as not requiring easy proofs. The > > planner might not the one doing the proofs, but you still need proofs. > > > > Even if the proving method is hardcoded into the partitioning me

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Peter Eisentraut
On 12/12/14 11:14 AM, Robert Haas wrote: > contrib is a nice middle-ground. I respect that view. But if we consider contrib a place where things can try to mature, then there must be a way to move things out of there once they have matured. If it turns out that moving things out of contrib cause

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Peter Eisentraut
On 12/12/14 11:45 AM, Andres Freund wrote: > On 2014-12-12 11:42:57 -0500, Robert Haas wrote: >>> I think the amount of effort a simple renamed directory which wholly >>> contains a binary creates is acceptable. Just use patch -p4 instead of >>> patch -p1... >> >> That is fine if you are manually a

Re: [HACKERS] On partitioning

2014-12-12 Thread Alvaro Herrera
Claudio Freire wrote: > Fair enough, but that's not the same as not requiring easy proofs. The > planner might not the one doing the proofs, but you still need proofs. > > Even if the proving method is hardcoded into the partitioning method, > as in the case of list or range partitioning, it's st

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Tom Lane
Peter Eisentraut writes: > On 12/12/14 10:16 AM, Tom Lane wrote: >> I think pg_upgrade should continue to have SQL scripts that create and >> delete the SQL function definitions for these. > That won't actually work very easily. LANGUAGE internal functions need > to be in fmgr_builtins, and the

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Peter Eisentraut
On 12/12/14 3:20 PM, Tom Lane wrote: > Pardon me for not knowing much about Debian packages, but how would > that work exactly? Is it possible to do make install-client, then > package the installed files, then rm -rf the install tree, then > repeat for install-server and install-contrib? In the

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Peter Eisentraut
On 12/12/14 10:16 AM, Tom Lane wrote: > I don't particularly object to having the C code built into the backend; > there's not that much of it, and if we could static-ize some of the global > variables that are involved presently, it'd be a Good Thing IMO. However, > the current arrangement makes

Re: [HACKERS] operator does not exist: character varying[] <> character[]

2014-12-12 Thread Tom Lane
Jim Nasby writes: > I'd say that array_eq (and probably _cmp) just needs to be taught to fall > back to what oper() does, but this part of the commit message gives me pause: > "Change the operator search algorithms to look for appropriate btree or hash > index opclasses, instead of assuming ope

Re: [HACKERS] operator does not exist: character varying[] <> character[]

2014-12-12 Thread Jim Nasby
On 12/9/14, 5:06 PM, Jim Nasby wrote: On 12/9/14, 4:30 PM, Tom Lane wrote: Jim Nasby writes: On 12/9/14, 4:19 PM, Jim Nasby wrote: Is there any particular reason we don't allow comparing char and varchar arrays? If not I'll submit a patch. We're also missing operators on text and varchar

[HACKERS] parallel mode and parallel contexts

2014-12-12 Thread Robert Haas
Attached is a patch that adds two new concepts: parallel mode, and parallel contexts. The idea of this code is to provide a framework for specific parallel things that you want to do, such as parallel sequential scan or parallel sort. When you're in parallel mode, certain operations - like DDL, a

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
On Fri, Dec 12, 2014 at 7:40 PM, Josh Berkus wrote: > On 12/12/2014 02:10 PM, Tom Lane wrote: >> Actually, I'm not sure that's what we want. I thought what we really >> wanted here was to postpone partition-routing decisions to runtime, >> so that the behavior would be efficient whether or not th

Re: [HACKERS] On partitioning

2014-12-12 Thread Josh Berkus
On 12/12/2014 02:10 PM, Tom Lane wrote: > Actually, I'm not sure that's what we want. I thought what we really > wanted here was to postpone partition-routing decisions to runtime, > so that the behavior would be efficient whether or not the decision > could be predetermined at plan time. > > Thi

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Claudio Freire
On Fri, Dec 12, 2014 at 7:25 PM, Michael Paquier wrote: > On Sat, Dec 13, 2014 at 1:08 AM, Robert Haas wrote: >> On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund wrote: Note that autovacuum and fsync are off. =# select phase, user_diff, system_diff, pg_size_pretty(pre_update - pre_

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

2014-12-12 Thread Peter Geoghegan
On Fri, Dec 12, 2014 at 5:19 AM, Robert Haas wrote: > Well, this is sort of one of the problems with work_mem. When we > switch to a tape sort, or a tape-based materialize, we're probably far > from out of memory. But trying to set work_mem to the amount of > memory we have can easily result in

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Michael Paquier
On Sat, Dec 13, 2014 at 1:08 AM, Robert Haas wrote: > On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund wrote: >>> Note that autovacuum and fsync are off. >>> =# select phase, user_diff, system_diff, >>> pg_size_pretty(pre_update - pre_insert), >>> pg_size_pretty(post_update - pre_update) from resu

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Michael Paquier
On Sat, Dec 13, 2014 at 1:00 AM, Alvaro Herrera wrote: > Tom Lane wrote: >> Heikki Linnakangas writes: >> > On 12/12/2014 03:07 PM, Peter Eisentraut wrote: >> >> On 12/9/14 4:10 PM, Alvaro Herrera wrote: >> >>> Maybe it makes sense to have a distinction between client programs and >> >>> server p

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Michael Paquier
On Sat, Dec 13, 2014 at 5:30 AM, Magnus Hagander wrote: > On Fri, Dec 12, 2014 at 9:05 PM, Josh Berkus wrote: >> On 12/12/2014 11:52 AM, Magnus Hagander wrote: >>> On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra wrote: On 11.12.2014 16:06, Bruce Momjian wrote: > On Wed, Dec 10, 2014 at 11

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Joshua D. Drake
On 12/12/2014 01:22 PM, Jim Nasby wrote: On 12/12/14, 1:44 PM, Joshua D. Drake wrote: 1. We don't need a full-time engineer to manage a commitfest. We need a manager or PM. I don't think that's actually true. The major points on this thread are that 1) we don't have enough capacity for doing

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
On Fri, Dec 12, 2014 at 7:10 PM, Tom Lane wrote: > Claudio Freire writes: >> On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas wrote: >>> I have very little idea what the API you're imagining would actually >>> look like from this description, but it sounds like a terrible idea. >>> We don't want to

Re: [HACKERS] On partitioning

2014-12-12 Thread Tom Lane
Claudio Freire writes: > On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas wrote: >> I have very little idea what the API you're imagining would actually >> look like from this description, but it sounds like a terrible idea. >> We don't want to make this infinitely general. We need a *fast* way >> t

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 4:54 PM, Tomas Vondra wrote: Well, this is sort of one of the problems with work_mem. When we switch to a tape sort, or a tape-based materialize, we're probably far from out of memory. But trying to set work_mem to the amount of memory we have can easi

Re: [HACKERS] On partitioning

2014-12-12 Thread Claudio Freire
On Fri, Dec 12, 2014 at 6:48 PM, Robert Haas wrote: > On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby wrote: >>> Sure. Mind you, I'm not proposing that the syntax I just mooted is >>> actually for the best. What I'm saying is that we need to talk about >>> it. >> >> Frankly, if we're going to requir

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

2014-12-12 Thread Tomas Vondra
On 12.12.2014 22:13, Robert Haas wrote: > On Fri, Dec 12, 2014 at 11:50 AM, Tomas Vondra wrote: >> On 12.12.2014 14:19, Robert Haas wrote: >>> On Thu, Dec 11, 2014 at 5:46 PM, Tomas Vondra wrote: >>> Regarding the "sufficiently small" - considering today's hardware, we're probably talki

Re: [HACKERS] On partitioning

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 4:28 PM, Jim Nasby wrote: >> Sure. Mind you, I'm not proposing that the syntax I just mooted is >> actually for the best. What I'm saying is that we need to talk about >> it. > > Frankly, if we're going to require users to explicitly define each partition > then I think t

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 12:19 PM, Alvaro Herrera wrote: >> If we decide that executables can no longer live in contrib, > > Nobody is saying that. The reason I though that might be on the table is that the first post on this thread, at least as I understood it, proposed to move every executable o

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 1:51 PM, Simon Riggs wrote: > What I don't understand is why we aren't working on double buffering, > since that cost would be paid in a background process and would be > evenly spread out across a checkpoint. Plus we'd be able to remove > FPWs altogether, which is like 100

Re: [HACKERS] Final Patch for GROUPING SETS

2014-12-12 Thread Tom Lane
Andrew Gierth writes: > "Tom" == Tom Lane writes: > Tom> That seems pretty messy, especially given your further comments > Tom> that these plan nodes are interconnected and know about each > Tom> other (though you failed to say exactly how). > I'd already explained in more detail way back whe

Re: [HACKERS] On partitioning

2014-12-12 Thread Jim Nasby
On 12/12/14, 8:03 AM, Robert Haas wrote: On Thu, Dec 11, 2014 at 11:43 PM, Amit Langote wrote: >In case of what we would have called a 'LIST' partition, this could look like > >... FOR VALUES (val1, val2, val3, ...) > >Assuming we only support partition key to contain only one column in such a

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Jim Nasby
On 12/12/14, 1:44 PM, Joshua D. Drake wrote: 1. We don't need a full-time engineer to manage a commitfest. We need a manager or PM. I don't think that's actually true. The major points on this thread are that 1) we don't have enough capacity for doing reviews and 2) the CFM has no authority

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 11:50 AM, Tomas Vondra wrote: > On 12.12.2014 14:19, Robert Haas wrote: >> On Thu, Dec 11, 2014 at 5:46 PM, Tomas Vondra wrote: >> >>> Regarding the "sufficiently small" - considering today's hardware, we're >>> probably talking about gigabytes. On machines with significan

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Jim Nasby
On 12/12/14, 2:38 PM, Josh Berkus wrote: Just as a note abot this one part along (I'll read the rest later). I do have the new version of the CF app more or less ready to deploy, but I got bogged down by thinking "I'll do it between two commitfests to not be disruptive". But there has been no "

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Josh Berkus
>>> Just as a note abot this one part along (I'll read the rest later). I >>> do have the new version of the CF app more or less ready to deploy, >>> but I got bogged down by thinking "I'll do it between two commitfests >>> to not be disruptive". But there has been no "between two >>> commitfests"

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Peter Geoghegan
On Fri, Dec 12, 2014 at 12:30 PM, Magnus Hagander wrote: >> It also automates a bunch of the emailing no? > > Yes. Please let me know the details (privately or otherwise). I'd like to try it out again. -- Peter Geoghegan -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Alvaro Herrera
Tom Lane wrote: > Christoph Berg writes: > > However, for PostgreSQL this means lengthy debian/*.install files > > (the equivalent of %files in rpm spec speak): > > Right ... > > > If there were separate "install-client", "install-server", and > > "install-contrib" targets, that would probably s

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Magnus Hagander
On Fri, Dec 12, 2014 at 9:05 PM, Josh Berkus wrote: > On 12/12/2014 11:52 AM, Magnus Hagander wrote: >> On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra wrote: >>> On 11.12.2014 16:06, Bruce Momjian wrote: On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote: >>> 3) manual processing tha

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Tom Lane
Christoph Berg writes: > However, for PostgreSQL this means lengthy debian/*.install files > (the equivalent of %files in rpm spec speak): Right ... > If there were separate "install-client", "install-server", and > "install-contrib" targets, that would probably shorten those files > quite a bit

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Christoph Berg
Re: Andres Freund 2014-12-12 <20141212152723.go31...@awork2.anarazel.de> > On 2014-12-12 10:20:58 -0500, Tom Lane wrote: > > Peter Eisentraut writes: > > > On 12/12/14 8:13 AM, Andres Freund wrote: > > >> Wouldn't a make install-server/client targets or something similar > > >> actually achieve th

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Josh Berkus
On 12/12/2014 11:52 AM, Magnus Hagander wrote: > On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra wrote: >> On 11.12.2014 16:06, Bruce Momjian wrote: >>> On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote: >> 3) manual processing that could be automated >> >>I think the CF site was a hug

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Magnus Hagander
On Fri, Dec 12, 2014 at 8:43 PM, Tomas Vondra wrote: > On 11.12.2014 16:06, Bruce Momjian wrote: >> On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote: > 3) manual processing that could be automated > >I think the CF site was a huge step forward, but maybe we could >improve it, to

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Joshua D. Drake
On 12/12/2014 11:35 AM, Alvaro Herrera wrote: We (not CMD, the community) with proper incentive could fund this. It really wouldn't be that hard. That said, there would have to be a clear understanding of expectations, results, and authority. Uh, really? Yeah I think so. Money can be easy t

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Josh Berkus
On 12/12/2014 11:35 AM, Alvaro Herrera wrote: > Uh, really? Last I looked at the numbers from SPI treasurer reports, > they are not impressive enough to hire a full-time engineer, let alone a > senior one. > > The Linux Foundation has managed to pay for Linus Torvalds somehow, so > it does sound

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Tomas Vondra
On 11.12.2014 16:06, Bruce Momjian wrote: > On Wed, Dec 10, 2014 at 11:00:21PM -0800, Josh Berkus wrote: >> >> I will add: >> >> 4. commitfest managers have burned out and refuse to do it again > > Agreed. The "fun", if it was ever there, has left the commitfest > process. I've never been a CFM,

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Alvaro Herrera
Joshua D. Drake wrote: > > On 12/12/2014 06:30 AM, Robert Haas wrote: > > >Yeah, that would be great, and even better if we could get 2 or 3 > >positions funded so that the success or failure isn't too much tied to > >a single individual. But even getting 1 position funded in a > >stable-enough

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Joshua D. Drake
On 12/12/2014 10:59 AM, Simon Riggs wrote: On 12 December 2014 at 15:10, David Fetter wrote: On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote: Josh Berkus writes: How about *you* run the next one, Tom? I think the limited amount of time I can put into a commitfest is better spent

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Joshua D. Drake
On 12/12/2014 06:30 AM, Robert Haas wrote: Yeah, that would be great, and even better if we could get 2 or 3 positions funded so that the success or failure isn't too much tied to a single individual. But even getting 1 position funded in a stable-enough fashion that someone would be willing t

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Joshua D. Drake
On 12/11/2014 02:55 PM, Tom Lane wrote: Josh Berkus writes: How about *you* run the next one, Tom? I think the limited amount of time I can put into a commitfest is better spent on reviewing patches than on managing the process. Agreed but That means committers/hackers have to suck it up

Re: [HACKERS] jsonb generator functions

2014-12-12 Thread Andrew Dunstan
On 12/12/2014 01:55 PM, Tom Lane wrote: Andrew Dunstan writes: Also here is a patch factored out which applies the find_coercion_pathway change to json.c. I'm inclined to say we should backpatch this to 9.4 (and with a small change 9.3). Thoughts? Meh. Maybe I'm just feeling gunshy because I

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Simon Riggs
On 12 December 2014 at 15:10, David Fetter wrote: > On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote: >> Josh Berkus writes: >> > How about *you* run the next one, Tom? >> >> I think the limited amount of time I can put into a commitfest is >> better spent on reviewing patches than on man

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Tomas Vondra
On 12.12.2014 19:07, Bruce Momjian wrote: > On Fri, Dec 12, 2014 at 10:50:56AM -0500, Tom Lane wrote: >> Also, one part of the point of the review mechanism is that it's >> supposed to provide an opportunity for less-senior reviewers to >> look at parts of the code that they maybe don't know so wel

Re: [HACKERS] jsonb generator functions

2014-12-12 Thread Tom Lane
Andrew Dunstan writes: >> Also here is a patch factored out which applies the >> find_coercion_pathway change to json.c. I'm inclined to say we should >> backpatch this to 9.4 (and with a small change 9.3). Thoughts? Meh. Maybe I'm just feeling gunshy because I broke something within the past

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Simon Riggs
On 12 December 2014 at 18:04, Bruce Momjian wrote: > Well, it seems we need to see some actual cases where compression does > help before moving forward. I thought Amit had some amazing numbers for > WAL compression --- has that changed? For background processes, like VACUUM, then WAL compressi

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Josh Berkus
On 12/12/2014 06:30 AM, Robert Haas wrote: > Yeah, that would be great, and even better if we could get 2 or 3 > positions funded so that the success or failure isn't too much tied to > a single individual. But even getting 1 position funded in a > stable-enough fashion that someone would be willi

Re: [HACKERS] Commitfest problems

2014-12-12 Thread David Fetter
On Fri, Dec 12, 2014 at 04:21:43PM +0100, Andres Freund wrote: > On 2014-12-12 07:10:40 -0800, David Fetter wrote: > > On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote: > > > Josh Berkus writes: > > > > How about *you* run the next one, Tom? > > > > > > I think the limited amount of time

Re: [HACKERS] jsonb generator functions

2014-12-12 Thread Andrew Dunstan
On 12/12/2014 01:10 PM, Andrew Dunstan wrote: On 12/08/2014 01:00 PM, Andrew Dunstan wrote: On 12/08/2014 04:21 AM, Alvaro Herrera wrote: Andrew Dunstan wrote: OK, here is a new patch version that * uses find_coercion_path() to find the cast function if any, as discussed elsewhere

Re: [HACKERS] jsonb generator functions

2014-12-12 Thread Andrew Dunstan
On 12/08/2014 01:00 PM, Andrew Dunstan wrote: On 12/08/2014 04:21 AM, Alvaro Herrera wrote: Andrew Dunstan wrote: OK, here is a new patch version that * uses find_coercion_path() to find the cast function if any, as discussed elsewhere * removes calls to getTypeOutputInfo() except w

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Bruce Momjian
On Fri, Dec 12, 2014 at 10:50:56AM -0500, Tom Lane wrote: > Also, one part of the point of the review mechanism is that it's supposed > to provide an opportunity for less-senior reviewers to look at parts of > the code that they maybe don't know so well, and thereby help grow them > into senior peo

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Bruce Momjian
On Fri, Dec 12, 2014 at 05:19:42PM +0100, Andres Freund wrote: > On 2014-12-12 11:15:46 -0500, Robert Haas wrote: > > On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund wrote: > > > On 2014-12-12 11:08:52 -0500, Robert Haas wrote: > > >> Unless I'm missing something, this test is showing that FPW > >

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Alvaro Herrera
Robert Haas wrote: > On Fri, Dec 12, 2014 at 11:00 AM, Alvaro Herrera > wrote: > > So let's put the whole bunch under src/bin/ and be done with it. > > I'm not really convinced this is a very good idea. What do we get out > of moving everything, or even anything, from contrib? We show that it'

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Bruce Momjian
On Fri, Dec 12, 2014 at 10:16:05AM -0500, Tom Lane wrote: > Peter Eisentraut writes: > > On 12/9/14 4:32 PM, Bruce Momjian wrote: > >> On Tue, Dec 9, 2014 at 06:10:02PM -0300, Alvaro Herrera wrote: > >>> (For pg_upgrade you also need to do something about pg_upgrade_support, > >>> which is good b

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Tom Lane
Andres Freund writes: > On 2014-12-12 11:42:57 -0500, Robert Haas wrote: >> And I don't know how well git cherry-pick will follow >> the moves. > Not well if the patch is done in master first. FWIW, I always patch master first, and have zero intention of changing that workflow. (I have given re

Re: [HACKERS] PATCH: hashjoin - gracefully increasing NTUP_PER_BUCKET instead of batching

2014-12-12 Thread Tomas Vondra
On 12.12.2014 14:19, Robert Haas wrote: > On Thu, Dec 11, 2014 at 5:46 PM, Tomas Vondra wrote: > >> Regarding the "sufficiently small" - considering today's hardware, we're >> probably talking about gigabytes. On machines with significant memory >> pressure (forcing the temporary files to disk), i

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Andres Freund
On 2014-12-12 11:42:57 -0500, Robert Haas wrote: > > I think the amount of effort a simple renamed directory which wholly > > contains a binary creates is acceptable. Just use patch -p4 instead of > > patch -p1... > > That is fine if you are manually applying a patch that touches only > that direc

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 11:40 AM, Andres Freund wrote: > The benefit of moving relevant stuff is that it'll actually be installed > by default when installing postgres on many platforms. That's currently > often not the case. The contrib umbrella, as used by many other > projects, actually justifi

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 11:26 AM, Tom Lane wrote: > Robert Haas writes: >> I'm not really convinced this is a very good idea. What do we get out >> of moving everything, or even anything, from contrib? It will make >> back-patching harder, but more importantly, it will possibly create >> the fa

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Andres Freund
On 2014-12-12 11:14:56 -0500, Robert Haas wrote: > I'm not really convinced this is a very good idea. What do we get out > of moving everything, or even anything, from contrib? The benefit of moving relevant stuff is that it'll actually be installed by default when installing postgres on many pla

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Tom Lane
Robert Haas writes: > I'm not really convinced this is a very good idea. What do we get out > of moving everything, or even anything, from contrib? It will make > back-patching harder, but more importantly, it will possibly create > the false impression that everything we distribute is on equal

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Joshua D. Drake
On 12/08/2014 07:50 PM, Tom Lane wrote: Peter Eisentraut writes: Last time this was attempted, the discussion got lost in exactly which extensions are worthy enough to be considered official or something like that. I want to dodge that here by starting at the opposite end: 1. move programs to

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Andres Freund
On 2014-12-12 11:15:46 -0500, Robert Haas wrote: > On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund wrote: > > On 2014-12-12 11:08:52 -0500, Robert Haas wrote: > >> Unless I'm missing something, this test is showing that FPW > >> compression saves 298MB of WAL for 17.3 seconds of CPU time, as > >>

Re: [HACKERS] [Bug] Inconsistent result for inheritance and FOR UPDATE.

2014-12-12 Thread Tom Lane
Etsuro Fujita writes: >> (2014/12/12 10:37), Tom Lane wrote: >>> Yeah, this is clearly a thinko: really, nothing in the planner should >>> be using get_parse_rowmark(). I looked around for other errors of the >>> same type and found that postgresGetForeignPlan() is also using >>> get_parse_rowmar

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 11:12 AM, Andres Freund wrote: > On 2014-12-12 11:08:52 -0500, Robert Haas wrote: >> Unless I'm missing something, this test is showing that FPW >> compression saves 298MB of WAL for 17.3 seconds of CPU time, as >> against master. And compressing the whole record saves a f

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 11:00 AM, Alvaro Herrera wrote: >> I'm pretty much -1 on relocating anything that's under src/bin already. I agree. I can't see packagers putting it anywhere except for $SOMETHING/bin in the final install, so what do we get out of dividing it up in some weird way in our t

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Andres Freund
On 2014-12-12 11:08:52 -0500, Robert Haas wrote: > Unless I'm missing something, this test is showing that FPW > compression saves 298MB of WAL for 17.3 seconds of CPU time, as > against master. And compressing the whole record saves a further 1MB > of WAL for a further 13.39 seconds of CPU time.

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 10:04 AM, Andres Freund wrote: >> Note that autovacuum and fsync are off. >> =# select phase, user_diff, system_diff, >> pg_size_pretty(pre_update - pre_insert), >> pg_size_pretty(post_update - pre_update) from results; >>phase| user_diff | system_diff | pg_

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Alvaro Herrera
Tom Lane wrote: > Heikki Linnakangas writes: > > On 12/12/2014 03:07 PM, Peter Eisentraut wrote: > >> On 12/9/14 4:10 PM, Alvaro Herrera wrote: > >>> Maybe it makes sense to have a distinction between client programs and > >>> server programs. Can we have src/sbin/ and move stuff that involves th

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Tom Lane
Robert Haas writes: > On Fri, Dec 12, 2014 at 9:15 AM, Alvaro Herrera > wrote: >> Robert Haas wrote: >>> (I note that the proposal to have the CFM review everything is merely >>> one way of meeting the need to have senior people spend more time >>> reviewing. But I assure all of you that I spend

Re: [HACKERS] Turning recovery.conf into GUCs

2014-12-12 Thread Alex Shulgin
Alex Shulgin writes: > Alex Shulgin writes: >> >> Here's an attempt to revive this patch. > > Here's the patch rebased against current HEAD, that is including the > recently committed action_at_recovery_target option. > > The default for the new GUC is 'pause', as in HEAD, and > pause_at_recover

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Andres Freund
On 2014-12-12 10:20:58 -0500, Tom Lane wrote: > Peter Eisentraut writes: > > On 12/12/14 8:13 AM, Andres Freund wrote: > >> Wouldn't a make install-server/client targets or something similar > >> actually achieve the same thing? Seems simpler to maintain to me. > > > Adding non-standard makefile

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Andres Freund
On 2014-12-12 07:10:40 -0800, David Fetter wrote: > On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote: > > Josh Berkus writes: > > > How about *you* run the next one, Tom? > > > > I think the limited amount of time I can put into a commitfest is > > better spent on reviewing patches than o

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Tom Lane
Peter Eisentraut writes: > On 12/12/14 8:13 AM, Andres Freund wrote: >> Wouldn't a make install-server/client targets or something similar >> actually achieve the same thing? Seems simpler to maintain to me. > Adding non-standard makefile targets comes with its own set of > maintenance issues. I

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Tom Lane
Peter Eisentraut writes: > On 12/9/14 4:32 PM, Bruce Momjian wrote: >> On Tue, Dec 9, 2014 at 06:10:02PM -0300, Alvaro Herrera wrote: >>> (For pg_upgrade you also need to do something about pg_upgrade_support, >>> which is good because that is one very ugly crock.) >> FYI, pg_upgrade_support was

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Tom Lane
Heikki Linnakangas writes: > On 12/12/2014 03:07 PM, Peter Eisentraut wrote: >> On 12/9/14 4:10 PM, Alvaro Herrera wrote: >>> Maybe it makes sense to have a distinction between client programs and >>> server programs. Can we have src/sbin/ and move stuff that involves the >>> server side in there

Re: [HACKERS] moving from contrib to bin

2014-12-12 Thread Andres Freund
On 2014-12-12 11:27:01 -0300, Alvaro Herrera wrote: > We already have src/bin/; the mixture of "src/" and "bin/" predates us. > Of course, the stuff we keep in there is not binaries but source code > that produces binaries. > > As for src/sbin/, we wouldn't install anything to the system's > /usr/

Re: [HACKERS] Commitfest problems

2014-12-12 Thread David Fetter
On Thu, Dec 11, 2014 at 05:55:56PM -0500, Tom Lane wrote: > Josh Berkus writes: > > How about *you* run the next one, Tom? > > I think the limited amount of time I can put into a commitfest is > better spent on reviewing patches than on managing the process. With utmost respect, Tom, you seem to

Re: [HACKERS] pg_rewind in contrib

2014-12-12 Thread Tom Lane
Heikki Linnakangas writes: > I'd like to include pg_rewind in contrib. I don't object to adding the tool as such, but let's wait to see what happens with Peter's proposal to move contrib command-line tools into src/bin/. If it should be there it'd be less code churn if it went into there in the

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Andres Freund
On 2014-12-12 23:50:43 +0900, Michael Paquier wrote: > I got curious to see how the compression of an entire record would perform > and how it compares for small WAL records, and here are some numbers based > on the patch attached, this patch compresses the whole record including the > block header

Re: [HACKERS] pg_rewind in contrib

2014-12-12 Thread Heikki Linnakangas
On 12/12/2014 04:20 PM, Andres Freund wrote: Not sure if the copyright notices in the current form are actually ok? Hmm. We do have such copyright notices in the source tree, but I know that we're trying to avoid it in new code. They had to be there when the code lived as a separate project,

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Andres Freund
On 2014-12-12 09:46:13 -0500, Bruce Momjian wrote: > On Fri, Dec 12, 2014 at 03:27:33PM +0100, Andres Freund wrote: > > On 2014-12-12 09:24:27 -0500, Bruce Momjian wrote: > > > On Fri, Dec 12, 2014 at 03:22:24PM +0100, Andres Freund wrote: > > > > > Well, the larger question is why wouldn't we just

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Michael Paquier
On Wed, Dec 10, 2014 at 11:25 PM, Bruce Momjian wrote: > On Wed, Dec 10, 2014 at 07:40:46PM +0530, Rahila Syed wrote: > > The tests ran for around 30 mins.Manual checkpoint was run before each > test. > > > > Compression WAL generated%compressionLatency-avg CPU usage > > (seconds)

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Bruce Momjian
On Fri, Dec 12, 2014 at 03:27:33PM +0100, Andres Freund wrote: > On 2014-12-12 09:24:27 -0500, Bruce Momjian wrote: > > On Fri, Dec 12, 2014 at 03:22:24PM +0100, Andres Freund wrote: > > > > Well, the larger question is why wouldn't we just have the user compress > > > > the entire WAL file before

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2014-12-12 Thread Rahila Syed
Hello, >Well, the larger question is why wouldn't we just have the user compress >the entire WAL file before archiving --- why have each backend do it? >Is it the write volume we are saving? IIUC, the idea here is to not only save the on disk size of WAL but to reduce the overhead of flushing WA

Re: [HACKERS] pg_rewind in contrib

2014-12-12 Thread Michael Paquier
On Fri, Dec 12, 2014 at 11:13 PM, Heikki Linnakangas wrote: > I'd like to include pg_rewind in contrib. I originally wrote it as an > external project so that I could quickly get it working with the existing > versions, and because I didn't feel it was quite ready for production use > yet. Now, wi

Re: [HACKERS] Compression of full-page-writes

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 9:34 AM, Michael Paquier wrote: >> I don't think that's a cost worth caring about. > OK, I thought it was. Space on the heap that never gets used is basically free. The OS won't actually allocate physical memory unless the pages are actually accessed. -- Robert Haas Ent

Re: [HACKERS] Commitfest problems

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 9:15 AM, Alvaro Herrera wrote: > Robert Haas wrote: >> (I note that the proposal to have the CFM review everything is merely >> one way of meeting the need to have senior people spend more time >> reviewing. But I assure all of you that I spend as much time >> reviewing as

Re: [HACKERS] Compression of full-page-writes

2014-12-12 Thread Robert Haas
On Fri, Dec 12, 2014 at 9:15 AM, Michael Paquier wrote: > I just meant that the scratch buffers used to store temporarily the > compressed and uncompressed data should be palloc'd all the time, even > if the switch is off. If they're fixed size, you can just put them on the heap as static globals

Re: [HACKERS] Compression of full-page-writes

2014-12-12 Thread Michael Paquier
On Fri, Dec 12, 2014 at 11:32 PM, Robert Haas wrote: > On Fri, Dec 12, 2014 at 9:15 AM, Michael Paquier > wrote: >> I just meant that the scratch buffers used to store temporarily the >> compressed and uncompressed data should be palloc'd all the time, even >> if the switch is off. > > If they're

Re: [HACKERS] pg_rewind in contrib

2014-12-12 Thread Bruce Momjian
On Fri, Dec 12, 2014 at 03:20:47PM +0100, Andres Freund wrote: > Hi, > > On 2014-12-12 16:13:13 +0200, Heikki Linnakangas wrote: > > I'd like to include pg_rewind in contrib. I originally wrote it as an > > external project so that I could quickly get it working with the existing > > versions, and

  1   2   >